Bay Area AI: Tatiana Erekhinskaya
Recording: Bay Area AI: Tatiana Erekhinskaya
[Music] Um, so my name is Tatan Hinsk. I work at Lima. It's a Dallas based NLP and semantic company. Uh, I do NLP, the whole NLP, nothing but NLP for more than 10 years. I started back in Russia where it was not popular at all. uh so and uh I even uh have experience with rule-based NOP because even data for machine learning was not available. So I kind of uh um know the whole history by hand. Um and yes basically today I would like to uh a little bit provoke you saying that deep learning is not going to work
Well, we use it as a compl compleimementary tool, but I want you kind of to see this another side. Uh, obviously, it's all based on my experience, but I just want to share consideration that uh you might find practical if you hit some real problems similar to this. Uh so to start with uh I will talk about use cases uh use cases that uh we work on on semantic approach that we use and then I will ask my questions about deep learning and uh share what we learned uh with kind of general working with customers on multiple NLP projects. Oh, one thing I wanted to do. How many of you do NLP ontology? Uh, okay. That's Yeah. Okay. And how many of you didn't go further than NLTK? Maybe a little bit of Corsera class
So, okay. So, mostly are professionals kind of here. Uh, okay. We'll see. So, that's what company uh does overall. So we take text in any format. We do a lot of pre-processing as well to get actually the text from documents from uh contracts and so on. uh and then we extract uh knowledge in different format and then we use this knowledge to drive different applications and uh so we do basically all kinds of things and recently feels like summarization is something that companies are interested about
So it uh got uh so his uh okay so historically we do semantic parsing like really deep analysis and question answering uh but uh so we are commercial company so people come to us with their use cases and uh last year we saw the same summarization projects coming again and again in different domains. So this is something that I want to share. Um so this is the first use case. Um see how good you could be a markettoologist because for me it was a real surprise uh to the company doing investment services and analytics based in London. Uh they want to be smart. They want to look only not on financial documents of the company but uh to know how green the company is. Uh so is it going to be sustainable or not? And what they did they asked to extract uh information about the company from court proceedings uh because everything uh all dirty details show up in the court. So uh that was a use case
I wouldn't be able to invent it really like uh I was very much surprised that somebody needs this combination of this data and this resource uh this information. So another use case is national security agencies. So we work a lot with government and uh basically they want to know um in what countries on what domains and who works who are the key leaders of opinions uh how advanced technologies especially if it is uh something that has issues with security. Um so the source was research publications and uh it's basically extraction of metadata plus extraction of uh the content of uh uh the of the research basically. So we created profiles about researchers. Um again so um before NLP it was like uh to annotate one researcher to create profile for one researcher it took uh half a year of somebody PhD level because it's a research paper so you need to know what's inside um and to extract all this manually and I mean you can imagine so with NLP is basically from the whole collection we extract uh have information about every everybody. So another example. So that uh so far with the uh company profile, person profile, this is a I'm sorry I'm okay
So it's a wrong title. This was proposal profile. Um so again government agency issue um call for proposals and then they get a pool. Some of them are well written, some of them are not so well written, some of them become funded, some of them rejected but after that there is a new uh call for proposals and the agency wanted to see what is the trend. So what people are interested about to make a more informed decision about what should be the next call. Uh so again so the information they needed was a kind of summary of proposal and u okay so this is about medical domain. We work with the center of medicare and medicaid uh who try to come up with quality measures for um for health care basically. So basically when you come to doctor with a high blood pressure uh the doctor says okay here is the pills u after some time you have a different blood pressure is it good enough? So doctor did a good job or not
uh and uh so for it's actually I still don't understand exactly how they do it but for us the task was here is pubet publications um and they want to know what was the clinical trial with all demography and all characteristics of patients and what was the intervention and what was the outcome and then they do something else on top of it to come up with the actual okay can you hear me okay because I sometimes get uh okay. Um so basically they needed to summarize information about clinical trial and uh here's my proono project uh with a um PhD student from uh management school just kind of to show you how deep it goes nowadays. Um so he wanted basically systematic review publication when he says okay here is my pool publication about big data and he only wants uh to collect information which company used big data it what industry what was the task what was the outcome so it in a way it even similar to the clinical trials because it's a problem with give a solution and there is some outcome So this in short what we see like again and again and so this is my kind of favorite use case because it's sort of recursive. So it's big data applied to big data and basically this is the common ground. So there is a lot of text and big text for humans is much smaller than big data for machine. So we get lost very easily and if only 5% of the text is useful then basically the whole text is very boring and nobody wants to read it. So this is uh what we normally do. So we um take oldfashioned semantic approach
So uh we sit with the customer we define template slots what they actually want to extract and uh then we collect annotations. Um and uh so this is where we get back when we talk about deep learning. And uh so basically then we train NLP and semantic tools to populate this slots. Um some important thing here is to balance recall and precision because uh depending on uh what customer prefers either they prefer a strong signal and they're okay to leave something uh in the data and never see it. Well then is one story. Uh so it's actually really depends on the customer. Um so this is something that we also need to be able to tune in our uh system and uh so for many tasks there is still human in the loop because uh so uh today somebody said a good phrase about it we take people away from copy paste business. So we do the rough information, we uh extraction, we don't do um uh like summarization on top
We don't generate text on top of it. Uh like paraphrasing because it's uh can be really tricky and you need to know what you're talking about. Um so we still allow human in the loop but uh uh so it's for review and maybe for uh uh kind of generalization some sort u but we also provide tools for them. So it's not like a black box that you cannot interact with. It's always some UI that supports users through the process. So here is basically what what is a template slot and it can be either sentence either named d entity it it can be classification label uh when we just uh first extract some fragment and then we say u is it use case about uh um is the result trans just sensing or is it transforming the company. So we classify on top of relevant fragments for that and another thing is semantic relations like uh in the case is medical domain. So the patient is uh undergoing some treatment
So do I have yeah I wanted just to show before you fall asleep. Uh here is this. So here is the domain model for medical. It's actually for nursing. Uh so okay uh let me just just so the size of the problem. So here is a patient and then it gets mapped to all kinds of metatasaurus tools uh concepts. So basically we know the parent is an organism and then we have patient can be part of population in some clinical trial and then all health status goes here and it's kind of hierarchical. Um and so we basically train NLP to extract instances of the concepts and relations
Um so basically if you look on NLP pipeline it's pretty standard. So it has sentence boundaries, tokenizer, name dy recognizer, word sense distribution, there dependency parsing and the last step is semantic parsing. This is something that it's our secret sauce kind of. Um so let me jump again here. So this is a piece of text from a nursing book and the representation that we extract for this. So uh first of all we extract concepts and the um relation between concepts that what we used to do till recently. So we for example say then 250 u so has a unit this measurement um and uh recently we started to say okay phrases we need also phrases around so uh we can do relations between uh uh like between graphs basically and uh we do it all in RDF so that we can do inference Here's what I need to check. How many of you heard about RDF? Okay, here is uh yeah, so it feels like a disconnect between an NLP community and semantics community
So we are kind of combining these two things together. So we say that okay here is an anttology. we prefer small ontologies uh so that we can train an LP to extract the instances and then we do inference on top. So like uh again for uh medical example for um let's say we have we know that uh under some uh medical condition you are not recommended to uh exercise and uh from somewhere like wordet we can say that jogging is an exercise. So from here we get inference that uh with this medical condition you shouldn't do jogging. So this is kind of something that has been around since uh 80s. So so it's basically uh subset of first order logic that is uh supported by uh RDF stores. Yes
So this is another model that we use for CMS and again it goes uh deeper. Okay. Uh so how many of you have have never heard the word ontology? Okay. Uh then just enjoy. So let's say that uh you are a six years old kid and you are going through exam to get to the school. So here's the teacher shows you four words and says uh so pick one that doesn't belong to the list. Are you uh okay? So remember your answer. So the teacher expects the kid to pick up wolf because wolf is wild scary and the rest are domestic
And the kid this was a real story. I know the kid. The kid actually said cow because cow is herbivores and the rest are carnivores and he was able to explain this. So to me this is something that taxonomy and autology can help us to do something that uh for some reason right now we are throwing all this knowledge from uh graphs away with deep learning. that is kind of still uh set for me. So and the way we go around is uh so let's say we have a text then we extract rel so if you have a definition or any explanation we just converted into semantic graph. So uh if you see that wolf dash a wild cardivorous mammal of the dog family uh then we can extract either relation by our semantic parsing and get ontology and do inference on top. Yeah this is uh okay something that really did mistakes
Um so again getting back to template feeling um so this is normally not very high paid projects and uh we need to make them reasonably fast and reasonably well. So what uh one of the problems we saw uh is ambiguous slots. A good example would be um drivers for proposals. we really needed to ask back like what do we mean drivers? Is it participants? Is it some causes? And uh what do you mean basically what do you want to extract? Um another thing uh even if definition is clear it's very possible that uh the the uh gold standard will be highly subjective and uh well it's so basically uh to get a label uh data correctly is a pain mostly because uh customer needs to be involved and they're not very cooperative and they don't know how that they feel like okay it we just label somehow they normally give it to somebody who doesn't care or doesn't know enough. So somebody uh whose time is not so precious and uh it brings back the results basically according to the standard. Uh so subjective slots is something that uh we saw several times and uh that's why we uh really ask them to have several people to look through the data and better if they don't communicate to each other. So like they do it in separation uh and then they look on the overall result fight and we take the winner basically. Um yeah and this is also something about sampling for annotation
Um we will never again allow customers to sample the data for us because they're very much biased. Uh like for example with the case for proposals what they did they annotated funded proposals because well they're nicely written they thought about them many times. um it was piece of cake for them to arate and it worked nicely on other um um funded proposals but when we apply the same system on not funded it's a different ball game so it was really hard to extract basically we needed to do a second iteration of annotation to to capture something that is not so well written. Um yeah uh this is another trick. I don't know why but all customers want 90%. Uh they don't know actually what is precision what is recall they cannot compute accuracy. Uh they even don't realize that for many NLP task 90% is like really really high. Uh they just want 90%
So the way around that for us was okay we give you 90% of time reduction of manual effort. So this is something that uh kind of uh run away sort of. Okay. So this is my favorite part. Uh I actually thought that I need to bring some tomatoes so that you can throw at me. But I want you to kind of stop sleeping and uh debate with me and I hope to learn something from this as well. Um so well very often we see that it's not enough of label data. It's either sensitive and confidential so we cannot put it on mechanical work or we cannot give it to some low paid people
um or it can be PhD hard so you basically it's so much in some biological or nano technology domain that you really need a PhD person to annotate so it is too expensive um or it can be it might be that it doesn't exist for example if it is uh some commands to some uh natural language uh interface then uh there is no corpus there is no nobody uh created the system yet. So there is no logs from uh which we can take at least unlabelled data. So the data doesn't exist at all. Um and uh so to me it's a little bit like deep learning versus traditional NLP. It's like a tradeoff between uh um writing rules. But if it's easier to write a rule, well let's write rule and just that's it. get our money and everybody happy. Uh then okay
So um instead of writing rules maybe we annotate data and then machine learning will learn rules for us. Basically um this is kind of again depends is it better uh is it more cost effective or not or we can say okay no rules no uh features everything is deep learning but then we need more data. So it depends on the cost of the unit basically. Uh so b uh I I agree that uh one rule is more um expensive than one unit normally but there are it can be vice versa actually. Um another thing that we see a lot so we start from some pilot projects when we don't have them have uh volume of data. So even if the customer is ready to share later on some really big data they kind of say okay you first demonstrate that you can do something useful and then we will give you on production stage everything. So it's basically like we need to learn swim first and then they will give us water. Um so this is like the third one is uh my like I I'm project manager uh and uh to me uh longer iterations with deep learning is a little bit uh uh worrisome so to say because uh one things uh I normally need to do is uh some talking to the customers they say okay we need that how soon it will be lady I need to say and uh I cannot say I don't know I I need to say okay in two weeks and I need to do it in two weeks u and uh it's okay if it will be half and half uh as soon as we can intelligently discuss why it is not working which features we can add and how it will be in future um so this is also something that is kind of a concern for Okay
And another concern yeah like okay we tried something let's say we tried deep learning what if it doesn't work um do I need more data right then how how much more data I need so it's again on costs what if it will not be enough again um yes so one of my concerns about deep learning is that uh it kind of makes us to start from scratch sort of well not always obviously like I really like what Soer did for sentiment analysis uh but again he didn't start from scratch he took dependency tree right and did it on top of something uh so what I see right now when we say that okay let's do it symbolically let's just look on the letters um I feel a little bit unsure what I will get because there is no way for me to say what except leveling data to say what I really want from that. Uh so I to me it's really like uh let's say there is metatasaurus it's huge huge ontology vocabulary of uh diseases uh how many times I need to show each disease to uh neural network so that it would actually remember it and uh I don't have this data uh even the corpus to support each instance in metadata results. Um yes and uh so what if the rules already exist? What do you see? Uh the recent project is extraction of uh information from contracts and they have like uh 600 fields and uh for this 600 they have like uh 200 pages of documentation how people would need to do it manually and it's basically if then rule for humans. So it's already written in natural language uh text basically in English for normal people what they need to do and we need to repeat um and so basically the number of contracts is finite. So it would be hard to uh really collect label data for each of these cases because that's what we need. We need to create a representative training set. So if we would go with deep learning, uh we would need to be super careful and like to put a lot of effort going through the rules and probably annotating not just the label but also the case like uh which sub rule in this for this field actually we need we would need to use because uh uh here is one I'm more concerned mistakes needs to look good because if customers uh see mistakes that looks good, they kind of believe well yeah computers are stupid but they can bring value but if they see something completely out of uh place they really freak out. Uh so in the testing period they it mistakes need to look reasonable
Um and uh so the topic of rules brings back kind of uh decision trees and uh explanability problem with deep learning. Uh because even with decision trees what I see when they look on the results they want to know okay why it is like that show me examples in training data that actually um provoke the system to make this decision. And yeah in case of decision tree um if customers are smart enough I can just show them decision tree if uh because what we do is essentially we don't do it like a bag of bag of words style we do relations so we uh we reduce dimensionality instead of uh words we use concepts or types of concepts we use relations so uh it brings from bag of words that would be huge It probably can be like 10 15 features maybe 30 features but it's still something that we can understand like humans looking on this um well and uh so basically when uh going to the next item when the project is over if we did good job creating some resources like dictionaries or some well regular expression set we can reuse it for the next project with deep learning I don't think so we probably can reuse word vectors but other than that I I don't see how we can well let's say if we built the super dictionary of company names in Chinese um we can bring bring this list again to a different Chinese project but With deep learning, it would be only if it train a separate neural network that specifically recognizes Chinese company names in the text. Okay. So this is uh something well okay. So I to me it feels like this is forgotten tool. How many of you have used this? Yeah. Okay
So kind kind of forgotten. Yeah. Uh so for some reason there is so many topic models uh that uh like in Python and anything. So and I was using them having some troubles with installation or something. So mallet is basically machine learning library that has been around for a while. is written in Java and it has topic models that can recognize phrases. So it's not just words but it's phrases also and uh it can work from common line and it's very smooth. Uh so this is just some examples of uh um what it brought back
So basically the first was extracted from medical publications and that's just terms it grouped it to together. So it was like uh just thousands of publications. It was not a lot. Um it was also able to combine together all eating disorders from the corpus and we also did we tokenized Chinese and printed it such that there are spaces now in the text so that the tool would be able to tokenize it kind of uh itself the text and uh so this was actually set on radars and nanotechnology technology but for some reason it was SVM there and sometimes then it was able to collect them all together. So BPnik as you know is the author of SVM and uh uh I think it's a little broken but uh in Chinese it's actually means support vectors. So I mean no annotation uh and it works very fast. It's you don't even uh need uh it uh like you don't need spark or anything. You can run it on your own machine
Um so the way we use it is like is for mining for lexicons. So if we are in a new domain, we don't know at all what we dealing with or if you are looking for some key terms in this domain, uh this is something that is very quick quick to try and see what is all about that the document set. Um yes and uh this is uh some uh so I I thought that uh uh at the beginning I should do like NLP 101 sort of but I realized in time that okay no no you know everything already. So this is just some things that for some reason uh kind of escape attention of NLP people. So one thing is RDF right? SA is actually implemented as a I know I wanted to say it. Yeah. Yeah. Uh so uh RDF is something that you can look at uh and there is a whole conference going on in the valley right now
So some people say that it's old and dead but now with no SQL databases and graph databases um I think it really will come back basically because it allows doing inference on top of the graph. So not just analyze graph as uh the uh network like uh to find the shortest path or some clicks uh but actually to look on the graph as a semantic data uh and do inference on top of it. So like a combine relation and create a new relation on top of them. Um uh so speaking about graphs uh well I guess word that everybody know just let let's make sure that nobody forgets wiki data dbpend and so on. So again uh it's very easy to extract lexicons from there um and just use this traditional NOP and it will give at least P very quickly and yes so Sienna is a u super monster like it does every all NLP from scratch with a single model it's actually based on uh neural networks I put it here just because uh it was done in 200 2011. So neural networks were not that hot at that time. Um like it was it wasn't even called deep learning. I think it was just neural networks
Um and yeah just I really don't want this tool to be forgotten. So this is if you how many of you never used Senna? Okay. So yeah, worth trying uh especially if you if you use NLTK uh and unsatisfied with semantic syntactic parsing and the absence of semantics there. So Senna is kind of something that you can write on top and uh okay so the last tool which is I guess missing from this list um Apache solar um again so it's a little bit not NLP but what is possible to do if you index fragments or if you index sentences u you can actually index named entities in the sentences or relations in the sentences and uh then you can write queries in solar language rather than uh creating your own uh uh well sophisticated reax 2.0 kind of stuff. So uh it's already enough to start with uh quering and see what you get. So it's again like a tool for exploring. So uh to me like first things that can be tried is LG. So that when you get approximately the key terms and then uh uh solar so that okay do you really understand what what we are talking about and it can be done like really in one hour
So just to estimate uh um should you take this project um so uh yes and basically the last thing I wanted to say is I really looking forward to uh to deep learning evolving and being able to take in knowledge graphs and uh so that that's all ontology and uh collected knowledge would be used for something and deep learning as a super flexible tool, super good for generalization would be able to catch up with this as well and would be able to tell why something is correct. So, thank you. [Applause] the questions for the recording. We'll have some questions. Okay, sure. So, how do you get the knowledge graph into the deep learning? That's I don't know. It's an open question. I really want I I like I'm not an active researcher in deep learning
I'm more like uh if there is only paper and no tool, I will let it go for now. I'm only playing with tools which is available. Question. Okay. Uh do I need a mic? No. Okay. So the question was uh how to put knowledge graphs knowledge basis into deep learning. So the qu answer is I don't know but I really want it to happen
Actually it's an open it's actually there's a lot of research aware of them and there are several solutions already. uh possible solutions are incorporating the constraint into the loss functions. Other approaches actually um let's do it this way learn a way of representing each node and each relationships in the graph in a way similar to vector where you learn like vectors for nodes or vectors for relationships or also other higher level structures like matrix or tensors for relations. So it fits in some of our number research field for example Google knowledge book paper and also there is like a nice paper from Max which covers the art and relational learning paper from actually questions. Sure. So I just ask so because we are recording and the Tiana has the only mic which so if you want to add something I'll ask you to stand next to the speaker and then she'll pick up the song. Well I can pass another mic or it doesn't have Oh okay. So for like when you're going through PubMed, how deep into the articles did you go? Did you just do the summary conclusions kind of thing? I mean the deep part of the N article even the researchers don't understand
Well, okay. So um we definitely go through abstract like deeply into abstract uh because abstracts are always available and the rest is maybe uh so but then we go through the whole publication as well. So we basically create it for everything create representation for everything. Um so it's good enough to power the search like uh uh really well. So if they need information uh so we power it like with solar we put relations into solar and uh so that works like really neatly uh if you say that how recall uh well it can be this and that so there's no magic it's like if it's something that we never saw in training data because we still train relations. uh the only thing that with when we train relations uh we say that we use two layer of representations so we use basic relations it's similar to semantic row labeling like Senna does but uh it goes into every word basically so it's a and it's standard set of relations similar to roles of verb verbet uh but expanded with uh value or manner so that everything is connected. Um so this is a basic representation. Uh customers nor normally do not care if it is an agent or theme
They just some boring stuff. Uh what they care is something like works for u treats between drug and patient. So we this domain specific relations we uh learn them on top of basic representation. So basically uh we do uh universal layer which we could use treebank we could use all kinds of uh annotated data. uh we did the more or less robust this basic layer and uh it saves us some effort on uh uh creating high level relations because we don't need to annotate again from scratch and the amount of annotation we need is smaller because we already have the basic blocks. Uh can you provide any um examples of the scale or performance of MBA? What kind of uh size corpus you could use it with? Okay. So uh oh okay so um the scale and performance of LDA uh I personally run it on quite small size. So it was uh all abstracts of 10 years psychiatry publications on on PubMed
Um so it was uh it was 10 thousands of abstracts. So it was pretty small small set. Uh it worked like uh in minutes. So I wouldn't be even able to drink some coffee. Uh we also did with Chinese uh so but we tokenize. So basically LDA doesn't care. It's just symbols. And um then it probably and we had like uh we had full PDFs and it's like 20,000 but it still was uh like probably launch time
Uh so it that was the scale and it's single machine. Yes. So it's not uh uh spark. something that you can really really quickly estimate. Sure. I had tried it like maybe five years ago um with uh scholarly content of uh over a million uh articles and it was uh taking a long time. I remember I had to switch uh elsewhere. I see
I don't remember exact times. But uh the benefit of course is the fact that you don't get a better words phrases. Yes. Yes. Many many things cannot do it. It just uh I was using mallet a lot with machine learning and uh I was actually starting with LDA with some other tools and then oh okay it actually can do everything already. It was like a surprise to me and that was another question back there. Oh um well so it's in many parts but let me just start by asking how much deep learning um does actually use so we uh so the question is how much deep learning we use and uh little so we train word vectors either with water to vehic or with glove recently so uh and we use it mostly for similarity between words because our applications if we need relations between words we need them to be typed
So is it is a relation or part and word to work doesn't really give it to you. Uh so but for similarity uh as additional feature like similarity assign similarity between vectors as additional feature together with all uh it was Ted Peterson uh uh pearl I I believe tool uh with bunch of like worded similarity like three different things based on the place in the hierarchy. Um so together with these guys uh that's what we use but uh again because a lot of things needs to be typed in in our things uh in our tasks I would say uh we use it only as additional thing and another so similarity and expansion of uh search queries it's again kind of similarity relatedness for question because we need to move to global sector. Okay. Um so in that you have a big list of uh um implementation problems with deep learning directions. Um, but a lot of those problems seem maybe not solvable, but at least addressable by the kind of human in the loop iterative techniques you talked about as being kind of a mainstay of the the things you you build. So like you know corpus uh annotation for training if you build a simple rulebased system with some basic stuff and you get some examples and then you build a little net and then you reanotate and go back and forth five times you get pretty far pretty quickly. Um have you guys tried anything? Um okay so the question is uh that my list of concerns towards deep learning can be solved iteratively with human in the loop
U so I can tell you like u uh one thing basically we need to do PVC successfully we need to go to production successfully after that most likely we will never see a lot of data because nobody wants to share data with us. So we uh there is probably only one exception. Uh so it was a project for medical diagnosis. Uh so the task was uh to classify diagnostic questions into uh more than 2,000 uh labels 2,000 questions. So it's basically paraphrase recognizing paraphrasing. U so just kind of for fun uh guess how many samples you had for each two. one. This is it's it's true
So for some of the data we had just one and we needed to create for the um and if you had data from customer we had five. So that was our start starting point and so um basically what we did there was uh we said that we played a little bit with the labels. So some labels were like really independent other labels were more like matrix. So we reduced a little bit the amount of them. So it was like uh uh so in medical it can be disease uh let's say and uh um some question about this disease and something like the severity have you been in hospital with this and so on so on so on and so it for different diseases there are the same questions. So it's like a metric. So we found several matrixes like that and it helped us to say that okay we first do big uh big uh buckets of classification then okay in this here is the matrix we need this classification and this classification. So this was uh something that we needed to do because uh uh interestingly even if u we would uh do more data ourself when it come to production the data the actual data looked very different from what we annotated
So our annotated data looked uh our questions looked very nicely and uh as a customer marketologist would use it and this was a show and but the actual students they just put some crap into the system and expected some uh suggestions for Yeah. Yeah. Uh like instead of questions I just give one word like hospital what hospital like for what what disease? um something like that. uh so uh coming back to deep learning now so the system has been in production and uh so I most likely have enough data to try deep learning and like luckily it's a SAS project so I have the data uh because so far it was retraining SVM and all this linear regression uh classifiers u and this is something that I want to try but it's also uh like now I have the data and the project is done I don't have very much motivation to only my curiosity basically powers that that is needs to be shared among other stuff as well. So it's kind of if the project is done then why do we need deploy link after that? Thank you. [Music]