Bay Area AI: Antonio Valderrabanos, Bot Training: How to Achieve Full Automation

Bay Area AI: Antonio Valderrabanos, Bot Training: How to Achieve Full Automation

Recording: Bay Area AI: Antonio Valderrabanos, Bot Training: How to Achieve Full Automation

We are going to be talking about bots and training. For those unfamiliar with bots, we are going to be using it as a specific case of our point in the industry, so to say. What we are trying to say to the industry with this presentation and everything we do is that there is a point in automating training data. So we build these beautiful algorithms that are like Formula 1s. We spend hours of expensive engineers. They have a beautiful purpose. And then we feed them with this pure gasoline. They are made by hand, completely inconsistent

So there is like a workshop or handcraft ingredient in the process that should be, this software should be fully automated. So that is the point we are going to make. So three slides about the company, so you know more or less who we are. Like Alexis said, we are based in Madrid, Spain, in Europe. We have the production center. And there and here mostly the sales and face to the client because 80% of our clients or something like that come from here, from the bay area. So we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going okay and then we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to make sure that we are going to always around the idea of automating the training is that we are going to solve typical language problems like negation okay many many many applications react the same way if you say I like it and I don't like it okay or turn it on or don't turn it on that we are going to explain what that is second point uh training automation third point as a result of industrializing the training you get better performance better accuracy and this is where middleware integration is a must with any platform okay so we are going to be using a couple of platforms as examples but don't think of those as the only platforms okay everything fine no questions next section okay i guess for the topic of the previous talk that not many people are familiar here with thoughts right who would say she or he is familiar with thoughts okay so for you this with the point this is like a really like an ecosystem around bots on the left you have the big platform so these names are the ones that are making the difference in this industry they are pushing strong these are companies that are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are so we have to make the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry they are making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this industry is making the difference in this I mean, but you are right, hardware, this new hardware is being a game changer, no one is using a bot for support in the office or for interacting with the bank or any of those things, hopefully we will okay I'll go straight to the results here and then with some more time I will go into the how we do it, okay, because I guess that's the most interesting part for today, okay, can you still hear me in the back? okay what we have done is something that is a bit difficult to explain but really simple to understand in order to have conditions to be able to test a bot with specific software and with another specific software we have defined what's called one intent, one intent is like enable the alarm or open the door, something you want or can do with the bot with that intent and one utterance or one sentence or one user query we train the bot and the additional software and the change in results is spectacular and spectacular in the sense that it's 33% to another percent in both cases and also spectacular in the sense that it's a big difference, no? so, the meaning of this is that if you train a bot with API.AI with one single sentence and then you give it 10 different sentences like before okay, like this sentence, you train it with this and you try it with this okay you can do the test with Luis and there are very different types of platforms and both of them will recognize like 3-4 sentences out of 10 okay, if you add an automatic training layer, a data generation layer, the precision grows dramatically okay that's the point, I don't want you to believe it yet okay, that's what we're going to see how we do, but it's important that we all understand where we're going the point we're making is if you make data generation an industrial process, an automated process, an engineering process then things will go much better, you will lose subjectivity, bias, etc. okay now, how does that happen? these are the typical steps in bot creation you create an agent, you create what they call the..

this is SWAT, sorry you create what they call an intent, enable an alarm, for example you create the entities, which is action enabled, object alarm and then you go into different paths, you train with bot tenders, or you train with all the automatically generated data so this is the bot development, or the bot training phase, and this is the bot in use, finally I'm going to try, I'm connected to the WiFi, so I'm going to try and show you the demo that we have, that you will, I'll be very happy to share with Alexei, and you can try it's nothing really spectacular, but I think it's very, it's very talented what we have here is a description of the demo task, we have here a sentence can you read that? is the size okay? it says turn on lights in the kitchen, please okay? so, we have two options here, to analyze without the bitext addition or the bitext addition, and with, okay? so, this is complex, this would deserve like half an hour comment on how do you exactly define the conditions for an evaluation like this I'll skip that, it's not probably the best thing for today so, anyway, if you analyze without data generation you will recognize that here there are missing pieces, no? so, basically, the application is telling you what to turn on, where to do it, but not what okay? the device lights, is missing, okay? and this is a very typical thing in machine learning we could call this 66% accuracy, because you have got two things out of three, okay? that's a very mathematical way of deceiving yourself, no? or making you think something that is not true I mean, this is something that you either get or don't get if you don't get right getting part of it, it doesn't make sense, no? particularly in environments like Google Home, or Alexa, or Amazon Echo where if you say turn off the oven, and it doesn't get the oven will be there forever and you'll have a big fire, no? so, there are many, this is not like sentiment analysis or many other applications where you say well, I'm 80, 82, 85, 90% accurate no, these things require 100% or nearly 100% accuracy, no? now, if you go back and process with the additional training data of course the application is going to understand they need to light to turn on the light in the kitchen okay? like I said, nothing really spectacular but it's there, you can try and hopefully it will work okay so, back here I have the slide eh just in case there was no wifi or you get the presentation so, this is the same content that we have seen okay? but in eh in graphical okay? here we have the same thing with eh API I'm going to skip this part because in essence it's the same the only thing that changes is the way we deal with the eh with the data because of the specificities of each eh bot okay? this is the main part of the presentation the solution how we do it I've been squeezing eh all that what's the attending time? oh, good one maximum okay so, I've been squeezing the rest of the previous part just to have time here and eh be able to describe a completely different way of understanding NLP from eh what we have seen for example with eh Alisha or what we've seen in other cases different still complementary okay? because remember that we do whatever we're going to do now and then we send it to a learning eh engine like Google or eh Microsoft okay so one key component in everything we're going to see is what we call the linguistic structure or what you call the parse tree okay those familiar with the Stanford parser will know that it's a like the skeleton of a sentence that you eh represent no? it sounds very natural to us it's what we did in the in grammar lessons no? at school remember? subject, verb, object that kind of thing so what we hear is this is a sentence this is a verb phrase this is the main verb or the head of the verb phrase this is a direct object instantiated by a noun phrase formed by the lights eh and this is the modifier to place in the room okay? so for us it makes sense to split things like this okay? because we speak English eh this is something completely out of the ordinary for a machine is comfortable handling bag of words as we were discussing before so tokenize everything put it in a nice row and eh and take care of that no? give me the frequency give me the you know format properties of words this is about content properties eh of words okay? everything like I said eh we're going to see three eh solutions eh two of them are in production and you can try them in the API and the other one eh is not yet so landing or going from that nice graphical representation to something more specific let me show you what are the basic ingredients of the solution the basic ingredients of the production of eh training tape so here on the left we have the input in this case eh this is an example of eh double intent which I didn't explain before double intent is a complex expression from something really simple which is asking two things at the same time you know okay? so leave me alone and leave the house like so that would be an expression of double intent typically it's so well known in the eh space of bots because for some reason bots have that limitation they if you ask two things they will answer nothing okay? and it's a typical classification eh problem you tend to classify or in the case of bots in a eh deterministic way so one single solution for every problem there are two solutions eh that's eh what they call double intent so from the sentence we produce as we have seen the bars three this is the same thing we have seen before but in a text mode okay? so these two the thing in black in the center is exactly the same thing as this okay? and then from the black eh box this is the idea black box because this is never shown in the final application then we go to these two solutions log rewriting sentence rewriting and natural language generation those are the three key blocks I'm going to explain them and then see how we change them to produce the result we have seen of a hundred percent improvement all right nobody lost so the first one is what we call intent detection and some other people call slot rewriting so for the sentence turn on the lights in the living room and the TV the software will produce two sets of slots because they are too intense one is actions are not object lights and place eh living room okay? very similar to what we have seen in the and then you will add there is another eh adderance or query which is parnond the Gb okay? two things in one this is what we call slots, no? and it makes sense, no? it looks like slots as you can see it's very similar to direct object modifier so it's not very very far from the linguistic representation that we have using as a page then there is another technology or solution that we call sentence rewriting okay? it means rewriting the sentence in a simplified way like for a robot which is the case okay? so we say turn on eh light in the living room or turn on the TV so every sentence will be reduced to a simplified form like when someone doesn't speak very well a second language we all have experience with that I used to be doing this if we have a friend that will make whatever someone is saying in a bar translated into something simple we are our chances of understanding grow significantly right? so this is the same thing for a robot and last and most important is once we have a sentence like this simplified we are non-TV and given the structure of the sentence we can start generating variations of that sentence and this is the key part this is probably everything will make sense at this point you can say turn on lights in the living room please can you turn on the lights in the living room can you turn on the lights in the room etc. so once you have the skeleton of the sentence now it's a matter of iterating you can say in the verb what can I change like turn on switch on enable whatever what can I say this is that living room how can I say that you go iterating through each of the components of the sentence you add all the pragmatic stuff which is the please and can you etc. and you have a reasonable description of all the different ways in which we can express the same command finally if you get the bot only the sentences it will not be very helpful because it's chunk of characters that doesn't make sense ok so we combine the sentence generation together with the slot generation ok so we'll feed the bot with something like turn on lights in kitchen ok with all the description of the slots for that particular bot in this case it's the type of representation that you use for Luis ok so I think this is the action of enabling that's something the user defines enabling or making something happen the thing that incarnates that action in the sentence is turn on the starting position is 0 and the ending position is 6 in characters ok nothing then the device is the lights and the place is the kitchen so you take those hundreds of sentences because you really generate it out of one single sentence 100 300 depending on the type of application and you generate them with this and believe it or not for those of you not familiar with bots when someone says I've written a bot they have done this by hand ok so even if the bot doesn't work feel respect for them they are going for a bot ok in the bot that we are using now and I want to go back to that there are like a hundred intents so like an enable action enable it being turn on like I said and anything you can enable the lights or the coffee machine or whatever and then all the places if you start combining those things this bot that we are going to see now has been trained with over 3 million sentences ok can you imagine anyone or even a group of people producing 3 million sentences in a consistent way it's impossible it doesn't make sense only typos could make up for 10% of error rate or something like that unless you are a professional typist which is not typically the case in our cases so training generating training data involves a lot of abilities that typically are not there simply so you can involve some engineering some knowledge of the language and basic timing so it's a very challenging it's a very complex task ok so this is the same kind of website but one thing first as you can imagine what we are seeing now are internal demos ok used for things like this for discussion for integration with platforms etc no when we do a website we do something fancy like this ok it's not that we don't know it's that we don't really give it much importance so here we are going to see again switch on the lines and another we are going to see what we have seen live ok all this have we have seen 3 representations ok as you can see we have here many more it's a like scale from the graphical representation ok here you have switch on the lights and the tv so it's a composed direct object there are two direct objects there are two things you have to turn on and then there is one place for the two objects ok you can have that represented in json the information too difficult to look at you can have that represented as a set of brackets so the sentence the verb phrase the title object the verb etc you can have it represented as a set of syntactic functions we have seen this in the black box remember we can we have seen this represented as a set of slots or the slot we are writing apart and we have seen it as a rewritten or simplified example ok like I said I'll send Alexi this url and you have here tons of examples ok of the things kind of things set the alarm at 4 ppm so everything that Google Home or Alexa should be doing that's what we have fed the things so feel free to try find mistakes which is always the goal of trying these demos and send them to us ok now in the case of Luis to try and make things a bit more specific we are approaching the end now how do we change all these pieces of information to get the final result like I said we have started with the intent definition this is something that the user will always do because we don't know what the purpose of the bot ok that we cannot automate that yet then out of that like turn on TV living room we will produce a query model a master sentence or a prototype sentence that will express the content of the intent with this master sentence we will generate all the variations and then we will add all the time so that's kind of the concatenation of pieces of knowledge make sense how do you generate this like the variations how do you go about generating these variations can you repeat the question ah he's saying how do we go about generating variations of these queries generating the variations of the query can you let me go to the ok like I said I'm going to say turn on the lights in the living room ok we are linguists we know that there is something called singular and something called plural ok that's the simplest so typically singular and plural tend to be synonyms tend to have the same meaning ok so if I take this sentence and I start going through it like in iterations the first thing that I will find is turn but we'll skip that for the example for the answer then I see the lights could be singular so some people could say turn on the light so I know they say synonymy between light and lights they can change they can be in the same position in the sentence and then I say turn on the light in the sentence so you can go through every note in this structure replacing something for something that will mean the same thing and how do you determine if it means the same thing? linguistic rules so you know I know if I change here like singular to plural typically nothing will change ok this is all clear on one single sentence but what if you have a compound what if you have a split in two sentences? like the typical thing is where somebody uses a pronoun referring to a previous sentence like the lights are on in the kitchen turn them off ok we're going to see a bit of that because that's usually where things collapse exactly knowing what the pronoun is ok he's referring to pronouns so when you say turn on the lights in the kitchen and then you say turn them off it's a bit difficult because the software has to give context ok you can say turn on the light turn on the TV and then turn it off you have to map multiple sentences exactly and they that's like you said that's typically where things collapse in context when you are using them like the people and it happens to us many times you say turn it off you say what? ok so only saying what is a really clever answer so not knowing what you don't know like I said so say I know I have to do this I'm a good bot I'll do it I didn't understand that's a very good answer ok and that that's a what you are asking now is the part we are struggling or working with right so you don't generate I don't it's not only a compound sentence but a set of sentences it's a what we don't do now is give the context of the whole conversation ok in a way we do we are taking the first steps we are going to see that now but that's very very complex ok so I was really close to the end so final slide what we have made remember the first slide where one sentence can be expressed in 10 different ways what we think we have made if you want to give one single idea after the meeting is that now the arrow goes into the two directions you can go from one sentence to many sentences or from many sentences to one a single sentence that's a contribution that we wanted to share with you today and if you have a bit of patience and you want to have some fun later with the URL like I said we have moved this all the way from internal purpose like the demo we have seen to a romantic language even you which is kind of like a ! a