Cognifest NYC 2017: Penina Kessler, Scala on the Brain: Building a Decision Making Agent with ML
[Music] so following that up I'm pnina Kessler I'm a software engineer at XIII and I work primarily on our response and decision-making system so I'm gonna talk about sort of the applications of data science and what our product actually does after we've resolved the natural language challenge but I'm also going to discuss how there's a lot of ambiguity in natural language in and of itself so xti is an artificial intelligence startup based here in New York City that makes a product that does one thing and exactly one thing and that thing is scheduled meetings for you so our products amy or andrew should you prefer I only use Andrew handle your calendar so in this example you see a meeting request is made someone asks hey can you meet up next week here's a range of times that I'm available and the way we think about scheduling a meeting at XII is that it's sort of a game of ping-pong you go back and forth about what times you're available what locations work for you anyone else who needs to be added to the meeting and finally if you can reach a consensus on an available time a good location everyone who needs to be there can make it you send out an invite so what our product does is when you CC it into the meeting it takes over that negotiation for you and ideally you end up with the invite on your calendar and no additional work on your part so what this sort of looks like from an email visit visualization Sun is someone say likes my talk here wants to meet up ICC and Amy or really Andrew and Andrew takes off he knows the hours that I'm available he knows where I like to get coffee and he tells you these are the times at work these are the times maybe that don't and you do any of these work for you and ideally we end up with coffee on I'm guessing this is November 7th and everything works out so the key points that I want you to take away from this talk is what Xai uses Scala for so we use Scala throughout our different applications in our data science in our decision-making engine and in our response system as well so in the part of the system that actually generates emails back to the user and why we feel it's important to use Scala versus another programming language is that Scala gives us a lot of flexibility but also safety to sound a little bit more human than the average chat bot so you have horizontal ai's like Siri and Alexa who can perform a task for you and sort of respond and then you have chat BOTS whose sole purpose is just to respond to your actions our agents are a little different they do one thing and because they only do one thing they're not gonna go back and forth with you if you ask them hey what's the weather like or tell me the score of the Vikings game and we are really proud of that we want to focus on one thing and getting that one thing right and in order to do that we sort of have to think a couple steps ahead of the user so from a game of ping-pong it goes to a little bit like a game of chess we're thinking to emails ahead okay if I say this then you'll take this action and then I can send out the invite so the three challenges that we think of when building our products are its ability to read so like a human we need to interpret natural language our agents work only over email we're thinking about slack integration it's coming and it works sort of to parse out the things that are relevant to the meeting from anything else in the email so the niceties the signature we really try to extract only the bits that we feel are important to scheduling the meeting the reasoning behind our brain application is pretty nuanced and that's a huge part of what I'll be discussing today but we really try to focus it on what action do we need to take to move the dialogue forward and our responding which strives to be as natural and as human-like as possible so the best compliment we actually get from people is oh I thought amy was human so the way I like to conceptualize our response system and this might just be because I have a background in cognitive psychology is it's sort of like our brain or it sort of mimics a lot of our brain like processes so here you see there's an incoming soccer ball and the goalie sees the stimulus and like our brain it processes it breaks it down into the parts that are important and reassembles that into the brain language it takes it from those rays of light and reassembles it in our brain as brain waves and then at that point where we realize hey there's a soccer ball coming at my face we have to act we have to make a decision because this has important implications at least in the real world for our survival and evolu everything like that thankfully Amy and Andrew only have two scheduled meetings but we really have to take in a lot of stimuli all at once and break them down into what's the most important bit and then think about okay what do I what do I and should I do next so the way Amy and Andrews brain works is actually pretty similar an email comes in this is the stimulus part and from there our data science predictors get to work on classifying the things that are important so any people mentions who seem like they're going to be relevant to the meeting any times any locations there's a lot of ambiguity there that we need to resolve is June meant who is mentioned in the body of this email is that a month or is that a person and then once we've extracted sort of everything we need from the body of the email we put it in context so we think about that coffee that was mentioned in terms of where does pnina like to take coffee where she told me before that she likes to take coffee or when she says next week does she really just mean Monday through Wednesday because Thursday and Friday are blocked off on her calendar and sort of taken stripping away that layer of social nicety that you put into an email when you're saying meeting up with someone you don't know that well and interpreting it into what actually works for that user and once we've computed what actually works we have to then tell people this is what we know about the meeting this is what we know works so this is a Sankey diagram which is a tool for data visualization so to go back to the ping-pong metaphor our goal at XIII is really to make that game of ping-pong as short as possible we want to shorten the length of our email thread so that while our users are hopefully experiencing a lot less pain in terms of sending e-mails back and forth we also want to shorten that dialogue with their guests we want to propose the best times for the guests and the best location so the top part of the diagram represents what we consider the ideal thread a meeting request comes in the agent proposes sometimes the guest confirms we propose a location the guest confirms we send out an invite but as you can see from the bottom there's a lot of entropy that could happen there's a lot of different paths that that dialog can take and we need to be prepared to predict what happens next and also predict the next best option so say the original times we proposed in that little red bar didn't actually work for the guests we need to think okay based on what you just told me about your availability what's the next best time that also works with this customer's calendar so our reading our NLU or natural language understanding takes similarly to the way our brain takes a sort of data and stimuli that come in from our senses and interprets it into brain language we take the human language and interpret it into our machine language our data models so we take we pick out the parts of the email that are actionable but we also attribute sentiment to that email if you're telling me hey does 3:00 p.m. work instead that agent needs to understand that not only is 3:00 p.m. referring to 3:00 p.m. next Monday but also that it's a time that's available for this user so there's a decent amount of sentiment analysis and ambiguity resolution that needs to occur to resolve it to a singular point in time and then pass the intention of that email to the brain making application the brain making up with the brain like application infers based on the context what action to take next so if in the middle of a dialogue we get the email hey just 3:00 p.m. work instead the action that the broker or the agent is supposed to take is actually very different based on what happened previously so if we propose 2 p.m. and that was the first email we ever sent you we said hey just 2 p.m. work and you come back and say hey does 3:00 p.m. work instead all the agent needs to do is check the customers calendar see if 3 p.m
is available and either confirm or decline that time but if say the last email that we sent to that participant was an ICS invite so we scheduled a meeting and we scheduled that meeting for 2 p.m. we then need to go and check that time against the users calendar but then also understand that hey does 3 p.m. work instead as an implicit request to reschedule the entire time of the meeting and based on that we then need to go out and tell all of the other participants of the meeting hey this other person can't make 2 p.m. is 3 p.m. good for you so a couple of reasons why we chose to use Scala to build in all of this decision-making the biggest one that I think any developer at X dot a I will tell you is type safety so Scala being such a strongly typed language allows us to sort of fact check our assumptions at compile time rather than in production if we're computing something and we're expecting a type of say reschedule response to come back and what we get back isn't that we'll know immediately that that part of the application is not working as expected we also do a lot of this type checking in our machine learning there's a type list open-source project on github if any of you want to check it out that actually does the same things for future generators so if our feature generator is using something it shouldn't we'll know that again at compile time rather than in production and finally we think of all of these different types of responses as things that need to be computed functionally so each type of response gets computed in its own independent part of the application and functional programming in Scala allows us to reason between these different responses and sort of interpret the hierarchy of responses in a type safe and also dynamic way so when we get to the time of sending a response we take those homemade homegrown types and send them to our response building application and as well as some functions from being able to parse them from that temporal model that resolved 3 p.m. - next Monday - back into a human readable format we have a wonderful AI interaction designer Diane who could not be here tonight who then has written a set of templates that get dynamically generated at runtime to decide what the messaging to our customer and their guests should look like so who gets told what at the moment where the emails need to be sent out and we pass a great deal of context from the decision making application to our templates so that there are almost many decision making applications in and of themselves they decide who gets chilled what based on what we previously told that person in the thread so we feel that it's very important to pass in a lot of context and sort of build in a lot of humanity to our agent the next slide will hopefully help you guys understand why [Music] so we have a lot of different kinds of validation on our response output in addition to our type safe checks we have integration and regression testing in Scala there's another open-source library called I think it's just called diff that helps you actually visualize the difference in your output so you can compare two versions of the application against one another and visualize the difference in all of the data structures and all of the responses that get passed on to the user our interaction designer has also produced both a white list and a black list of acceptable response combinations so it wouldn't really make sense to say hey can you meet next Tuesday at 3:00 if say the meeting was about to be canceled or if the last email we got and said hey this meeting is cancelled so based on that if something illogical were to occur in the application we would actually trim down the response until it met the whitelisted criteria and finally all of the context that we pass on through the application helps us we believe humanize the agent so if I asked Alexa in three different ways what the weather was like and she told me the exact same answer but I had asked her what's the weather like on Tuesday what's the weather like I'm Wednesday and what's the weather like on Thursday I would maybe think that she was broken and we sort of take the same approach to Amy and Andrew if we if you ask us a different question and we give you the same response we think that that's maybe not the best interaction for you as a human to take with a machine so the column on the left is what our response would look like if you proposed say three different times and we just came back every time saying I'm sorry that's not available the column on the oh sorry my left your right the column on my right your left is what it actually looks like in our templates where we have slightly tweaked the wording based on what we previously sent to the sender so if we've previously declined a time to you and said I'm sorry this customer is not available the next time we'll apologize for it we'll sort of take that extra step of knowing this must be a frustrating experience for you and if we've already had to do that twice we'll go the extra mile we'll say I'm so sorry so I hope that you guys have gotten a good picture of xai and the different applications of Scala in our system should you like what I've talked about and want to work with us we are hiring should you want to challenge me on anything I said that's my twitter handle I don't check it that often so that would be great you can also reach out to me at pnina human or xji as a company on twitter yes we use stanford but a data scientist would be able to tell you a lot more so we actually have a bunch of different sort of responses built in depending on what you say if you swear at Amy she will probably shut down and that she will forward your message on to the customer and say your guest has treated me rudely or your guest clearly does not want to work with me I'm not gonna schedule this meeting unless you tell me otherwise so but we also definitely try to mitigate that sort of frustration so that no one becomes that enraged with artificial intelligence so we're focused right now on scheduling smaller group meetings the number grows larger the more people who are customers so internally we all have accounts we all have our calendars hooked up so scheduling is a breeze but right now we're focused on solving small group meetings so the maximum number of people external to your company who you can schedule with if none of them are customers is four acha so know which is both good and bad I think that if you're rescheduling a meeting for the tenth time and you're sort of trying to blow someone off without blowing them off you really do want that AI touch of they're just gonna say the exact same thing to you no matter if they're rescheduling for the first time or the tenth time they're just gonna try to get the job done that said we do try to tell our users that it's really all in the way you queue in the agent so if you're saying I'm gonna blow you off here's my bot go talk to Andrew or Amy that's a much more frustrating experience on the guest side immediately whereas if you're saying hey I'm so sorry Amy's gonna handle it from here you've sort of cushioned the blow so yes and no it really depends on what you tell her so if you tell her hey schedule this meeting with Katherine but it's okay if Jo can't make it then she'll primarily negotiate with Katherine and Joe will get the invite and if Joe doesn't show up that's fine but she won't say Taylor unless you explicitly cue her to say hey this person is not essential to this meeting she'll treat everyone is equally essential so it's sort of equalizing and also maybe not nuanced in the same way like this person is indispensable so we try to treat all of our guests as indispensable if no it but we also have so we have a lot of logic built-in of like how many times will follow up with someone before will give up on the meeting but then also how many people have to respond before will cancel the meeting so if one person hasn't responded and two other people have and they've confirmed of time we're just going to go ahead and schedule without that third person but if three people haven't responded and one person has unless that person was explicitly marked as sort of the most important person in the meeting we would cancel that meeting but if you say like schedule a meeting with Katherine and sort of only Katherine is important to the meeting Amy and Andrew will pick up on that yeah [Music] you just tell it you need to reschedule so we tried very very hard not to double-booked you so Amy and Andrew will never schedule over another meeting that they've already booked for you and they'll never schedule over another external meeting on your calendar unless you explicitly tell them to schedule for that time and in that event they'll send you a little push back email to say hey I see you have this other meeting are you absolutely sure you want me to use this time yeah so that's actually what we call scheduling nirvana which is a term that Dennis Mortensen coins I can't take credit for it but all of that logic lies in the brain application and it's actually a very seamless experience because we don't have to negotiate those times with another participant we don't need to say ask your guests for their phone number we should already have that information on file yeah so like if you were a customer and you were like hey let's meet next week Amy wouldn't even reach out to me to propose time she'd just be like hey here's this meeting so they negotiate 24/7 365 which is maybe not the way humans operate I personally don't try to check my work email on the weekend but as long as the guests seem like willing participants so as long as they're responsive Amy and Andrew will work their hardest to drive that meeting to completion so they are very sort of singularly focused on let's get this meeting on the calendar yeah there's no limit to the amount of emails they'll send like you could schedule and reschedule an email for a year or a meeting for a year and they would still work it every single time yeah so I think we did some internal study and we found that the ideal number was eight including the invite so eight emails back and forth from the email initiating the meeting to the ICS email yeah so we do what we call supervised learning so should our predictors not be able to predict with a certain interval of confidence that June is a person or a month it'll get kicked back for human supervision I can't speak to the exact figure of people supervising but if Dennis said it Dennis probably knows better than I do in in real life sure do you want a free trial [Applause] [Music]