Text By the Bay 2015: Jean Sini, Keynote: Text(ing): The Rebirth

Recording: Text By the Bay 2015: Jean Sini, Keynote: Text(ing): The Rebirth

thanks a taxi good morning everyone Saturday morning 9am and already we're talking about hashtag collisions it sounds sounds a bit scary i hope breakfast was tasty we're gonna try and keep things light hearted today and not dwell too much on the specifics of fountain but i wanted to talk about texting and and why we're excited about it that so quick show of hands before we begin how many of you wake up in the morning really pumped about messaging that's what i felt okay so let's talk about that there's just so much love and hate about the text messaging right them it's an old-school technology it's it's mostly evocative of awkward you know best-case scenario autocorrect scenarios right and that's pretty much the only thing that seems to be relating to a natural language processing when it comes to texting right so in many ways texting actually sucks it's obviously ready mentally it's slow particularly on mobile phones which you know we are all seem to be glued to and and have now grown an appendage in the form of an iOS or Android device right it's just painstakingly slow to the point that a significant fraction of the work that goes into designing user experience on a mobile phone goes toward trying to find work around so you never have to type right what kind of sensors are we going to be able to use what kind of widget are we going to be able to put together can we use the camera to avoid texting right voice obviously and video as well have much better fruit boots they also not as tone death you can carry a lot of nonverbal signal through voice and video right we've probably all been in more than a few incidents where some kind of email thread escalated very quickly just because we were lacking in in tone perception right and yet text is absolutely ubiquitous we we count the number of text messages sent daily in the billions and and there's probably a reason for that obviously lots of qualities counterbalance the stuff we just talked about it's a asynchronous you know sender and receiver can process this not only at different times but at different speeds it completely de correlates what's going on between the two sides actually it's not just two sides it's you know ready for one-on-one but also one too many or many too many connections right from a cognitive standpoint it's actually much cheaper we perceive voice as being the more natural thing to do but it actually takes a lot of work to listen to speak to react to what's going on in real time while paying attention to your surroundings so that's actually not that cheap text is naturally you know archive aboard searchable obviously nowadays you see a lot of stuff going on around automatic translator transcription right but that's certainly available it's not just that widely deployed right whereas Texas you know something that you just check a box on on your RDS instance and you get all kinds of indexation and search capabilities right because of that while it's true that texting is certainly not the most efficient medium it's often perceived to be right and that's probably why you know we end up spending an hour texting back and forth with someone when we could have been picking up the phone and be done with it in like two minutes right there's this sense that with the images see that you get particularly if you interact with a system and not a human being you get instant feedback and we all know that we are hooked on these sort of feedback loops where there's constant stimulus happening back and forth and suddenly you feel like something's moving really fast that's great that's fun right so some have labeled texting the most comfortable you I write it allows all these characteristics we just described and essentially I perceived as superior even will be objectively not that great right in fact what's interesting is that texting remained you big remains ubiquitous even though gas often like an app for that right that's that's very much the case but we've seen a massive explosion in the number of apps that specialize in doing a specific thing much better than texting can write often though they require us to learn something so the interface that we all used to when it comes to texting is one that doesn't carry immediately over into just any app you pick up you have to learn how it works right so that's annoying it also is the case but ABS fall short when it falls short when it comes to expressiveness right what you get in specialization you lose in the ability to just be free form and just say anything right so I'm probably not telling you anything you have not felt given the number of messages but you've sent around text is here to stay right amazing staying power and what we're going to look at in a little bit more detail here is what's happening at the intersection of texting messaging and natural language processing right there's actually a long lineage of activity on that front I don't know how many of you have heard of alleys it's actually an acronym obviously the artificial linguistic internet computer entity and it came out in 1995 it turns out it won't the lead me apprised three times best a I at the time between 2001 and 2004 but it's actually sadly not Turing test ready and that's okay because there's a lot of goodness that can happen you know below that threshold right a lot of you probably remember advic came out and was sold to google i think in 2010 for about 50 m onion and what it lets you do was effectively text the system and the system would essentially try to figure out roughly what you were saying assign some tags to that and then route you to someone who could actually help you answer the question you were asking right today the bar has gotten much higher right with with the set of expectations around what it's like to interact with a system like that and about you know however cool it was probably wouldn't cut it today and certainly not Alice We certainly have come to expect that any system we interact with has some kind of mechanism in place to retain context to build memory about my preferences around what's going on so short-term long-term context and obviously given what we just talked about and the fact that it's so slow to text we expect you know our texting centric apps to leverage any kind of sensor we can write we also expected of integrations this phone you know but I carry in my pocket has hundreds of apps on it they all do something specific as we saw but it would be nice if we could all talk to each other and be glued together through text so what's going on now is actually fascinating for a number of reason we're going to look at the kind of players that have come on the scene in the last few months I kind of feel that Siri paved the way when it came out in 2007 out of si si everyone was kind of blown away by the number of integration points they had with third party services right and that's probably why you know 200 million dollars later we all have Siri on our phones right it's also the case but it was a bit disappointing i mean after the first moment of excitement we all came as well is she's kind of done right and you know the question has become you know what do we do next right the reason why this is becoming more and more relevant now is the offline and online words are very much blending right there's a lot of the tasks we perform today that can very much you know cross the line between offline and online so you turn to your phone you know increasingly often to do just about anything right the on-demand economy has essentially given rise to a service on multiple services to do just about anything you know uber and lyft homejoy in handy you know hotel tonight or all these services are there to essentially put a lot of power at your fingertips right there's also a massive explosion of data maybe it's cheap storage maybe it's cheap compute but there's certainly like a massive amount of data that's get that gets created every day and we're trying to exploit that and make use of it for you know to our own interest right we're also busier than ever and so it's natural that we are effectively trying to constantly do more stuff faster right we're competing on knowledge we're trying not to drown in to drown in the noise right and there's so much technical fabric to leverage when it comes to what's available for me out there as a consumer right that finding the way to like string all of this stuff together is becoming more and more crucial right even simple tasks can very quickly get complex right if I want to arrange a business trip to New York you know I'm gonna need some transportation to and from the airport I'm going to need a hotel room I'm gonna need some restaurants reservation and I have not even begun doing anything business related right so there's a new crop of players that's coming to the scene literally in the last couple of months and most of them have a messaging centric interface we're seeing to current schools of thoughts and I think they're going to end up blending so I think of them more as on a continuum on a spectrum rather than just two discrete points but you see a lot of human powered services and then you see separately today you see these BOTS right and I'll kind of walk you through some of what we've been looking at as interesting stuff I don't know how many of you are all of these services or any I've tried them all they're super interesting for very different reasons you can see that so this is the crop of stuff that's essentially human powered right there's a text interface in some cases where isn't any application for you to download if you take the the case of magic or go Butler these guys promise to get anything done for you what you do is you literally pick up your phone text magic or text go butler and someone materializes out of the ether on the other side and starts you know talking to you via text right and essentially get done for you right they have integration points with things like PayPal so that once we figure out exactly what this flight to New York is going to look like they can book it for you right operator is focusing on on retail and basically put you in touch with someone who can figure out exactly what kind of pair of jeans you need assist is all about making you feel like a local when you travel and you want to figure out like where to go and spend your evening right so night on the town sort of facilitated by a local expert Chloe kind of those similar things and as far as we're concerned fountain we are essentially on a mission to connect you on the spot with a specialist and expert who can help you and assist you answer any kind of question you're facing we came out about four weeks ago with our first vertical having to do with home and garden so it's saturday you you're here today but you're thinking about this project that you have to deal with at home could be a leaky faucet it could be something more ambitious maybe you're redesigning somebody's some kids room at home and you know you know that this may be it is five ten percent of expertise you're lacking and so fountain is there to essentially through a text centric interface at first understand what you're asking about and put you in touch with the exact right expert who kind of be available to help you right now right I won't dwell too much on fountain you guys can find it on the appstore but you know we're kind of like i would say at least right now part of this crop that is primarily putting you in touch with a human being through a text interface there's another side to this current crop and those guys are effectively human free right what they're trying to do is you know figure out a way to converse with you on a somewhat limited basis so that you can achieve you can complete a task right I won't go through all of them but essentially look at phones as an example tries to do with AI and NLP what Chloe and assist are doing with a human being right so it's it's a team out of New York we've been working on this for about two years and actually invite you guys to go and check it out it's pretty interesting they effectively able to figure out from freeform text the kind of restaurants you're interested in the size of party the neighborhood this kind of stuff right I want even I won't even go into what happens when you try to reach into the offline world so Alfred is super interesting it's literally a human being showing up to your house twice a week trying to coordinate your instacart deliveries and Google Express deliveries and your home Joe services so now with with you know crossed way into the future right but as far as online you know these are like some of the key players right now so what I want to focus on next is like what kind of technologies what kind of building blocks are there for us to build this kind of stuff and bring natural language processing to bear in an environment like that that's actually particularly suitable because there's literally no distinction in the ways we interact with a system versus a human being when it comes to text right like machines and humans and literally on equal footing in that sense it's the exact same interface right so first of all the arms race is on I don't know how closely you guys have been following what's gone on with deep link it was acquired for 500 million dollar by Google and even though they've been in the press for all sorts of things including for instance being able to teach their system how to play vintage video games from Atari and they want a price for that I bet that they are doing much more than that with their with MLP right another team that has to be called out is there is the viv team so these are actually the founders from Serie so the guys who spend out of si I in 2007 soul to apple launched this and essentially what their ambition is is to take things like that and actually get it done for you so there's for instance this example at the bottom where you know you're on your way to dinner at your brother's house and this is the kind of stuff that they are able to act on so what's going to happen there is not just the natural language processing component but they're going to be able to figure out what you're trying to accomplish with that and how to effectively look up your brother's address based on the fact that here he might be in your address book figure out where you are decide essentially what kind of detour you're willing to take so that you end up you know on the corridor that has wine stores it's going to be able to look up lasagna as a type of food and figure out what kind of one pairing works well with that it's going to be able to make a recommendation and then it's going to suggest a couple of wine stores along the way for you to do that so it's very much about agency we talked about how there's very little tolerance for lack of common sense in these services so I'll give you two examples this actually happened to me one of them interacting with a human being the other interacting with a machine they're actually very similar in sort of where we fall short I was looking for a restaurant in the Mission District and that's what I wrote and I was given a suggestion for something on Mission Street but at second so it's you know anyone who lives in San Francisco knows that it's not the same thing it's perfectly forgivable for a system not to know that but it's very annoying as well so the bar is very high the second system the second example sorry essentially came out of something where I'm like hey I want to go to Seattle i'm in san francisco and the system didn't know that Washington DC is on the other coast from Seattle Washington right and so little mistakes like that make it so like so quick to realize that you know you get frustrated and it's not worth it so to sort of remedy this situation and what we're really looking for of course is a magical experience right you guys have all heard of the third law of arthur c clarke right any sufficiently advanced technology is indistinguishable from magic clearly we're not yet there so suddenly like any good magician it's all about great misdirection right what kind of approach is can we take and what kind of tricks can you play so but it feels right we talked about the the hybrid approach I won't belabor that I just wanted to point out these guys it's a team out of Rochester university they are working on a project called chorus and what they do is they essentially have not one but a group of humans who are helping you and appear to you the consumer as one entity and so what happens is web said they've set up a system that has a consensus building mechanism baked in a bunch of people are multi tasking multiple conversations at once they they all essentially submit a suggested answer to your question and they start voting on who's given the best answer and next thing you know about ninety-five percent of the time the subject the consumer in question is convinced that Ava's speaking with one percent but BB are getting a lot done because you know you have a bunch of you know sort of specialized domain experts who can handle different parts of the conversation from making a great recommendation to getting you there for instance right one of the most sort of broadly used trick is to essentially constrain the input right one of the one of the jokes you know that comes to mind is how you know everyone feels like that's such a great artist when they are using Instagram filters right when really constrained your input so you essentially end up picking between the blue filter and the red filter but your fins so creative right so it's all about misdirection is like how do you you know move people along while giving them a sense that they are being creative in fact you know you lost freeform expressiveness right of course multiple options but something we do at fountain right sometimes we're not sure about what you said and we have a couple of options in mind that could be a good fit and instead of like sending you down a rabbit hole we actually asked confirmation early on right something that the guys that thumbtack are doing that also super interesting is essentially they put together through a combination of you know hand crafting and learning a number of questionnaires but they put you through the adaptive questionnaire so that you know you start with an open-ended question or sentence they try to categorize what you said but then before they hand you off to someone who's going to help you let's say with a project at home they actually put you through this five to six to seven questions and by the time you're done you've grown smarter because we actually know exactly what they should be asking so that the question becomes better qualified right and then of course that's what we do at fountain for instance we harvest specialized knowledge so I mentioned earlier the example that the viv guys have given around how you know when they'll be able to give you great recommendations for one what we care about that fountain is putting you in touch with the exact right person so Alexi put us on the spot here when you mentioned that we started our life as a company by scraping a bunch of skills and job postings on linkedin we actually went to a bunch of websites and what we've acquired is essentially a scale graph for what we think is is the entirety of human skills out there seventy thousand of those also and more importantly beyond the skills alone the way they connect right so we able to know that someone who lists on their resumes something like jdbc is probably a good all-around Java programmer right and SBT and scholar are fairly closely related as well what that allows us to do in practice is expand the search of experts that we want to put you in touch with when you come in with a question so you know ideally when you come in with a question around wanting to remodel your your guest room with you know a theme that's about mid century modern we look for that exact interior designer but we don't want to get you stuck because we just so happen to have someone who is a slightly different style of interior designer so we broaden the scope and the radius very quickly we have essentially algorithm that gives us interesting data around the distance between skills right and how much we are willing to decay the accuracy of the match before we put you in touch with somebody right of course one of the things that we've gotten used to because we've all learned to speak Google is instant feedback right you go you make a search if you don't like the results you see you know what to do you quickly change your query one or two words right and so that's something that you don't want to lose with text you want to be able to quickly fix your mistake at no cost and get a different set of results right so that's something that's baked into a lot of these user experiences as well and then of course we talked about long and short-term context and memory very important for for people to get a sense that you know the magic is working and that the system understands and knows them so build a set of preferences so if the lasagna wine pairing suggestion was a hit you know I'm going to remember that you like cabs if not I should probably keep that of the list for next time right now that was all about misdirection and let's talk about the tools that we can actually bring to bear when it comes to trying to put together these agents that are going to get us help when it comes to executing tasks right quite frankly where we spend the most of our time today at fountain is you know classic classification problems right so we've gone from linear classifiers that are very much an exercise in supervised learning where we've we've handcrafted the features over time essentially trying to understand what matters most to the problem at hand for us right we are starting to dabble into more interesting sort of approaches that try to minimize the supervisor aspect of the learning I'll go a little bit more into what that means in a second when it comes to essentially a more open-ended clustering exercise you know you're starting to see a lot of advances there where you progress is being made for instance using various technologies such as we talked about mixture models for instance right and then you go into you know sort of like the the best words of the day right so neural networks frankly today most of the stuff that's happening there is happening on on GPUs it's happening on CUDA is happening with torch right and so if you want to help us make this run on the JVM on the GPU come come and help us but feels like it's a very early stage problem but there's a lot of interesting stuff so when it comes to recurrent neural networks right that's something that's now starting to be used in very interesting ways for instance when it comes to figure out what a word you haven't seen before is about right so it's about analyzing the morphology of a word right so being able to look at the word and break it down into its parts and understand sort of what you know what class of concept it designates better on its sole morphology right so people are now using recurring neural networks to try and figure that out in a way that's unsupervised right when it comes to the convolutional neural networks right we're starting to see a transition from primarily image processing and image recognition tasks right to natural language processing tasks right there's one team from Oxford that has essentially put together dynamic convolutional network that is able to compete in terms of performance with a classifier has been trained with with essentially no supervision they are able to achieve similar levels of accuracy on a number of similar tasks right now there's another team I think a little bit earlier from NEC and Google that's achieved like very similar results as well so again keeping in mind the context of today's conference we're talking about text and we're talking about skala a lot right but certainly what we use there's a lot of work still to be done to make that stuff available directly on on Skala a lot of that stuff again cuda and torch right when it comes particularly with the very compute intense tasks of convolution for his neural networks the last thing I wanted to touch on was beyond just the pure NLP component of what's going on I don't know how many of you have looked at this BDI system so belief desire intention right that's effectively what's powered Siri essentially a model to represent intention in rational agents right so the ability to predict you know intention and to tackle problems and tasks by having a model for how to weigh conflicting options keeping into account state which is effectively the set of beliefs that are held in put as in desires and intentions that I effectively plans right so if you think about a a BDI system it breaks down you know the tasks of reasoning into two subsets you know deliberation and effectively the what we're trying to do from the means to ends reasoning about how we're going to do that right and what what a BDI system effectively does is is in a very modular fashion because these goals are effectively trees that go from a you know a goal to a sub plan that is in self you know recursively a goal with a set plan and so on and so forth they're able to build a lot of complexity but in a way that is iterative and progressive okay so you start with a very simple system and then you build more wisdom into it from a logic standpoint you know the gold successive at least one of the plans succeed and when the plan succeeds all of its steps taxes and so you start traversing this graph this way when it comes to Skala which is close to what we care about because the entire stack is built is built on Skyler BDI systems are very interesting because there are some libraries out there I'm just mentioning one here that leverage the fact that actors systems you know acha obviously are very sort of closely matched to the kind of of tasks that the BDI systems do so a very interesting area of focus and interest to us you know just wanted to you know have a quick shout out for the tools that we are looking at we're very much fans of the Stanford NLP library at fountain and we're also looking at things like like factory which is something I think you guys if you've been to some of the talks at the the scala meetups you you may have heard the work of the the factory the main contributor to the to the library datum box and deep learning also to interesting skala or java ready libraries that's what i wanted to cover sort of like the state of messaging the way we look at it today so thank you and if you have any questions I'd love to hear them or coffee of course so given all this entrance to the market rent amazon obviously 800-pound gorilla where do you see this cone what do you envision is the kind of the best value for the consumer and how you I think for a you know at least a couple of years you're going to see most of the the more palatable and most convincing solutions be hybrid solutions right I think I'm definitely looking at viv very closely I think these guys are definitely hardcore there perhaps like you know one of the more credible teams out there when it comes to like really hard you know AI so you're going to see you know the two ends of that spectrum kind of come close together and what do you think do we need to improve in nov what are the weak points like you mentioned no washington state in seattle washington so a bit like this is world knowledge right so do we need to focus on building better world knowledge or do we need to bring in kind of cut that you can deny grammars and parsers what do you think the best prove it is so as always with these things i think of it as something that you know carries like diminishing returns and yet you know even though we get closer to the asymptotic a point it still gets frustrating so it's still worth the effort right and that's very much something that i think has to do with to your point you know better recognition for like specific entities and whatnot i think the other side of that is what I touched on when it comes to agency right and the fact that we are able to start plugging into modules that know how to perform the task so we looked at the example around like getting me on a flight to Paris tomorrow and returning next weekend but that presupposes a lot of interaction with with modules that can that can do something also another question have yesterday the panel a General Court said that basically if you were doing ml now basically you should drop everything and do deep learning right the glory is gonna kill everything if you're not doing deblurring you're doing it wrong so I wonder if you subscribe to this what's your take that's why I was saying that we were hiring and we definitely want help making the transition yeah thank you so I see a lot of movement around interfaces that are text centric as in you you literally like you type but they're not necessarily outside of an app there sometimes in a nap and you sort of end up with the best of both worlds where a lot of the responses you get come in the form of not just like a piece of plain text but something that's let's say a card where you see like a restaurant review or you see a button to call an uber for your things like that right so suddenly you know you sort of multiply and amplify the throughput and sort of like the richness of the interface right and again it's more of a hybrid take on that so lots of freeform ability through texting but also augmented by a specialized rich you I when when it matters so I don't I don't imagine that just text as in there's no app is necessarily the only way mention our founding are 700,000 skills oh I mean so what we effectively have is is there so 70,000 skills a lot of em have synonyms right some of our work essentially is about disambiguation there's a lot of words that you can imagine have different meanings so the classic example of course you know sea bass versus the bass guitar right so acquiring the skill when when we look at someone one of the things we're trying to do at fountain is when we on board one of our experts we try to minimize the amount of work they have to do and so we try to infer their skills by looking at their you know linkedin profile and things like that so being able to represent these skills in our system with not just you know the word that we attach to them the label but also the context for it what kind of general category they fall into is important so we can derive general context from let's say the resume we're looking at and assign the correct meaning for you know java is it coffee or is it programming right so we essentially maintain this graph in the form of a large you know matrix that's sparsely populated and represents the connections between these graphs right and the weights in terms of proximity in meaning yep oh yeah I definitely think when I when I mentioned in passing that the arms race was on I don't think Apple is going to sit on their laurels after the the serie acquisition particular unless unless they just by viv again I don't think that's the ambition for them though I think babe is very much about like being ubiquitous this time around Google now you know and the fact that they picked up deep mine definitely feels like they're gearing up to to achieve the same kind of of service thanks guys