Grant Ingersoll, Question Answering with Solr/Lucene, SF Text @VigLink 20150210

Grant Ingersoll, Question Answering with Solr/Lucene, SF Text @VigLink 20150210

Recording: Grant Ingersoll, Question Answering with Solr/Lucene, SF Text @VigLink 20150210

i'm going to do some demos and stuff so holding a mic doesn't really work for me but i've been told i'm not the quietest of person so hopefully i can just project loud enough uh i missed the show of hands how many people actually like do real nlp in production like as their day job and how many people here are just learning this stuff hey text it's all the rage you know we speak it we you know language that's all good stuff so okay so when when people were saying things like stemming and morphology and all that kind of stuff like we glazed over right some of you did some of you didn't okay so my name is grant ingersoll i'm the cto and co-founder of lucid works as alexis said you know my day job my day hat is go build out business around lucine and solar and take we invest a lot in the open source and then we build products on top of that i'm also the co-author actually one of alexi's uh colleagues at penn when he was doing his phd there of a book called taming text all of those people who raised their hands of hey this tech stuff sounds interesting but i don't know what the heck it is that book is for you it's an engineer's guide to nlp and search it doesn't get into all the math it just gets into hey what are what are the concepts behind this and then here's some code examples that show how to do this it's primarily driven off of open source examples so you may or may not find it useful in your journey with this stuff a little bit of a an ad for our company uh real quick we put on the largest lucinas solar conference in in the world i think last year we had uh over 600 people added you know companies from small mom-and-pop startups all the way up to apple and salesforce and and target and big name companies there so this year it's uh in october in austin texas so if you fancy learning more about uh leucine and solar you can definitely come to it's a very technical conference there's not a lot of commercials going on there we very much drive it towards helping the community uh understand lucine and solar so uh with that i'm going to actually uh you know as i was talking with alexi there's a couple of different ways i could have gone i could do a solar talk of hey here's what's going on in solar actually if you go to our website i just did a webinar what's the latest in in solar so if you're interested in that just go download the webinar instead what i'm going to do is actually talk a bit more about my book and talk about text and some ideas around how you can process this and in fact how you can build out with a lot of great open source capabilities a pretty decent question answering system so in the book i kind of walked through all of these different parts of text processing ultimately leading up to building a very simple question answering system so that's kind of the premise of this talk and i'll talk about some of the things along the way i'm going to kind of wear two hats i'm going to talk a little bit about lucy works and some of the things we're doing with question answering and nlp and stuff like that but then also uh put on the the let's go sell a book hat and talk some about that so before we get started i always like to put out a big caveat right uh the text world you know uh alexis said you know it's often called unstructured data i think that's kind of a it's one of my pet peeves of mine is that text is actually very highly structured right we're just not very good yet we're getting better at helping computers understand how to treat text in fact i often think the reason why text is called unstructured is because all the database salesmen back in the 80s knew that they couldn't handle text and so they said oh well this is unstructured thing because it's almost like a derogatory uh thing we're structured well that sounds great so we can sell structured we can't sell on structure whereas nowadays i think we all know that and recognize that we live in this fuzzy world right where there's a lot of ambiguity and stuff like that and in fact i think uh you know one of my favorite movies in book series is lord of the rings so thinking about how does text relate to that i think you often when you're coming across text and the the big link guys did a great setup for this you often run into all different kinds of characters when it comes to text there's a whole lot of text out there that's just you know evil there's people who write bad things there's things like sarcasm there's people who are mad et cetera right and so they take all this stuff and they're working against you there's actually whole tracks in academia around how to deal with adverse adversarial situations right where you know people are actually being bad actors and of course there's all kinds of things like jokers right you've got people who are sarcastic people who are telling jokes all of those kinds of things how do you deal with that kind of stuff in text right again you know one of my favorite sayings is there's a reason why the saying is the numbers don't lie right nobody ever says the text doesn't lie text lies all the time people lie all the time right so you've got all of these different characters coming in and hopefully at the end of the day what you come out of this is you get to be the hero right uh one other one that's pretty interesting and i think the big link uh people got this too is there's a whole lot of unicorns in text just think about the fact that there are new products that come out every single day you've never seen this thing before what are you gonna do about it i don't know what that ipod thing is or iphone or whatever it was until that actually was announced and came out and and you had to deal with all the different spelling variations and all of that so there's all this really unique stuff that's happening in text that you have to be ready to deal with and you know what it's really hard and so at the end of the day you kind of got to keep in mind that you often are just striving for good enough here or better than what an individual person could do or better than what you know the previous state of the art was there um the other thing i like to kind of seed you with uh especially as you're on this journey of getting into text is that take it all with a grain of salt all right um you're going you know a lot of this nlp stuff and and doing all of these kinds of things that they talked about all the fancy algorithms those are really great your day to day isn't going to look like that you're going to spend a lot of time on pre-processing they talked about you what you had 20 different data sources right let me guess they're all in the exact same format with all the right tags and all of those kinds of things right yep okay so you know you know what this is about right content comes in right if you don't believe me just go try writing a web crawler okay this big internet the whole entire world is on you think we would all have a standard way of dealing with all this kind of stuff guess what we don't and so you have to deal with all this crap up front and then you're going to spend a lot of time on feature selection right it's often the case that these fancy algorithms they may or may not work simply by the fact that you didn't do good feature selection what do i mean by that that might be something like choosing which words in a sentence you want to keep as part of your uh machine learning algorithm so when he was showing you that big table gabor i think it was that big table that had you know first character and he had that you know all the matrices and what they're contributing that's your feature selection process there's a lot of good tools out there that help you with this other things is you're going to do a whole lot of staring at text yourself and one of these days this light bulb is going to go off and you're going to be like oh i get it now and so you're going to dig in and then things like validation content extraction etc etc long story short start simple try out the tried and true approaches chances of you actually innovating on something really new and unique in this maybe not so good right off the bat so start with what's easy and get going from there you'd be surprised how far you can get off of just kind of common sense and open source and then iterate from there as you learn your domain more as you invest more resources all of those kinds of things so that's kind of my caveat it's a lot of fun it's a great field to be in it's also really hard and i you know firmly believe that as an engineer you want to work on hard problems so it's a great space to be in and i think speaks to a lot to the the popularity of this group and and some something that you can grow on here okay so with a little warning up front um one of the things i kind of wanted to dig into is you know how many people watched watson beat you know ken jennings and does anybody remember the other guy's name no that's a good question to ask of a system right so question answering is essentially i think what many of us in the search field think of as the future of search you know it's it's great to get 10 blue links and you have to go click on them and then go read through some article and try to figure out whether it answers your information need or not i think we all dream of the day when you can just simply ask or you can type in your question and comes back with just the right amount of information right and so if you ask the question who is the president of the united states right now you should get you know barack obama and ideally you could also ask the kind of question of like what were the causes of the united states civil war and you would get back the right amount of information right that's what you know kind of the the ultimate dream is of question answering i think the the really cool thing is we are finally living in an age where this stuff is becoming more and more practical i built my first question answering system in uh when was that uh 2000 people have been working on it for years before that you're finally starting to see with a lot of these big data techniques and much more uh capable open source projects as well as you know kind of standing on the shoulders of giants that we can do a lot more than we ever have the fact that you can pick up your phone and say hey you know what's the stock price for whatever company you're interested in is a really cool thing and there's a lot of interesting processing that goes into that i'm going to show you a little demo of what looks like in the book context and then i'll talk through kind of what are some of the steps behind that just to set the stage a little bit the book one is very toy in the sense that it is a fact based question answering system it's not doing a lot of reasoning the way that works is you take a a version of truth a data source if you will and that acts as kind of the the baseline which you're going to go and try to determine what an answer is from and then we're going to do a bunch of processing on that to hopefully come up with the answer to a particular question so the causes of the civil war stuff i'm not going to do that here but hopefully we can answer some interesting questions about facts that are assumed to be true in the index or in the database excuse me the database that we have here so good lord willing live demos are always a fun thing one of the things they just do is in the book is it walk you through kind of starting with search how to build up more interesting things like clustering and classification and tagging and all of these concepts and then finally towards the end of the book we end with all right let's tie all of these things into a question answering system so what we do there is we take solar we index wikipedia into it okay and then we've added in some plug-ins into solar that do things like question parsing and all this kind of stuff and at the end of the day then what i've got is a very simple interface built on solar that is doing a search when uh when it doesn't know how to answer a question but then when you give it a question it will try and answer that so that's what you're seeing here i've only indexed uh there's it looks like there's about 8 600 or so actually actual documents in this and so the only things i can answer are those things that exist in that index i'm not bringing any other outside resources in et cetera which of course in a real system you would do so i can come in here and i happen to know a few things like here say i'm a chemist i might ask the question what is trimethylbenzene and you know obviously that's a somewhat canned question and i can do the search behind the scenes i've gone to solar and i said give me some candidate passages that that mentioned trimethylbenzene and then do some scoring things around that right just like they were talking about they go to elastic to get their candidate offers right then they funnel those down same kind of process that you're gonna do here in a search system or in a question answering system and so if i scroll in here you can see for instance what we then do is extract out of this much larger document uh this passage that says one two four trimethylbenzene is a colorless liquid with chemical formula ch and then there's some other text in there and i'll talk about some of that later but you can see i've also actually get a few other candidate things these are the ones that actually came out of the scoring process the interesting thing here is i also show you the underlying document that provided these answers and you know it can go and ask other questions of this like who is 10 minute warning and here it does a 10 minute warning is a hardcore punk band from seattle washington you can see again i've got some extra stuff here this kind of speaks to starts to speak to some of the problems that come in around asking these questions if i asked uh you know something like um you know who is big link you know it's going to come back with nothing because that's not in my index right so just to kind of set the stage there and set expectations right but so you can see that we're we're extracting content out of it it's actually using this in this particular case it's using that full document to get those particular answers so far so good makes sense yep so the power set that was bought by microsoft was attempting somewhat harder problems like asking questions who did the 49ers play in 1994. they're looking for semantic relations where this is largely look up by name or proper name right which is fairly straightforward so can you do anything with semantic relations yeah of course and and you know we don't i don't get into we cover some of that in the book and i'll talk about some of the things that we do in the book context but of course in a real world system yes you're going to bring in lots of other things be beyond just the basic passage yeah well scaling this stuff is always the one of the hard parts right so if you're talking about things which are fairly straightforward yeah i mean i think you're starting to see more and more people be capable of that i've worked on systems just in i won't say it's solved by any means i know i'm not saying that right so uh i think you're starting to see more and more people look at relationships between entities in the text and try to do reasoning on those kinds of things i don't think it's a solved problem by any stretch right so you know and it's certainly one of those things that's really hard to scale right um i'm not aware of anybody offhand that i know uh uh who's doing it you know who's doing it really well but you know they may be out there i just may not be aware of them so um so that's kind of the the basis here of this um of what we're getting at so let's dig in a little bit you know to the gentleman's point here this is obviously a very simple query workflow based off of looking at facts like around named entities or other kinds of things everybody familiar with named entities people places locations all that kind of stuff so if you think about like so what we do in the book one of the things we do is very simple qa workflow one we've got content coming in so we've got a essentially you remember all that pre-processing stuff that we talked about right so some of the typical things that you do is first you know you've got to recognize a sentence right you think oh well what's so hard about that you know a five-year-old can do it you know computers these days are actually quite good at it but you'd be surprised how often they still are wrong you know that's usually typically pretty high uh capabilities there and from there kind of that that kind of degrades down the next step we do is as we're indexing we just take and we've got we're using an open source project called opennlp which is this is a decent library it's not the best library out there but it kind of can get you going uh pretty quickly on this kind of stuff and they've got general models for doing things like named entity recognition and sentence detection and all of that kind of stuff so as we're index as this content comes in we break it up on sentences we take each sentence feed that to the named entity detection and we mark up the content of this is a named entity this isn't etc and then we take and we send that into your search engine and and then that makes it available for us to query on the question side this is really where i think you know a lot of a lot of work is obviously can be done here in terms of enhancing that content as it's going in same thing on the question the question comes in the first thing we've got to do is parse that you know such that we're you know we tokenize it we split it up into chunks etc and then we need to do this thing called determining the answer type right and if you think about it there's a lot of different kinds of answer types out there this is essentially a classification problem is this person asking for or is this question about uh a person is it about a location is it about you know whatever you know whatever category you want in terms of the uh the data that you have and then out of that we need to then generate a query against our search system right and then from that get the search that gives us back kind of a candidate set of results and then we've got to go and do ranking of those if you go read like for instance the ibm watson paper and stuff like that they'll tell you about they spend a lot of time on answer types and they talk about you know somewhat about their ranking problems and they talk about how they bring in different uh authoritative data sources and all of this kind of stuff it's a really nice ro uh read but you know essentially you still can simplify it uh up to some of you know essentially something that looks like that obviously in the real world you would dig in a lot more but for those people who are getting started this is a a good way of looking at it etc digging in then you kind of start to think about what are the building blocks for doing this text processing stuff you know a lot of great stuff from the big link uh people around all the the deep learning and all this uh you know very advanced stuff there's a lot of just fundamental things as well that people have been doing for a lot of years around this how many people you know remember their high school grammar class how many people remember it fondly anybody so all that stuff gets to come back to you so things like part of speech tagging parsing you know breaking down sentences into things like noun phrases verb phrases adverb phrases you know part of speeches like adjectives and adverbs and all that stuff that you probably wish you had forgotten uh you know is going to come back into play here on this stuff although interesting enough you know the deep learning stuff maybe doesn't need as much of this anymore the you know at the end of the day these things are potential features for you they often go into building nlp systems i walk through a lot of this kind of just foundational stuff in the book and so you may find that useful in your journey here you know but essentially this is building out kind of some of these things around how do we parse and how do we detect excuse me detect sentences how do we recognize when something's a named entity and all of that kind of stuff well first of course it's got to be a noun and all that kind of stuff right so those are kind of your building blocks and again i'm just kind of giving you a high level view of this you know obviously you could go spend your entire life just focused on one particular area of this you could get a phd et cetera i think we covered nar already so i won't go into too much of that but you know it often comes into these systems of you know a lot of times these systems are really focused on how do i find the named entities or the interesting things in it and then what's all of the surrounding stuff around those such that i can start to make uh i can start to reason about it and do other interesting things from there we take what i do is or what we do is take and build out a query parser so that query parser needs to be able to take in a natural language query so for instance you've ever used lucine or solar or elasticsearch or endeca or fast or whatever search engine much like with a database this is all they usually give you a very structured query language where you're doing boolean logic you've got fielded searches and things like that what your users want however of course is just interact naturally with it using human languages right so what we do in is build out a query parser that takes in the natural language query we classify it into i think in the book we use just four different answer types uh they actually look like uh the the training set that we use is a pretty simple one uh that looks kind of like this let me i can scroll in on that a little bit where essentially i've got a category field so essentially this is the label that i'm applying to this question and then an example of a question so i know you probably can't read that in the back but you know for instance this particular one is looking for a person or says this this particular uh item is is a person and the question is which boxer representing canada won the 1988 super olympic super heavyweight gold medal defeating riddick beau of the usa in the final right so that's an example of a question that is looking for a person and there's a bunch of different ones in here this is essentially that same annotation problem that they were talking about you're going to spend a lot of time doing that kind of stuff that's the fun stuff uh you're the not so fun stuff but you know it's it's often necessary for especially for getting started uh these days so from that we generate a solar query underneath the hood one of the things that's interesting unlike keyword search is you actually have to know where in the document this matched if you're familiar with leucine and solar at a very low level this is called a span query or a positional query so what we do is we take that answer type that we got and we take the original text of the query drop some words off of it and kind of massage it a little bit we bring those together in a span query which says i want to find these words with this answer type within a certain window okay and that now forms my candidate system right and from that candidate system we can then uh go and look at how to score that and i'll cover that in just a minute real quickly on the answer type classification it's essentially like i said a classification problem you can see there's a bunch of different examples of that the most popular things that most people are looking for person location organization time etc so all of that's covered in there we also show you how to do that that problem then of scoring all of this stuff obviously there's much more sophisticated approaches out there this is a very simple approach i think at t uh published on this back uh in in the mid 90s works okay you know you can get kind of what you saw in the the book obviously you know there's a lot more that we can do off of it but essentially we take and we find exactly where we match and then you know what we're doing is looking at the context around that we want to make sure that the answer type because remember at indexing time we we indexed the the kind of the tags that said this particular thing is a noun or this particular thing is a named entity etc so when we go to search we want to make sure that that named entity show is in the matching window first and foremost right and then we want to do some analysis to try to figure out what is the actual answer out of that system right so it's pretty simple um and and works okay you know you're not gonna you're not gonna blow anybody away with the quality of this system but you know for somebody who's getting started and trying to learn from all of this uh you know it's a it's a good uh example of how to go about doing this kind of stuff so with that then i'm going to shift gears a little bit and talk about some of the things that we're doing around this kind of building off of taking this classification stuff and and and all of that because i think when you think about search again if we go back to this assumption that people want to interact naturally with the computer system via natural language you need to then be able to turn that stuff into you know queries to your system and then retrieve all of that stuff out of it so at lucidworks we build a product called fusion which sits on top of solar kind of three core foundations to it one is you know this kind of notion of signals which is you know kind of how can we gather information from the way users interact with content and then feed that back into the system such that we have more insightful access to the data so they talked a lot about click stream data for instance that's an example of a signal to us we can take click stream data and automatically churn that thing and turn that into more enhanced search results as well as things like recommendation engines etc other examples of of a signal might be the fact that you and i emailed or we're emailing all the time and we're talking about solar together or we're talking about sports together another example might be a twitter feed where you're sitting there looking at hashtags that are mentioned and using that as as uh accounts on how interested people are we take all of that data shove it into another index and solar and then as we're interacting with the data we can then leverage that for providing essentially popularity uh based enhancement of the original search results so for instance you can take your core e-commerce catalog which is already running on solar you don't have to touch that index all these signals right alongside of it hit fusion it will go and get the weights for that particular query to figure out what people care about and then search against your core catalog to get back the results that match on content as well as match on popularity or on the signals for instance as an example so with that then one of the things that we often have people ask us for is how do i get better insight into what a user's intent is right and that often comes down to how do i classify the user's question into something more interesting so i'm going to show you this this is a little prototype i've been working on it isn't in production yet but you know we're we're getting there with it uh and so with that i'll just shift over here and uh come into fusion so like i said you know one of the cornerstones was the single stuff the second part of it is go and deploy more solar ask more interesting questions of solar you already have solar deployed wouldn't it be great if if your system was just simplified you could just deploy more solar and ask more interesting questions of it you already know how to do that the last part of it is we really just provide a lot of tools around it in terms of managing it pipe things like pipelines and relevancy work relevancy tools and and system dashboards and all that kind of stuff so i'm going to kind of walk you through a little bit of two things one of the things that we have built in here is a search system or a search ui where you can come in and just have right off the bat you know enterprise search within your system and what i've done is we've actually taken a bunch of tax questions so this is like people asking hey how do i take a home office deduction as an employee you know i've actually indexed that we actually have some categorization on that on that data right and so you can see i can do some questions here and you know interact with this data and i can say things like home office deduction and do a search here and all that stuff and so that's all well and good pretty straightforward vanilla search uh going on i'm just going to open up another tab here and now i'm going to go into the admin capabilities here you can see for instance we give you things like managing your data collections you can run all this multi-tenanted you get information about what's going on so here's my questions index along that lines we then have what are called pipelines pipelines are essentially a lightweight etl tool that you can scale out to do kind of pre-processing remember all that pre-processing crap that i talked about earlier that is where you're going to spend all your time we try to make that easier here we have both index and queries queries side pipelines so one of the things we do is we actually have uh some training and nlp capabilities in here so you can come in here and for instance i'm gonna this is building out a model of taking in that question and categorization stuff and building out a model automatically for you off of that annotated data right so that's an example of an index pipeline i have a second one here that then can take data as it's coming in you so you remember that diagram i showed you around qa where as the content comes in we've got to do sentence detection and then add the named entities and all of that kind of stuff and then ultimately index it this is a lightweight version of that i'm doing classification of that data as it's coming in and then i'm indexing it to solar for instance if i wanted to do named entity recognition i could simply just add a new stage here and fill in my rules and and my model for doing all that kind of stuff and voila i've got named entity on my content going in there and if you want to train your own models and all that you can do that right so that's on the indexing side where it gets more interesting is on the query side i can now take as requests are coming in and modify them as they're going and i can build up lots of these pipelines i can do a b testing and all that kind of stuff with it so out of the box i've got a pipeline set up here for this questions collection it's doing things like saying hey you know which search fields do you want to uh search on it's doing things like what the search fields do you want to facet on you can put in logging stages and ultimately you can send queries to solar and so all of this is drag and drop live you can arrange these as you see fit such that if you don't like that particular order you can change them around and my demo is not happy with me right at this moment and you can see it all automatically updates and all that kind of stuff let me just make sure i've got my order here correct so one of the things i wanted to do for this demo is say okay if a question comes in i want to actually try to understand what this user's interest in so we took this data set we trained on the question in the category and now i'm going to at query time leverage that to maybe direct the user when they ask specific kinds of questions to a landing page where it's like this page talks specifically about your question as opposed to giving them back those 10 blue links right so the way i do this is at the query time i've got a stage that i'm going to pop in here called just the nlp classifier that takes that model that i trained at indexing time and so when the query comes in it's going to look at that query and say oh this query is about home office deductions it's going to add that into the request and then downstream from that i've got a landing page set up that says whenever you see a category of home office deductions here let me just make that a little bit bigger whenever you see a query that matches on where the category is home office deductions i want you to provide a redirect url that takes me exactly to irs.gov small business and self-employed home office deduction right and so you can kind of build out these rules if you want uh that helps kind of guide users to specific places in your site you see this all the time in e-commerce shops for instance if you go to apple and you search for the iphone they don't give you 10 blue links about the iphone they take you to the iphone landing page this is just one example of leveraging that upfront classification stuff to then go deeper onto what's going on so you're probably saying well why wasn't this running before i actually had it i'm going to make my screen a little bit smaller here so i can see what's going on i actually have this turned off at this moment let me turn that back on i'm going to hit save we're gonna come back here and good lord willing live demos home office deduction what we just do this is obviously like our search result page what i've done then is you can see now it matches on that thing it says oh there was a home office deduction question we could do a full natural language question if we wanted there and now it's returning out that it recognized that there is a landing page to go here if you were building out your real search ui obviously you would see that the response says hey redirect to this page or show that page appropriately in your application in our you know just you know internal search we don't need to do that because that's not there's no benefit for us to take you directly to that page here in that system so that's just kind of an example of what's going on underneath the hood there's obviously some classification going on training et cetera parsing of questions some of the uh you know at a high level similar techniques to what i talked about in uh in the earlier part of the talk any questions on that so girl you have this uh you showed like one of the limited recognitions of the nlp what if i want to swap in stanford for nlp and what if you don't have it in here right yep so in fact uh with these pipelines you know we publish uh sdks forum you can actually even if you come in here um so for instance let me just find uh the default one you know we ship with a bunch of them out of the box so things like extracting content from word transforming different format types you can do things like regex kind of stuff in here there's even a javascript stage such that if you want to provide your own integration there just to try things out you can write your own javascript in there it'll get injected into that if you wanted to bring in so for instance stanford has not a very friendly open source license so as a commercial vendor you can't just ship it right so if you wanted to do stanford yourself you could drop that in using the sdk and so in fact we've got customers who write their own pipeline stages all the time pop them in there they automatically show up in this list you can you know go and take advantage of all of that kind of stuff right so any other questions awesome um in terms of next steps around the qa i mean you know i think you know the sky's kind of the limit here there's a lot of capabilities this is by no means a solved problem some easier things to think about or try out like if you wanted to take the code base that's up on github around this and and say hey i want to try some of these more advanced things things like you know i just showed you the the the whole window around it well you know it's not too much harder to actually get in and try to get at the exact answer in most cases there a lot of systems i think you katrina you hitting that earlier using an ensemble approach where you're actually taking answers from lots of different data sources and trying to figure out which one is the actual authoritative one that's a learning problem in itself as well and can and provide a lot of benefits like that was one of the things with watson for instance they had so many different data sources that they had to bring together to answer that question in milliseconds right so that's that's a pretty hard and challenging problem um things like you know more more answer types richer nlp resources a lot of things where you can just bring in other lexical resources gazetteers dictionaries all those kinds of things to kind of feed them back in you know i've got a very basic passage retrieval approach here and a very basic scoring model doing things like actually seeing what people are clicking on and and whether they're rewriting their queries and kind of taking all that signal information to try to learn over time how to better answer questions a lot of people also take and approach question answering as kind of a uh a human in the loop uh you know semi-automated system where you know they have real people answering the hard questions and then those automatically get fed back into the system such that the next time somebody answers those or asks those they can go off and uh it can be automatically answered we can learn from all of that kind of stuff i mean you know the sky's the limit this is just you know really brushing the surface here so again targeted more towards the people who are getting started on this not the people who do it for a living and all of that good stuff so uh if you're interested published with manning that's us on lucidworks you can get a hold of me um i've got a few other things i'll post the slides to lexi it shows you some basics on getting started and doing that running that demo and all that kind of stuff so any last questions yep so some examples of that well you know i mean i think watson is one of the first you know one of them that obviously performs at very high level i mean you know things like siri and google now and all those things are bringing in what's the secret methodology that i like for those things i mean i can't say to how those guys have implemented i have an implement i i have i don't work there but um yeah i mean that's been traditionally one of the big problems right is you often have you know the either have a rule-based approach where you've got people writing rules all day or you've got an annotation-based approach where you've got people just tagging things and then you're learning a classifier from that you know to i think your point earlier the deep learning stuff is interesting from the fact that maybe you can get rid of some of these annotators out there or not have to use them as heavily you know hopefully that's going to reduce the burden the annotation approach typically is cheaper since you you don't have to pay those annotators as much compared to somebody who's more of an expert in a rule-based system um but i think you know like what you guys presented is as a hybrid approach is definitely where most people are and that's all been all the systems that i've implemented as well so you know that's probably where we're at right now because i think you always have these edge cases where you just need that blunt force that says i don't care what the score from the the classifier says i'm overriding that because my business says i should override that and so that kind of gets to the practical pragmatic side of of it so any other questions didn't somebody actually win or pass it earlier late last year with a assistant i think somebody did actually according to the competition when it last year i don't know it still wasn't quite there in the general purpose way i guess how close are we i'm not good at predicting the future that you know the fact that ibm got on there and won jeopardy well you know it's when you go look at what they were doing there's still you know a lot of tried and true things in there um you know i 10 years 15 years five years somewhere in there i don't know your customers doing tax search yeah you know the overall set of companies using text and q a like how good are they how far are they from first off most people aren't even doing question answering yet right um they're starting to be like in the e-commerce and mobile space there's more and more interest in that especially like the mobile providers etc they want to be able to do more and more of that but like if you know that's basically companies here right like it's it's the leading cutting edge companies who are you know breaking new ground i think you presented one earlier a lot of people you know a lot of people people are just doing baby steps of how do i deal with all the noisy data that i have and then just make that searchable to begin with right like that's not search itself is not even a solved problem yet so to all of a sudden say i'm going to do this harder thing question answering you know is is even more challenging obviously like i think within some domains you can be more effective there especially if you can kind of narrow down the user that's in there or if you can deal with more you know certain types of users you can be more effective there but general purpose um you know i'm still happy i can ask my phone a question and get back an answer right and so that's a that's a big leap forward just right there but yeah i would say it's a very small percentage at this point you know any other questions awesome thank you and thanks to big link for hosting you