Scale By The Bay 2021 : Amin Ahmad, Semantic Search and Neural Information Retrieval
Recording: Scale By The Bay 2021 : Amin Ahmad, Semantic Search and Neural Information Retrieval
all right uh well thank you it's a pleasure to be here with all of you today um devon thank you for the introduction and uh as you mentioned i'm interruptable uh we have a lot of material to cover so i'm going to move quickly but the point is your understanding so if you're confused or if i need to explain something better please stop me and let's let's address it so the talk today is going to be about semantic search and neural information retrieval what is it uh what can it do beyond a standard keyword search what are some of the architectures that are employed for this kind of search so before i get into the details there let me give you a quick background on myself i've been a software engineer for about 20 years 10 of those at google eight of them i was in google research and then the last four years from 16 to 20 i was leading uh research into neural information retrieval at google using using neural networks so uh i was involved in all aspects of designing the neural networks uh for large-scale information retrieval uh but as an engineer i'm very interested in not publishing paper so much as in building systems that people can use so all of the auxiliary machine learning that sits before and after the neural networks in the information retrieval pipeline i have experienced working and designing that so for instance statistical calibration and re-ranking we'll talk about all of that briefly today and then also i'm very interested in how do you actually deploy one of these systems to production neural networks for those of you are familiar are notoriously computationally intensive uh and so you know you have to pay careful attention if you want to be able to scale up to ten thousand or a hundred thousand documents and doing retrieval over that okay um so i'm going to move along kind of quickly but i think it's it makes sense to let's let's say first start with what what does semantic search even mean it's it's had many meanings over the decades and i mean something very specific the word semantic itself in english just simply means of or related to meaning in language so a semantic search in a way not very descriptive a keyword search uh is a semantic search it captures some aspect of meaning in a language or else it wouldn't be useful so specifically in this talk semantic search refers to doing information retrieval with neural networks using techniques that didn't exist before 2017 so this is all very new this is cutting edge this is not widely deployed outside of i would say the fang companies um and these neural networks are able to understand language at a much deeper level in a more human-like way i would say compared to a keyword search so to motivate this discussion i'm going to quickly step you through how a keyword search operates let's say that i'm running this search for dog adoptions in silicon valley and i have this big data set of you know thousands of documents and those are portrayed down at the bottom and uh you know one of the first things i want to do is look at you know which words in the query show up in which documents okay so different words show up in different documents and then of course there's some words uh like the word in that show up in almost every document and we would say that that has very low information content so in a traditional keyword retrieval pipeline uh words like in tend to be stripped out that's called stop word removal and it's automatically configured in systems like solar or cloud platforms like elasticsearch and algolia and then of course when i'm searching for adoptions i also want to match adoption adopted adopting all of the variations and conjugates of that word so i need to strip the words down to some base form before i run the search some kind of normalization that's called stemming so it's a language specific step that's done and again it's provided out of the box so now i have you know dog adopt silicon valley and i'm going to match these terms okay and i need to match these terms fundamentally based on something that correlates to in what's in statistical terms known as the mutual information uh between the terms and the documents so um you know uh that that is that is captured in in in algorithms like tf idf and bm25 which are used for keyword retrieval uh their scores essentially correlate in some way to the mutual information between these terms and the documents okay so uh let me stop there that's what you get out of the box with the keyword search but in order for the keyword search to really be intelligent uh i would want to be able to do something like searching for uh bulldogs uh for adoption in cupertino and match that that would be really intelligent so how do i get that with a keyword search to do that is more tricky and it requires something called term expansion where i need to go in and basically define essentially that you know for a dog here are the types of dogs these are things that are dogs bulldog chow chow the eagle kali similarly silicon valley is actually san jose mountain view sunnyvale and cupertino and then i can generate a much larger set of queries essentially i rewrite this original query into a whole set of queries and running these against my document store i can pull back a much richer set of results and so now i finally kind of achieve my objective of being able to search for uh you know dog adoption in silicon valley and finding a bulldog that's up for adoption cupertino now this is the point where the search looks you know pretty impressive and it seems intelligent the problem is the term expansion uh is not something that's available out of the box with a keyword system these term lists have to be defined and then they have to be maintained and no sooner unfortunately you define these term lists then the exceptions start to pop up everywhere so let me just give you an example that now if i've got this nice term list that expands the word dog and now someone enters this query good hot dogs in silicon valley i think it should be obvious that expanding dogs in this query isn't a good idea it's not the same kind of a dog so you have all of these corner cases showing up because of the flexibility of language this is not a problem for humans to understand but keyword systems really struggle with it and the end result is that until now uh sophisticated search systems which rely on keyword search in the back end um have have remained the domain of companies like google and bing which have a multi-billion dollar budget to maintain uh maintain these systems and keep them running optimally so for instance when i was at google there was actually a team of engineers dedicated to this problem of keyword and term expansion in the queries and finding the edge cases and fixing the edge cases and new ones would keep coming up so uh roughly speaking uh we have this graph that as you try to improve the search result quality linearly uh the cost and complexity of of of that search both in terms of developing it and then maintaining it uh deploying and maintaining it go up exponentially all right so that's that's a little bit about keyword search okay and uh let me now pause and i'm going to show you a demo of what a neural network based search can do because it really levels the playing field between smaller players who have search problems they need to solve but don't have the budgets to do it and then the large companies like like uh the googles and the microsofts of the world so i'm going to go over to the demo we've taken a uh a router from amazon which is very popular it's a netgear nighthawk router we took about 2000 reviews and uploaded them into a semantic search system this happens to be uh you know zero semantics search uh my company's platform uh and i'm just gonna put a query in here um uh can i stop my child from using the internet i'm a parent maybe i'm worried about this right and now what you see is a list of results coming back and within the results our system is actually able to identify with machine learning the most relevant part of the part of the uh review okay and so you seriously i like being able to disable the internet when my son plays video games so i didn't say child but it said sun okay and that's a that's semantically similar so the system was still able to return that there's no way to block social and time wasting sites at night for your children's device devices so here you've got you know no explicit mention of the internet but social time wasting sites it's clearly we're talking about the internet here right so that's the kind of intelligence that i'm basically getting for free i didn't need a whole team of data scientists to define these term expansions and maintain these lists for me i'm getting it for free here and it's a much more sophisticated experience one of the other interesting things that you can do with the neural search is if you've trained the neural network correctly you can actually work across languages so i'm just going to show you a real quick example if i if i go to google translate i'm going to ask um uh you know can i get signal outside maybe that's a question i've got about this router okay i just translated chinese i assume this is a decent chinese translation and i'm gonna put it in here and uh here it is um because of the way these neural networks work uh people when they see this they ask me oh did you go to google translate and translate to english and run the query no uh this is actually the neural network directly interpreting the chinese and returning the most relevant results it can even though those results happen to be in english all right so um there's the demo uh hopefully you get a sense of what you can do with a neural retrieval system that you couldn't do with the keyword system easily i'm going to go back into the presentation and give you a bit of historical overview now on um the 2010s and nlp because this was a watershed decade for for natural language understanding with computers we entered the decade computers were struggling uh to understand language at the level of a three-year-old and as we're leaving the decade by around 2018 computers were matching the performance of graduate students on question answering data sets from stanford university so there's been an enormous change and i want to share my perspective on the three key breakthroughs in the past decade that have brought a computer and machine comprehension of natural language up to a human level okay so i'm going to go quickly but please stop me if you have any questions okay so neural networks require numeric input to operate so you've got to turn the words in your language into numbers somehow the way this was traditionally done was with what's called a one hot vectors the way one hot vectors work is like this you're going to take let's say the 50 000 most common words in the english language okay and you're gonna have 50 000 dimensional vectors you're going to assign every word an id okay and then in that position of the vector there's going to be the value one and everywhere else is going to be zero so let's say the king has id 17 in position 17 there's going to be a 1 and everywhere else there's a 0. and here in this example queen has id2 so it's got a 1 in position 2 and a 0 everywhere else this is how words were fed to neural networks and neural networks were never able to make much progress on language understanding uh with this paradigm and one of the reasons for that is because there's really no connection between one word and another king and queen are semantically connected but they're no closer together in the vector space than any two randomly chosen words if i'm going using this kind of a scheme so the first breakthrough happened in 2013 with word vectors word vectors uh provide dense embeddings so you're talking now about vectors between 50 to 300 dimensions all non-zero entries and crucially the the geometric structure of the vectors and the relationship of the vectors to one another reflects the semantics of the words themselves also just as crucially the procedure for training word vectors is unsupervised so for instance you can you can feed project gutenberg all the works of project gutenberg to this algorithm and you're going to get vectors for every word in the english language as an output and then uh you know looking at the diagram on the right there going back to the king and queen example now we have word vectors uh and you can see that the two red dots there represent king and queen in this word vector space okay and uh they're much closer together than two randomly chosen points on the sphere okay so that's the idea of word vectors and this concept of vectorizing the meaning turning the meaning into a vector in a vector space this can be extended beyond individual words to sentences paragraphs even to full documents okay and this is one of the key foundations of modern neural information retrieval all right so moving on uh the next breakthrough came uh in 2017 language is fundamentally a one-dimensional sequence uh that we uh receive this one-dimensional stream of words and we construct meaning out of it in our minds uh the way that one-dimensional sequence uh sequential information is handled in neural networks traditionally was with recurrent neural networks a recurrent neural network at time t takes some input and also takes its output at time t minus one to produce uh the next output so they're able to model sequences of information so until 2017 in the introduction of transformers systems like lstms for those of you are familiar were dominant but the problem is that these kinds of systems are very difficult to train they're slow to train they're difficult to train they're very difficult to productionize so transformers brought the benefits of sequential modeling into the paradigm of feed foreign neural networks uh feed for neural networks are much more scalable uh they're easier to run in a production setting and you can train them much bigger so now feed four neural networks for the first time gain this ability to consider sequential information so man kills bull now looks very different than bull kills man okay and uh i can't uh overstate how critical the transformers have been to the field of deep learning in general uh whether it be image processing video processing or anything else however nowhere was the uh was the improvement more drastic than in the field of language understanding and that is because language human language is fundamentally sequential so about 10 months before transformers came out i had embarked on uh you know leading this team uh on on large-scale information retrieval neural networks when when transformers were released by google research we were one of the first teams to incorporate them into our system and that one change alone gave us a 27 relative jump in f1 score so anyone on the call who knows about data science f1 scores is the geometric mean of precision and recall that is an enormous jump and it especially coming from just a single change inside the neural network normally you're talking about fractions of a percent improvement um this is i give credit to transformers for allowing us to launch google talk to books it's an ai experiment from google you can actually try it out yourself you search 200 000 books with neural networks i actually train those neural networks to power the system we couldn't have launched or we would have launched something much weaker if it hadn't been for transformers so that was the next uh big breakthrough in 2017 and then the last big idea um became uh popularized in 2018 a system called bert if anyone follows nlp i guess you might have heard of this so bert introduced the idea of unsupervised pre-training you see think about humans and language if i am going to uh if you're going to ask me about medical topics okay i understand the english language i'm not a doctor i'm not an expert but you could hand me a few medical papers and if i read them i might be able to get some idea of what they're talking about and be able to have some kind of a discussion with you on the topic afterwards i don't have to learn everything from scratch every time uh i i carry my uh knowledge of language around with me so essentially the way what the idea that bird introduced it's a very deep 21 layer deep transformer neural network they trained it ahead of time on this unsupervised task unsupervised is key because they essentially had unlimited amounts of training data then and it's a kind of a fill in the blank task so for instance john bought eggs at the blank okay store makes sense here you know school probably doesn't make sense the neural network has to predict and fill in the blanks and over time it gets better and better at that and getting better on this task actually means understanding the mechanics of the language okay the two are directly related so you you you pre-train on this one uh generic task and then what they found is that then with a very small amount of task specific training data the neural network can get very good very fast so with only a few hundred examples on the task that you care about now the neural network is able to perform almost at a human level and so what they showed is that they took this one system and they were able to smash all the previous records over a wide range of language benchmarks i think 15 or 16 different language benchmarks they set new records and crucially it was the first time an ai system ever exceeded human level performance on the stanford question answering data set which i alluded to at the start of this talk there was actually a new york times article published about about bert so so this was i would say the third key moment or the key insight in the past decade and uh bert by itself for all of its power is not productionizable it is extremely cpu intensive and it is a it provides an on search so if you if you want to search for the answer to a question in let's say 10 documents uh it's kind of got to linearly scan through the 10 documents so searching through 100 000 documents with a cross-attentional bird type system it's impossible okay um all right so that was that was a quick recap there um i wanna get into the uh uh the actual architecture of a modern neural retrieval system next uh but i'll pause if there's a question otherwise i can move on uh i'm not seeing any questions so you move on all right all right okay so let me uh let me go through what a modern pipeline looks like uh this is these are the same stages that would be present in a traditional pipeline but there are neural correlates to each of these so i'm going to comment on that when a query comes into a large scale search system generally there's several different backends that you might potentially hit depending on what the query is about so you might have noticed when you search on google sometimes you get video results sometimes shopping results sometimes it's just plain web results other times you might get flight results it depends on what the query is so determining the intent of the query and routing it to the appropriate search backends that's what query routing refers to this is done almost exclusively with deep neural networks these days the next step is actual retrieval so one of these backends has been selected now it receives the query how is it going to search through thousands of documents to retrieve the most relevant set so this is done actually in two ways in modern information retrieval systems neural information retrieval systems it's done surprisingly with bm25 keyword retrieval uh and that's a popular choice because it scales very very well um and then the other option is to do it directly with neural networks and that's what our company focuses on so we generate these vector embeddings uh that represent the documents and we we essentially uh generate a vector embedding of the query and then we find the nearest documents to that query and those tend to be good search results so for that you need a specialized piece of infrastructure called a vector database uh there's several good ones out there but if if you're familiar like there's um from facebook research annoy hns live all of those are out there as viable candidates for for fast vector search and that's an area of very active research at the moment is you know how to scale those things up so you've got two options over here and then uh the next step is re-ranking okay so re-ranking refers to two problems if i have result sets from multiple search backends how do i merge them into one final list and and then how do i uh reorder the entries in the list so that the best items come to the top so neural commercial neural information retrieval platforms which i'm going to cover in the next slide some of them will use a bm 25 search and then use the use re-ranking with neural networks and that's why they're dub neural and others will actually do a direct do the retrieval step directly with neural networks and they both have some pros and cons which we can discuss at the end if there's time uh the next key step would be snippet extraction particularly when you're doing something more sophisticated than matching keywords in the title of the document like you saw in the demo uh you know there's relevant information very deep in the review you need to you need to be able to highlight what's relevant so that people know why you're showing them the search result otherwise it's just going to be a bad experience so some some retrieval pipelines can actually do the snippet extraction automatically as a part of retrieval and that happens to be the case in our system in that case it can actually be fed into the re-ranking step and produce an even better re-ranking but in other cases uh it can't be and and and then snippet extraction is actually fairly expensive so it needs to be done after re-ranking so you can limit to running on like the top three or top 10 documents however many you're going to show on that first page of results okay and then the last step is something a little bit more advanced it's statistical calibration of the results uh whether it's bm25 or a neural network you're going to get some kind of numeric score out of the system well what does it mean to get a score of 26 from bm25 or a score of 0.73 from burt is especially 0.73 deceptive is it 73 probability of correctness well it turns out it's it's absolutely not um um you know uh these deep models deep neural networks in particular are extremely uncalibrated they tend to skew towards the extremes 0.99 or 0.01 with very little gray area in between and you for the sake of human interpretability of the results you want to bring those scores into alignment so that they reflect the true likelihood that the answer being shown is is relevant so 0.7 should mean 70 chance of being relevant or you know correct answer to the query all right so um moving along uh here's an overview of three commercial uh past solutions on the market today and just how they compare to one another so there's kendra from amazon which was launched in 2020 microsoft semantic search launched in 2021 i believe is stone beta and then their semantics search as well uh key point the retrieval is done with bm25 in both kendra and microsoft semantics search they use neural networks for re-ranking one of the nice things about this approach is again it scales really well to very large data sets and you kind of get an immediate boost over the performance of keyword search on the other hand doing a dense retrieval directly with neural networks is has some advantages you can pull back results from other languages which by definition are not going to have any keyword overlap and you can even pull back results from the same language which have no keyword overlap our platform does not perform the re-ranking step our customers who want to re-rank and merge results from other systems implement the re-rankers themselves unfortunately none of these systems perform calibration but i imagine that's something that should be added in the future all right so any any questions that i'll pause otherwise i'm going to kind of conclude with one more uh example that's not directly searched but should give you an idea of like the power of neural information retrieval and and you know where else you could use it um yeah i'm looking for people typing but i'm not seeing it so yeah i think we're free to go on all right all right good all right so this this last example is a let me put it's the category of a non-obvious uh application um so there's a government we're working with in the midwest uh and it's a decision support scenario for child welfare so let me quickly explain how this works okay so i'm going to switch tabs here um the child welfare system uh works like this if a child is in foster care or in state custody the state will send a welfare worker a couple times a month to document the conditions of the child make sure he's being properly cared for and they write long hand notes okay and they get training in how to write these notes and what you have on the screen in front of you is a representative note it's fictitious okay but it is actually very representative it's used for training uh child welfare workers so you got a case here of sandy who was removed from her home by child protective services and placed in foster care and now now the child welfare worker is visiting and documenting the conditions and the way it works is every visit generates one of these reports these are collected into a case file and then at certain points in the year uh there's someone in central administration who has to sit down and make a decision that will affect the trajectory of this child through the child welfare system and uh you know the the determination is is made on the information in the case notes but it's easy to miss information in the case notes and when information is missed sometimes the results can be catastrophic for the child so they really wanted to get the error rate down because it happens a lot you know people overlook some key piece of information so what what we did here is we took this just this one case note and we loaded it into our platform okay and i'm going to switch now to the platform and this is you know sandy's case notes this is the corpus where it just literally contains that one case note and now just imagine with me for a second uh that you're now the caseworker uh sitting in the central administrator sitting in the central office trying to decide you know what what needs to happen with sandy is she fine with this family can she go back to her mother so i'm let's say first and foremost i'm interested in physical abuse okay so i'm going to put that into the search okay and and imagine that i'm not putting it into the search but it's being proactively retrieved from the notes for me so i can see anything related to physical abuse it shows up on the screen and then i make the decision oh is there physical abuse yes or no and i've got this information on the side okay so sandy was removed from her home by child protective services and placed in foster care after a teacher observed multiple bruises on sandy's body mother claims she was unaware of injuries sandy said mother's boyfriend often hit her for no reason okay all connected to physical abuse scattered throughout the document but a semantic search system is able to go and retrieve all of these and kind of bring them to my attention even though there's no keyword i overlap physical and abuse don't appear anywhere in these sentences if i change physical abuse to substance abuse that would be another thing that uh someone would be interested in mother has a history of reported drug and alcohol use and was intoxicated at the time of investigation caseworker explored the frequency and quantity of her alcohol consumption i like this one here mrs white stated she'd been drinking two or three cans of beer a day with friends so beer alcohol consumption drug use all of these are connected to the idea of substance abuse but the substance abuse term itself is never explicitly mentioned in any of these sentences so you can see the potential for neural information retrieval to pull back really relevant information information that if you miss it could have a very high cost in the life of that child potentially but neuro information retrieval will catch it where a keyword system would miss it and this is an example where you can enhance a decision making system a decision support system using these kinds of technologies so i mean this technology is is very powerful it's very broadly applicable but it's also very very new uh outside of the fan group of companies very few other companies uh even know about these capabilities uh much less you know have started to deploy them but i think that with these cloud platforms now becoming available in the next two to three years you know you should see uh you know widespread penetration of this technology in the industry so we will uh we would just leave it there and i'm available for any questions now or afterwards you