Faster and Cheaper Training for Large Models

Faster and Cheaper Training for Large Models

Recording: Faster and Cheaper Training for Large Models

all right excellent all right so uh and uh again uh welcome mate today my table talk about faster and cheaper training of large language models i think this area is super interesting to everyone and and without further ado take it away all right thanks a lot alexi uh so let me just share this um cool you can see these slides right yep very good oh yeah yeah and yeah since you mentioned apache spark i do want to very quickly say that uh version 3.2 just got released today if you use that check it out but um for the rest you know for the rest of the talk i'm actually going to talk about research that my group is doing at stanford with a whole bunch of collaborators on faster and cheaper large-scale model training so that's been a really interesting area to work in and i'll talk about different ways to tackle it and you know what could work best in the future at least what we think about it um so i think everyone well everyone has seen how machine learning models are going larger and larger every year and for natural language in particular that's where they've you know they've been going at the fastest rate but this is also true in in other domains um so this plot here shows the uh just language models um and you can see they're going exponentially it's a log scale and they've actually grown in size by uh basically like um you know more than a thousand x since uh since 2018 um and the interesting thing is this is actually faster than another exponential trend which is how quickly uh we're making hardware faster so um the hardware people are working really hard you know getting lots of revenue from selling all these gpus and stuff and even then they can't catch up with uh you know with the way these models are going so obviously there's a big problem to make you know state-of-the-art ml broadly accessible uh is you know how do you make it cheaper uh to train these things so i'm going to talk about two approaches and talk about some research we did on on each one the first one is just take today's models and try to design computer systems that are going to train them faster makes a lot of sense there are a lot of people working on this and in my group we did a bunch of interesting you know early work in a few directions and one that i'll talk about is opening up new ways to parallelize uh ml training that actually improve uh device utilization and ultimately improve the throughput you can get so we have this idea of pipeline parallelism and some of the the things we developed are now part of open source software like megatron lm from nvidia that can be used to train large models and they're also being adopted in others but then there's another approach we can take which is don't try to use today's dnn models try to design new machine learning models and algorithms that are actually more systems friendly and this is the one i'm really excited about because i think the first one even if we work really hard there are going to be limits to how fast we can make things and i'll talk about something and specific to nlp which is we we've been exploring these retrieval based models and we actually produced a bunch of models like colbert and baleen that are setting basically state-of-the-art results in nlp but also on much faster than the large language models you usually hear about like gpd3 okay so let me start with that first one so what can we do with you know existing models how can we hunt them better so obviously with these huge models and huge data sets you have to parallelize training to make it go faster and it turns out there are a lot of ways if you're training a neural network to parallelize it um so in the past there are two approaches that are really common that you can hear about everywhere and some of them are easily supported in frameworks like tensorflow too um the first one is data parallelism in this one you take your model and uh you you just put different copies of it on each machine you have or say on each gpu um so that's a copy of the whole model and the thing you parallelize across is just the input data so for the batches of data you train on you just split them say across you know 100 devices and each device does like one one 100 of the uh of the work um so that's really easy to implement that's why systems like tensorflow and pytorch can easily do this by default um and you know it definitely gets some benefits but it has some problems um one of the problems is you every you know like your model actually has to fit on every machine and that itself can be a problem when you get to these huge models so that might be a showstopper if it just doesn't fit and the second problem is that you have to communicate to synchronize the weights or the gradient updates between the workers and that cost is also proportional to the size of the model and that can easily become a bottleneck as well so this isn't usually the best way to to just scale up a model and the second thing is is um the second approach is model parallels where you take your model and you actually split up the parameters onto different machines or split up say the layers and then uh the nice thing about that is if you have a huge model no worker needs to have the whole thing in memory it can each have a little subset so that's really good is only one copy of each weight um and then the other thing is it does usually result in cheaper communication so this is what a lot of the you know the people who train huge models were uh we're doing uh is actually model parallelism so what's you know what's challenging about parallelizing things well in both cases the the the thing you have to worry about is utilization of the hardware and this is where we spent you know a lot of work trying to improve this and it turns out that actually model parallelism the thing that people use for huge models also has um challenges with uh with utilization um so i'll i'll explain how that works um so um so to understand that so actually turns out model parallelism is one of the worst things you can do if you want high utilization of your hardware and therefore high throughput and you know low cost and basically to see why that is you have to think about what each worker is doing at each point in time so remember we took our model and we divided its layers across workers in this case i'm showing four workers so so what are they each doing at a point in time so in ml training you have basically you keep doing uh switching between doing forward passes and backward passes that together help you update your weights and so in the forward pass you take some input batch of records and labels and you pass it through the model from you know from the first layer to the last and to do that with model parallelism that means that you know in the first time step only the first worker is doing any work because it has the first few layers um so the other ones are all idle which is shown here on the timeline they're not doing anything then when it's done with that the second worker does its layers and the rest are all idle because they've got nothing to do and so on so throughout the forward pass basically there's only one worker active at a time makes sense because only one worker has the weights for that layer and then the same thing happens in the backward pass um you know to you you go back and pass back some information and again only the last worker can do anything because it has the last rates and then you know what worker three can do stuff then worker 2 and then worker 1. so even though you fit your model onto lots of devices and you know you purchase these really expensive gpus from nvidia or whatever to do it or you rented them out in the cloud most of your gpus are idle you know three quarters of the times here because they're waiting on other things to finish um so it's not it's not a it's not a very efficient way and this is where we can actually make the cost you know a whole lot cheaper than this and get a huge improvement in cost like you know 4x ideally is the most we could imagine getting here okay so how can you improve this so one one general approach um that you can do is called pipelining and so we we looked into this starting a few years ago and others have been looking into it as well and i'll show first a more basic pipelining scheme and then a more advanced one that we came up with um so one of the ways to do pipelining and this this is actually a few papers proposed as well an example um is the gp paper from uh from google um is basically um looking at the previous picture let's actually split the the batches that are passing through each worker into even smaller batches sort of take your mini batch and split it into micro batches they're getting pretty small here and the nice thing about that is you know as the first worker begins processing each batch once it finishes say a quarter of it it can pass that on to worker two and that one can begin computing stuff at the same time so and then same as it as it keeps completing chunks of it it just pushes them forward to all the other workers so compared to the previous picture you can see there's less gray space on the picture these workers are actually overlapping a whole bunch as they do you know the forward pass and then the same thing happens in the backwood pass um and you know in an ideal state you could get basically like 50 utilization by doing this so that's not too bad um but there is a problem um with this uh which is uh the way that uh you know gradient descent works in in machine learning um you have to wait before you begin the next iteration of forward and backward passes you have to apply all the updates from the previous one to actually move your weights forward and you know apply the gradient you compute it and so that means you have to flush the pipeline you have to tell everyone hold up wait a little bit at the end of each of these steps and that um that really you know keeps your utilization low overall because even though you're ramping up keeping everyone busy for like some fraction of the time you always have to slow down it before you can start the next thing um so that's that's the problem with this so one of the things that we worked on uh that addresses this is we actually changed the um the machine learning uh algorithm like the the gradient descent algorithm to eliminate the need for weighting after each update um and so we designed a few ways of of doing this and one way that i'll talk about that we publish this year is called double buffered weight update so we have this system uh pipe dream that does it and we also implemented it in other ones and basically what this does is um if you look at the bottom here um it actually changes the the gradient up descent uh sort of uh formula that we're going to use so normally in great descent your weight at time t plus 1 is weighed at t minus you know learning rate times the gradient of weight at t and we actually change this from t to t minus one so actually at each time step we're applying a gradient update from computed with an older version of the weight so all the updates are delayed but it's delayed in a deterministic way you're still gonna get all of them um and um uh and and so so so we're we're applying this sort of stale update and the nice thing with this is it turns out if you're willing to do this you can actually um ramp up and keep the pipeline full all the time and the way that will happen is each each worker will actually uh have to hold basically two copies of its weight it will have to hold the weight um from you know a few time steps ago and the current weight um and what will happen is you know as you do this forward pass everyone computes that everyone computes the backward pass and when the backward pass for that first version of the weights gets to the first worker that's when it will begin using uh these updated weights but in the meantime it already started doing another forward pass with the with the original weights and uh that's going to keep going uh until uh actually i guess eight also has to use the the original weights and and that's gonna keep going and give you an update based on those so it's a little bit uh maybe it's a bit tricky to see from the picture unless you think about it you spend a lot of time to think about it but but basically after the first step here you can always keep everyone busy at all times and the the main cost is you you are getting these delayed updates and you have to uh double your use of memory on each worker so you have to keep two copies of the weights but that's not too bad because you know the whole point of this was we we were gonna um you know we could split our layers onto different workers so hopefully each layer is you know small enough that you can do this so that's uh that's the approach that we did there um and um just to show you a little bit of like the performance you can get with this it actually can give you pretty big performance gains for you know real models like gpd um uh two in this case uh without affecting convergence really so at the top here we have um basically we're showing throughput with different batch sizes and we have uh the the blue here and the orange are simple ways of doing tens of doing model parallelism um like i showed in the first picture and and then these uh g-pipe is the you know the the more sophisticated way that still has flushes and then this pipe dream 2 bw is the one you know is sort of the the highest stupid one that we came up with um and you can see it's quite a bit faster right so this is how i think this was on um on eight servers with eight gpus so and overall it's uh you know close to a factor of ten faster and and therefore cheaper um so and at the bottom here these are the training and validation loss you can see we're a little bit behind on on both we take a little longer to get to the same loss but uh at the end of the day we do get to the same thing and it's it's worth it given the speedos this is showing in iterations so that's one of the you know one of the techniques that we've come up with um and you know there are other ways as well um a question about you know all this stuff is hey i talked about how utilizing the hardware is the really difficult part uh especially if you have to spend time communicating or waiting for other things so how close do we get so we actually collaborated with the megatron team at nvidia and we tried um you know we used pipeline parallelism we also used other tricks to use the combination of data parallelism and splitting each tensor too across multiple devices and we actually tried training a trillion parameter model on our 3000 gpus and by carefully tuning this this was all led by my student deepak who recently got his phd and graduated we were able to get 50 of peak efficiency on on 3000 gpus that might not sound like a lot but actually in high performance computing at this scale uh that's considered pretty good it's hard to get that to you know kind of balance communication and computation in a way where you're actually using like that high of a percentage of the floating point operations that that these uh devices can do um and this is open source uh on nvidia's site recently they also published some um accuracy results with this type of model not using our training system it's actually they were training that one earlier but uh but basically you know it does actually get good results on on language tasks um so that's pretty nice um but there is you know there is kind of a problem like if you think about it right one way to think about this is regardless of how hard we work on the system side we can only make things two times more efficient and this type of system even though it's you know like a factor of 10 faster than previous uh parallelism approaches it will still take weeks or months to train a model on lots of gpus so it's still really expensive um i should also mention there are many other interesting pieces of research in this area so another project i worked on that i won't really cover here is to automate parallelization like in this case deepak manually you know configured models and figured out how to how to tune them and how to divide things but most people probably don't want to do that and my student jihao jia actually developed a framework called flexfo that's also open source and that does this automatically he's actually giving a talk this thursday at the stanford ml system which is streamed and recorded on youtube if you want to check it out um so definitely encourage you to see that because he's done really cool kind of compilers and automatic optimization work to make this type of thing accessible okay but at the you know at a high level like okay we've made good strides but it's still it's still going to be pretty expensive to train things this way and there's also some bad news even if you have very high budget if you're saying you know i don't mind using 3000 gpus or even like 10 000 gpus you know just just give me the model um it's actually hard to scale beyond um you know where we are now for other reasons as well and this is basically because as you increase each dimension you start to get diminishing returns so for example in that result the megatron result we did use some amount of data parallelism we took each batch and we split it into smaller pieces and gave them to different parts of the system the problem is you don't want to make the batches too large in um in gradient descent because you get limited signal like basically it's better to take more gradient steps with sort of a medium-sized batch then a few steps with a with a huge batch so there are diminishing returns to just increasing the batch size and if we keep the batch size small the problem with parallelizing across more nodes is that each node starts to get so little input data that it can't actually fully utilize the hardware on the gpu so it can you know like for example you're trying to multiply matrices and if one of your matrices is like 100 by one you're not going to get all the peak flaps that like nvidia can attain for matrix multiplication those are attained when you get big matrices like say 100 by 100 or something where it can actually optimize and cache stuff across both dimensions so that's a problem and same thing with pipeline parallelism you know if you if you already put each layer on a different device it means you can't do more pipelining you're already maximally pipelined so it's actually hard to even to scale this kind of thing past the 3000 gpus we had okay so that's that's kind of the state of this stuff hopefully gives you some insight on what can happen there um so then the other thing i wanted to talk about let me just check my time here okay it's good is a totally different approach which is uh what if we design ml models differently so that it actually they're actually more systems friendly and this is an a i'm excited about it's still pretty new but a bunch of people are are working on this kind of thing um so in this we focus specifically on natural language but you could imagine doing some of this and other stuff too and you know we're excited to try it uh there as well so let's think about like hey how does large-scale nlp work today all the results you read about where people are you know doing really well on benchmarks and so on recently have been on using these huge dnn models like gpd3 and the idea there is you take this model and you train it over you know massive amount of text like everything on the web or curated subset of the web that you think actually has accurate information and then it it basically encodes knowledge in its weights somehow uh about all kinds of topics and then you can ask it a very you know specialized question like what protects the digestive system against infection and it can actually give you a meaningful answer based on the knowledge it learns you know modeling stuff on the web so that's you know nice uh very very impressive that it can do something like this um but then if you think about hey practically using this there are a lot of challenges um first of all it's really expensive to train and even to do inference with which i talked about because you have to you know multiply and add all these weights every single time um but there are also some other practical challenges um one challenge is is that in any like real world nlp system you probably need to update the model's knowledge over time um for example you know let's say this is answering like hey what's the the nearest pizza restaurant to me and actually one of the pizza restaurants is closed this week you don't want it to keep telling you about that one so you'd actually like to update this model really often um and um and if it takes you know three months to train you can't do that so that's a problem with actually stashing all the knowledge into the weights um and then the final problem is it's also hard to uh interpret what this is doing uh when it gives you those answers or even to figure out how to trust those answers once it tells you this how do you know whether there are like real sources backing this up or um it just kind of hallucinated that answer and actually you know gastric acid has nothing to do with this it's hard to tell unless you're an expert in this topic okay so let's look at just the efficiency part so from a system's point of view it just seems that stashing all that knowledge into the weights and then actually running the whole model each time you get an input is very inefficient it's basically like doing a linear scan through your knowledge so one one analogy to this is if i asked you to answer this kind of scientific question i gave you an encyclopedia you could you know you could use that to answer the question and your approach was okay i'm gonna open the encyclopedia and read every page from the first page to the last page and only when i'm done scanning to the 10 000 pages i'll answer your question it's not efficient and you know it's it's it's it's not the way that a human would do it um so what would be a lot better than a linear scan is if we could actually use lookup somehow right lookups into memory are how you know basically a lot of things on on large data sets can can be made faster um so that's the idea of this retrieval based nlp as i said there are quite a few works that tried to do different parts of it and we we basically came up with some um improved ways of of doing it over some of the past work um so the idea here is your system's still going to get some kind of question or some kind of input for a task um and you'll still have some dnns but you'll have basically you'll also have this collection of documents that's called your corpus of of documents that encode the knowledge you want the model to have so for example this could be every page on wikipedia if you trust wikipedia or it could be other resources you think have valid knowledge in them and when your input comes in the first thing your model does is based on this input it actually will look up some some documents in the scorpius and pull out documents that it thinks might be relevant to answer this question so there's a retriever dnn or more generally some kind of retriever system that does that that tells you which documents to pull out and then the once you've got those there's something called the reader that takes the original input and takes all the relevant context you got for it and then answers it and one of the cool things about that is it can even give you a reference to which document it got an answer from like for example if this is an extractive dnn that tells you okay here's its output is is a pointer to like some snippet in an existing document then you know exactly where that came from and you can actually show that to a user and say look you read about this if you want to evaluate this answer so that's the idea so it's you know lookups plus still some some deep learning stuff to actually do the language processing um and the benefits of this you know if it works if it gives you high quality results are um it's a whole lot cheaper like these dnns were using you know just bert for example which you know a few a few years ago that was considered a huge model but today it's like hundreds of times smaller than uh than something like gpd3 so that's great um another cool thing is you can also update your knowledge really quickly maybe one of these pages was wrong or you know new research got published and it's not actually gastric acid how do you make the model return a new result well all you have to do is you know edit your corpus deleter document and add a new one you can just do it in a few milliseconds so no needs to retrain for months um and then the final thing is that it's easier to interpret because you do have links to sources although you know there are still dnns it's hard to tell why exactly this thing decided to eat this but at least for humans using the system it's actually you know you've got some benefits um so we have a nice blog post on this as well that i'll mention is a link in here but i'll tell you a little bit about like how how to make this work and some of the things we specifically added in this field so with our results and this is with my student omar khattab and professor chris spots we we basically set uh state-of-the-art performance on many on a few hard nlp tasks at much lower compute guys than before so we looked at three tasks um one of them is information retrieval which is just the search part can you find you know documents relevant to a query and we designed this model called bear that can match um expensive ways to do uh retrieval that involve calling birth a lot of times and but they can do it uh you know hundreds of times cheaper um that's actually now being used in a bunch of commercial systems and incorporated in open source ones like this uh system called vespa um so so that's that um then we looked at things that require actually you know reading and answering stuff uh so first one's question answering we have a paper on a qa system based on colbert that actually set the state-of-the-art scores in question answering on a lot of data sets and is still ahead of stuff people have published since it came out uh in the first half of this year um and then um another one is multi-hop reasoning where you you're asked to verify a claim and you actually have to look across many documents not just one to answer it and we also set the state of the right result on that and this is this actually is appearing at eurips uh this year um so that's what we've been working on so how did we get this so there are kind of two things we worked on that i'll explain how they help the system the first is improving retrieval so we came up with this approach called late interaction that basically lets you when you're looking for documents relevant to a task it lets you do some of the kind of modeling benefits that these transformer models like berta do but it still remains very fast to search over your collection using this so it's about how do we how do we do this like complex modeling of language like attention for example uh in a way where we can still efficiently search through documents uh through billions of documents maybe and pull out the ones that match best so that's that and then the second thing we worked on that helped us really improve over best results here is new ways to supervise these models where you basically can teach a model how to search for relevant documents better for a given task and the challenge here is usually you know the answer to the task as your training label but you don't know what you should have searched for nobody no human is sitting there and labeling that and there might be many possible ways you can search so so that's the challenge and we figured out some ways to improve the the training for that okay so let me talk about the retrieval part i think it's actually nicer and and it's pretty easy to understand um so how do people do retrieval based on you know neural networks to begin with um so there are there are kind of two ways that people have tried um so one way is so imagine you have something like bert that's good at encoding text and embedding text in a way where you know similar phrases with similar meanings will be given similar embeddings so a natural way to use it for retrieval is you basically apply birth to your quays and you apply it to your documents and you just take you know the last layer out of it as you're embedding and then you do similarity search on that so you basically for a query and a document it's easy to produce just a single embedding from it and then there are there are data structures that can cheaply search given a vector for the the nearest neighbors in a in a data set so um for all the for given a query you can very cheaply search for the documents whose embeddings are closest to it in in terms of distance um so that's a very natural way to do it um you get cheap search computation you can just encode the documents once offline then they're sitting there in your index and then for each query you just have to run birth once on it and then you do the search and then you've got your results but the downside is you've condensed all the stuff that bird is doing is condensed into like this one vector here and it turns out you can lose a lot of um it's hard to train this to like really represent the full nuances of the text and the query and the document so um it's it's hard to get you know the highest quality with with this approach um so that's that and then the second approach that people did that's actually the gold standard in terms of accuracy is uh all to all interactions so in this what you do is you take all your documents and you take your queries and for each query and and every document you actually run a birth model over both things together like you concatenate the query and the document and you tell it hey given this query and this document how good of a match do you think it is that's what you train it to predict and so the nice thing about this is bert is really good at um you know looking at words in context so it can look at this word in the query you know given all these things in this document what what you know is there some way that it could be related to these right so it can help deal with things like you know if there are you know ambiguous words in a query or whatever the context becomes a lot clearer when you can look at the document and allow them to interact one by one to give you a score um so that's what this gives you but it's expensive to compute on all the documents so this is actually used in search engines uh like google and bing today and basically uh what they do at a high level is they run a cheap retrieval method maybe like this one or maybe something even cheaper just based on text matching to get say the top 1000 documents that seem relevant to the query and then they run this method they on board basically a thousand times uh to re-rank them and to give you the uh you know the ones that they think are actually best so that's kind of what um you know what's out there okay so how can we do better than these we'd love to have this all to all interaction here to get the fine-grained modeling but we want something that's really cheap where we're not calling burt uh you know thousands of times doing all this computation um so this is where our approach came in uh called late interaction and basically what we discovered is that um if you if you take your crayon document and you design the system so that you know you run the first n minus one layers of birth independently on each one and then for the last layer you allow them to interact um you know one to one you can actually keep a lot of the matching power of bert and you can get something that's much cheaper to run and it's actually efficient to index so that's basically what we do that's why it's called late interaction we create a model that can encode both queries and documents but they're only allowed to interact to see each other's terms in this last layer and this last layer is basically taking each term in the query and comparing it with every term in the document and telling you which one the embedding is most similar to so it's like a bunch of distances and a max over them and then you sum this up to get your final score um so that's like you know very very little interaction that these embeddings can have but it turns out when you train this it can it can come really close to matching you know the the accuracy of the ones that allow these to interact at every layer and the nice thing with this is once you is basically you can take your documents and you can do all this stuff offline just once you can run all these layers and it turns out you can build a data structure where you stick these into an index you stick the the term embeddings into a nearest neighbor index and then you can do this scoring computations cheaply using index lookup so you don't have to do a linear scan over all the documents to figure this out um so you get sort of the benefits of both approaches and these are some um some performance results on how it did on retrieval and this is a data set from microsoft i'm running on um so basically it's showing you on the y-axis is square latency and on on our hardware and on the x-axis is this um mrr score which is a score for retrieval it's basically how often you know you find it's like mean reciprocal rank so it's basically telling you when you find uh answers early on like in your search results page when you find the right answer earlier on and you can see basically before colbert there were these bird models here that were way better than anything else in mrr but they were also costing you know um like a hundred times or even a thousand times more than the previous methods and that's because they had to run this huge dnn on the top hundred or top a thousand documents that are that you retrieved and it's just like super expensive and with colbert we were able to make these um or close to a hundred times cheaper and more or less preserve the accuracy that they get so this is uh you know it basically um uh these are two variants of cold bear but you know it's it's it's keeping it you know it's you get a almost the same mr so it's a it's a it's a nice kind of step forward in terms of uh modeling these interactions okay um so does another uh the another cool thing about colbert is you can also see how it match things and actually um understand why why it did them to some extent uh and this also shows you it's um sort of the the the power of actually matching using neural uh embeddings of each term instead of like you know using just the word matching so for example this is a query in this data set when did the transformers cartoon series come out and this is one of the passages in the document that you're supposed to return for that and the best hit and what i'm going to do is highlight for every word in the query which word had the maximum similarity that colbert matched it which with so for example for when colbert matches it with on because they're both about time for transformers of course it matches it with the word transformers which would have been found with keyword exact keyword matching too for a cartoon it actually matches it with animated because um animated and cartoon you know means something similar in this context right animated could mean something else in other ones and then for come out it matches them both with released so it's doing you know these are some of these things wouldn't have been caught with simple like keyword-based matching but they are caught with these um you know these uh these learned embeddings and one of the key things about this is also like the meaning of come out for example of the words come and out are different in this sentence uh uh where you know come and out are together and where there's also a when in the sentence that means there there's something about release like you could have you know sentences with these words where they mean something uh super different uh but but the the model can actually look at the context of like all the words here when it comes up with those embeddings and same down here okay so that's called bear that's one of the things we built on and uh it's also open source so we're actually um we we actually also are working on a compressed version that's uh that uses um you know very little space to store this by basically clustering all the embeddings um you can find it out on this website if you want to check it out and we're definitely happy to talk to people who may want to use it for stuff let's see someone asked can you tell us the meaning of mr at 10 value of 0.35 uh yeah i can try so i mean mr i have to remember mr 10 is so it's mean reciprocal rank so basically the so what we do is for every query we look at uh there's one document like there's one right answer it should have retrieved that a human labeled um and we give it a score of one if um the uh if uh if this is the first hit that the system returns so then your score is one if it's the second hit your score is a half if it's the third hit your score is a third um if it's the fourth hit your score is a fourth and so on so that's why it's called reciprocal length so for every query you can see hey did it you know what um you know where is the right answer in the list and if it's the first one you get at one point or if it's the second one you get half a point and so on and um at 10 means we stop at the tenth one so if it's not in the top ten results you get zero you don't get like one out of 11 or whatever if it's the next one and then mean reciprocal rank is averaging this out um so one one way you can get mrr of 0.35 is if for 35 of the quays you return them as your first hit and for everything else you didn't return it uh then your your value would be point c five but more realistically what's probably happening is that for most squares your result is in the top you know three or i guess top six or something like that um and so on average you got about a third contribution from each of them so yeah so you can definitely the ideal value is one if you always got the right answer but you know for a search engine getting the answer in the top two or three is not too bad either okay all right so i went to this so um cool so okay so next thing i just want to talk about uh briefly is there is there is the supervision for more complicated tests so retrieval you know it's it's great that you can do something cheap there but the exciting thing is how do you use it for a bigger nlp task you have and so one example of that is question answering so this task is harder than retrieval and retrieval you put in a query we just want to find the best hit in the you know the best page on the web that people want to see for this you put in a question and you're supposed to get back a string like for example where does the volca river end it ends at the caspian sea um and we're actually specifically we're looking at extractive qa where you're supposed to take a substring of one of the documents and returned that but you know you could do it in other ways too so what's the problem with this so the problem here is um if you're going to do this using retrieval it's hard to supervise the retrieval given your training data set the training data set has questions and answers right but the retriever has the retriever needs to know which document you should retrieve to get that answer so you know given this question like what pages should i pull out of wikipedia that are likely to tell me you know where the server ends um and no no one told us there might be multiple pages that contain the answer um and so how do we know like which one we're actually supposed to pull so the retriever really needs question positive passages and negative passages so that it learns to distinguish you know good pages to pull out from bad ones and we don't really have that so that's that's the that's sort of the question um so basically in a nutshell the what we did is something called relevance guided supervision which allows you to take an initial retrieval that maybe isn't great for question answering and gradually make it better specifically for the task of answering questions before us a lot of systems like either use the an off-the-shelf retriever and never tried to make it good for questions so then you know if it if it gives you back the wrong document you just can't answer the question they just focused on answering it if it happens to find the right one or people had uh you know other ways of training the retriever that were that were pretty expensive basically and you know didn't didn't give you great results uh at least compared to what we did so what we do um basically we came up with a way to um to guide the retriever based on just knowing the final answers so what you need for it is you need an initial retriever that can be even a simple like text matching thing like bm 25 is a traditional text based index or something like colbert that was just strained for search can be your initial retriever and then you also need a heuristic about given some passages which ones are likely to be useful for answering a question and in our case this was easy to come up with because we just said hey if the answer like caspian c appears somewhere in that passage then uh we're going to consider it a good passage to retrieve and if it doesn't then it's definitely bad there's no way we could answer the question correctly if you know the thing we pulled out doesn't have the answer in it so that's what we had and what this does is it basically does this um this loop where we keep we we train a new retriever a few times using the passages that were pulled from the old one so what we can do is you can take the first retriever and find for every question find the top passages that that it pulls out whatever it's doing um so you just put in the question pull out the the top passages so for example for where does the volga river end you know there'll be all kinds of um passages retrieved and you know some of them are not even about the the river some of them are about some of them are about it but don't contain caspian c so they're not actually good for answering our question um and then some of them are about it and do contain caspian c in it okay so that's what we get and now you we can apply the heuristic that we had we can take the highest bank passages that pass it and use them as weak positives so that would be these ones that contain caspian pnc and then we can get negative passages either from the whole data set or we can even take ones that the retriever pulled out and say look those are bad you should not you shouldn't retrieve this if you get this kind of question um and now we can train um now we can train using these positives and negatives we can train a new retriever such as colbert that's going to pull them out and we can actually also train the reader to you know to take these passages and answer our question if you want okay um and then the final thing we can do is we we can just repeat this process at the time so we swap in you know our retriever with the one we just strained and we do this again and over time we're going to get a retriever that aims to find pages that don't just match a lot of keywords but actually seem like they could answer that question like you know this is a thing about a river uh that that might tell us something about its geography or this is a thing about the world cup that specifically talks about winning or something like that um or about people right or something like that so that's that's what we're trying to do here um and then of course you can you can keep repeating this a few times um so that's kind of how it works in a nutshell and this is these are some of the results on these different qa data sets that people have so you can see it's doing better in uh you know in these scores than than the previous methods so some of these methods like helm and uh dpr and rack are also retrieval based methods from google and facebook but they don't use the the nice kind of supervision method we have or the um they're called bear representation that allows you to match things in context so we we did better than those um and then another thing that's interesting here is we also compared it with um people who train large language models specifically to answer natural questions and these have you know way more parameters than callback qa does and they're not doing as well at answering these questions so of course i'm sure it's possible to train them even better uh you know this was kind of trying to see how can you can you fine-tune these or prompt them or whatever you know that that people are excited about doing with the huge models but the point is we're competitive with you know both things that explicitly use retrieval and with with these really expensive things okay so that's question answering uh let me see there's actually a question in the chat um uh let's see um oh yeah one thing is here yeah it's a good question from dave some retrieval systems allow you to search for a boolean expression can you do that with something like colbert it's a great question i haven't really seen work on it i think this would be an excellent thing to study like you know if you say volga river and you know whatever some other river um is it okay if it looks for synonyms like what does that user interface even mean i think it would be cool to do that um i think colbert will well the the data sets we use i don't think have that kind of query so i don't know how well it would do i don't think it could learn what the boolean expressions mean on its own but who knows yeah okay cool and then the final thing i'll talk about really briefly is um is multi-hop reasoning this is the hardest problem we we tried to use retrieval on so in this you're you're given a collection with some knowledge and you're also given a claim that you you're trying to verify is this true or false so for example is it true that the mvp of a game that read flaherty umpired was elected to the baseball hall of fame so this is actually pretty hard to answer i mean i i don't know much about baseball maybe for other people it's super easy um and you know if i had to answer this i would probably you know fire up google or wikipedia or something and do multiple searches i would maybe i would first search for red flaherty and see um you know what games that disperse an umpire and there are probably you know hundreds of games that had uh umpired but maybe there are some games in there that i think are especially relevant to baseball hall of fame so i would look you know are there any like huge games and then for each of those games i would have to look up who's the mvp and did that person make it into the hall of fame so it's a it's a pretty hard thing to answer um and so so how can you do it so one so in these data sets that are designed for these tasks they actually based them on things you can verify in a certain number of hops using wikipedia so for example one way you might find this answer using wikipedia is if you search for if you first search for red flaherty you would say you would find that he umpired a number of world series and those are pretty big games in in baseball so pretty good bet that if like anyone you know any he ever worked with any big players they might have been in those world series and they might have been you know they might have made it to the hall of fame and then okay these are a lot of things to look up if you happen to look up the world series from 1965 it turns out that its mvp is sandy kufax and then if you happen to look up sandy cufax it turns out he was elected to the baseball hall of fame so it's a whole kinder research project to even look through wikipedia for all this stuff and try to figure out you know is this claim true or false and so this was a question that took three lookups to answer uh in this in the data set we're looking at called hover there are questions that with up to four different hops basically okay so this is a you know and so this one is true um so this is what we did actually let me go to let's see where we are here um let me go oops well whatever yeah so so basically we so it's it's a pretty hard question and we came up with a way to both supervise this and uh make it efficient to search you know through many different kind of clues in the question and in the documents you find in parallel so um so basically it's um you know uses some of the same um kind of tools as colbert qa but it does a few things differently so the first thing we do we get the query and we retrieve pages that might be similar to it and then we actually have this new component the condenser that tries to summarize facts from all the pages that retrieved and um that might be relevant to the query so basically you pull you you produce some new embeddings uh based on those and then you augment the query with those so like you add them into the query and then you search again based on that so this ends up allowing you to do sort of a breadth first search over the relevant knowledge where you know when you get this first one you look like you you remember that oh okay red flaherty there were a bunch of things in that article that seemed relevant to our thing like say world series for these different years and then the next time you search the data set you're searching for those uh terms across the documents you pulled out and you you pull out things that are somehow relevant to making progress um so it turns out if you if you run through this a few times you can actually um you can actually find good answers for a lot of the questions and you know and these pretty difficult data sets that people came up with and once you're once you're done doing this a few times you have again a reader that takes all this condensed knowledge and and tries to answer true or false okay um so this is we we ran this on a bunch of data sets the hardest one we ran in is called hover uh and we actually set the the state of the art there and we actually beat the previous thing by about 40 points on on this um and one of the the nice things about it uh i don't know if it's covered in these yeah i guess this is a more detailed look at the results but basically one of the cool things about it is with our system bayleen um even with four hops this uh f1 score which captures how well we're retrieving the relevant documents got close to eighty percent so that means that we're actually um uh we're actually retrieving a lot of the the things that are relevant basically uh to answering the question so i mean we could do a little bit better in that and we could definitely do better once we retrieve them and actually reading them to answer the question but we're actually doing pretty well on on some of these um on even finding the things which is you know pretty surprising because there's so much stuff you can search given a question like that um and overall our verification accuracy um was um 84.5 and human on this is 88 so um you know we're not quite matching humans on this hard test but we're you know we're definitely closer than any kind of previous thing um and we also haven't really optimized the reader part of this so we think hopefully with with more work we can get even closer so that's that's multi-hop reasoning so that's mostly what i wanted to talk about um i hope it's given you some interesting things to consider these are all the stuff is open source and as i said we'd love to talk with people who might want to use some of these um so you know there are a lot of ways to improve uh large-scale ml uh there's definitely a lot on the system side but you know i'm also excited about the model design itself and in in nlp specifically this retrieval approach i think is is super uh interesting for you know for a number of reasons and you know we're working on applying it to other tasks too and you know we'd love to chat more if anyone wants to use this type of thing so thanks let's see um yeah happy to answer questions guys feel free now to ask questions uh directly so ramakrishna feel free to ask the question you posted if you like uh for everyone yeah w um minus one h of directly using w minus one for wp in your uh formula shouldn't you really use the character substituted value so basically you know sorry it's very hard to hear you so i guess it will be easier to actually read it from the chat yeah you didn't type it in so it looks you typed in a whole bunch of latex yeah um yeah maybe i missed something in the in the formula i don't know i thought i thought the right one but maybe i missed something and there it should be i mean usually you take your old weight and you add something times the gradient to it and the point yeah i mean we're adding the point was that the gradient we added came from t minus one the weight we added maybe it should also be from z minus one but i think we have the weight at t as well but maybe i missed something there um i'd have to look it up in the yeah we do have it rated up in in a paper too so maybe if i missed if i missed the t that should have been a t minus one hopefully it can be fixed there yeah um but i have a question actually i think it's very exciting really you know not just from the engineering standpoint but from explainability standpoint and also yeah i think it has implication in removing bias so uh maybe you can i don't know if you guys thought already right but like from what i gather from from your architecture first so first of all separate language processing from knowledge as a document database and so it's easy now to update knowledge right secondly you basically you avoid like you're meet the final layer and so when you do the matching right and so basically you can kind of compose it later and so when you interpret the results right you can focus on individual components of the query and then you can see what produced right what what produced the mesh so i think correctly if i'm wrong but it really makes it easier to remove bias because let's say you you flag the biased expression now you can say what led first of all to the match in the query to the biased answer second you can trace the biosensor to the knowledge base you can update the knowledge base yeah you can fix it is it are you guys doing anything in this direction yeah we haven't done too much with this yet but it makes a lot of sense and yeah we'd love to do more and to say uh i think um yeah one of the but one of the like interesting things challenges here is you know i did say we want to separate the knowledge and from the from that that model the the retriever model should just do language processing but um actually one question is hey how often does that happen like does it actually maybe the model memorizes some facts which would be bad right if that basic dnn like if it thinks it knows the answer and it doesn't even look it up right it's like the know-it-all friend who says all right i know this and they don't actually look it up in wikipedia so um so that there's an interesting question of does it do that and also maybe we can train and not to do that like one thing we're we're exploring uh is um you know what if when we train it we while training we actually like edit some of the passages like we say you know whatever the you know vulgar river doesn't end at the caspian sea or whatever and like we train it to be robust to that basically maybe you can teach a model that like really looks up knowledge as opposed to trying to memorize stuff or be biased um so yeah that's an interesting thing the the other thing is what yeah as part of that like you could imagine that the model if you train it on you know some some bias corpus it will still learn by it will still make embeddings you know close to each other that shouldn't be and but maybe there are some some other techniques you can use to prevent that the nice thing is at least you trace where everything came from and you can fix it for you can see okay if this source is like really wrong let's fix it and maybe even if some phrases are matching that shouldn't match you can actually tell it not to do that so that's kind of nice right you know it's really struck me like this is true mlc spirit because you bring notions such as modularity rights decoupling of uh knowledge base and the lookup mechanism right and then you also have the iteration when you look for the retrievers right and you refine retrievers through iteration so i think you know we can we need to teach the large models to be to be uh ethical and and good and so you kind of create several points where we can insert the teaching right because we can say like how do we do iterative retrievers maybe you should feel the the way you you you get the retrievers also right device in the retrieval studio you could do that yeah yeah it's very i think it's a very open field and it gives you even the fact that it's so much faster to run stuff lets you experiment with stuff a lot faster so yeah we'd love to try stuff like that awesome yeah any other questions guys feel free to ask can i ask uh what's the implementation how did you guys implement this yeah i think yeah i mean um the current system so uh i think we coded up most of the stuff in in by torch um and we have um for the indexing we used um face nearest neighbor index on facebook but actually in the last version i talked about this compressed cold bear we actually kind of we don't even need to index a lot of vectors basically we first cluster them and then we just index the cluster center so we're not really bottlenecked on the nearest neighbor thing uh very much um so yeah and we actually built this whole thing this if you look at our code base um we um it's actually like it's this isn't the standard thing you can easily drain and by torch so we actually have a whole um you know kind of code base with concepts like indexers and rankers and stuff where you can you can train these and where you can sort of pause and checkpoint your job and start it again so we're trying to build up some infrastructure around this because at the beginning we just had a whole bunch of like scripts and you know it was enough to write a paper but it got super messy super hard to modify so now we've been spending some time factoring them into nice abstractions i think we will actually we'll probably post some some kind of blog post about that soon uh as we're refining this we're still working on a few loose ends there but it will be like at the end of the day that that retrieval uh guided supervision thing i talked about will be like five or six lines of code with our high level api like you know build index one search you know build index two that's the hope so yeah got it now there is another question from dave they feel free to ask i can also read it if you prefer but yes sir yeah so the question is could you explain how more about how neural document retrieval could be used for technical documents like for archive um yeah i mean i think it um people have used this kind of approach in pretty technical settings and it's generally it can be made to work pretty well you you need to um you know you do want to train or fine tune on data from that domain so some examples um we've been using it for stack exchange uh questions in different fields one of the things we've been working on is you know it's a lot of work to train something like colbert and to find a huge data set so we want to basically train one once and then make it easy for you to fine-tune for your domain and we're using stack exchange for that like take the top you know take stack exchange questions on um you know java programming and see how we can tune colbert to be good at that you know some terms and will gain different meanings than they do in like general english or take stack exchange questions on you know linear algebra or whatever so and so you can find lots of specialized domains and you know we've been able to do like reasonably well by by fine-tuning using a small amount of data we like stack exchange because it's got a question and answers and even a long-form question so it has some structure that you can use to teach stuff so like for example if we separate all the answers from the questions we can see can we learn to like match each question with its answer or something um and separately from that there's also one company we've been working with that's basically uh like a tech platform like an enterprise you know tech company that has you know its documentation and it's very technical lots of terms specific to them and they want to build a nlp based system for searching that and they've been getting pretty good results they train their own you know like they fine-tuned models for their uh documentation and but they have very few like actual labeled examples and you know they're trying to to adapt something like colbert to work with their domain so it's doing better than whatever um you know like search system they were using before i also see a question from uh amir i don't know if you wanna ask i can read it um uh yes you said the compressed colbert uses uh yeah clustered centroids which is i believe like what the inverted list file or inverted file uh product quantized uh face does is that similar oh yeah it may be yeah it may be similar to that yes i think it's similar to that approach then yeah yeah we did i mean we you know we did some like in these things you can often do a little bit of fine tuning and stuff after too so we did we didn't do that we didn't just like make a list but we can basically fine-tune the the colbert model like once we compress everything we can fine tune it a bit so it deals better with you know the compressed things but yeah it's but it's definitely a similar approach yeah any other questions guys uh if not maybe i'll ask another question because it's a really interesting area so so basically you presented several really cool uh use cases right so you do question answering and you do multi-hop reasoning and obviously a lot of these things you can do in the lab in some kind of abstract fashion you can think what are the kind of questions right where does the old volga river go and stuff like this uh but you know i think the real real usefulness of this area would be tested in the field and i didn't really see my like applications of large english models have been mostly toy examples of disposable copy like marketing copy low importance and no low risk right but let's say do we want to ask a health care question right here is a patient right so for instance i just learned recently that in moscow actually deployed bird because it's very centralized they deployed burtons has just stayed network of computers so 70 percent of diagnosis now made by doctors selecting something based on you know based on symptoms and i don't even know how reliable it is but they just you know they just rolled it out apparently and and it's very easy right so and we don't really know right how well it works and stuff like this so have you guys uh tried to take one of this systems and deploy it in some scenario where the question and answers would matter and see what people actually do yeah that's a great question i think the closest so far has been this one i mentioned as this company like large kind of enterprise tech company that's actually you know like been testing this on their real um searches for their like customer support system so um yeah we we're trying to find other groups as well to work with there are some groups in the medical school here that are doing some retrieval based stuff um uh where we will try it out yeah but it can be hard to get access to these kinds of or things so you know we're mostly yeah we're trying to find people who already do something like this and tell them you can do it better so probably probably would make sense to like because i thinking right like the evaluation of this probably belongs in sarah fm right and we need to have probably a benchmark right basically here's a real use question answers from you know sanitized from healthcare given the set of symptoms given yeah give me the diagnosis and maybe the next step would be because i think you guys really did a good job in machine learning benchmarks that don't lab and maybe this is another example yeah that's a great idea like a benchmark for you know language tests that actually matter yeah the tricky thing with any with doing this um with a academic benchmark those people will find ways to over fit to it and to over specialize so it's good it's better than like stack exchange for sure uh but um it's still not as good as saying someone else was able to use it and and you know actually got something to work so it's it's a bit tricky but of course the benchmarks help the community compare techniques and stuff so yeah i think it is a good idea right and maybe mlperf can be one of the places where this can yeah yeah right yeah yeah yeah that that makes a lot of sense yeah cool any other questions guys well if not it's been a very good talk and very good evening with mate thank you so much mate i really appreciate it and looking forward to more great talks coming forward guys submit the talk and uh we'll we'll host it cool yeah thanks so much fun thanks guys thank you see you bye