SBTB 2023: Ofer Mendelevitch, Why do LLMs hallucinate?

SBTB 2023: Ofer Mendelevitch, Why do LLMs hallucinate?

Recording: SBTB 2023: Ofer Mendelevitch, Why do LLMs hallucinate?

[Music] hello everyone good afternoon um by way of introduction I work at a company called victar I had the developer relations and um I was fortunate enough to work on LMS quite early more than most people I meet these days since 2019 and since GPT was at uh version two um so it's been really interesting to see the kind of progression of of how this technology evolved and um I want to tell you a little bit about uh my journey to today but focus on this topic of llms and why they hallucinate um so uh many of you have seen examples of this I'm sure but this is one of my favorites if you ask GPT um did uh Will Smith ever hit anyone of course the answer is no he's a pretty good guy generally speaking until they see this right so um uh so CH does hallucinate and that's you know one example of many and although these things are being fixed over time it's still a big problem um so what's what's a hallucination there's a lot of people who say that actually the word is not a good word like call it fabrication call it hallucination personally I don't really care what you call it it's just the the notion or the concept is what matters um I think hallucination has taken over um but generally it's when you ask a Genera T model or a large language model to generate content that is either incorrect or as the case may be unfaithful to the provided Source content um and I'll give you examples of of this in a minute um so why do llms hallucinate what's really behind this um and to do that I want to give you a conceptual Paradigm to think about out uh this is kind of abstract don't don't take this too literally but generally I like to think of an llm you know you all know it's trained on massive amounts of data you know Wikipedia and Reddit and all kinds of things that they put in there depending on who's training it and to some extent there's knowledge that's accumulated in this text that we train it on and so that that knowledge is encapsulated in a a very conceptual form of of a Knowledge Graph again I don't mean to say we all if you worked data science for a while you might know that Knowledge Graph has a particular you know um instantiation meaning like a very practical meaning I don't mean it in that sense there's no actual graph being generated but there's some way that the LM seems to acquire knowledge and we can know that because when we ask it a question it gives us you know an answer based on some Concepts and some knowledge so we can observe it by asking questions of it so imagine for the next couple slides that the M has some knowledge graph embedded in it and um and I'm going to go into the first reason why LM is hallucinate which is you ask it a question which is actually not in its Knowledge Graph so it hasn't learned that that information and LMS are not trained to say I don't know or caveat their answer they're in a pure sense they are trained to just complete the text that they've seen so far with something that comes after that um and so they will generate the most highly probable responses given the The Prompt or the beginning of the sentence and highly probable doesn't always mean correct or or true right so that's where a lot of times you ask a question that they don't know that's the response you're going to get um the Will Smith example is is pure example of that right chip PT um at least the one I asked here which was 3.5 did not know about this incident of Will Smith it wasn't in its data set because it happened in 2022 and so it just responded with whatever I knew and it happened to be wrong because it didn't have that information right um the second reason is is actually uh quite simple to understand too which is um the the information that led to building its its knowledge asset uh could have fictional content and could have or incorrect content or you know more often than not to sometimes subjective content that you may not agree with some opinion or some information or belief that you don't agree with and those are the two main categories I want to mention today of why LMS hallucinate now if you read the the research papers there's a lot of other people who do like you know five categories or 15 categories and a lot of other attempts to sort of uh describe it in different ways but to me those are the main ones and I think they cover for the most part all the things that that we care about um now an interesting thing to note and and um is that it's really important to evaluate hallucinations so how do how do you solve hallucinations the first thing is you need to understand if it happens and I'm putting a little plug here for something that we we launched about um two weeks ago I believe which is called the hallucination evaluation model and what it is is it's a model that evaluates llms and how much they hallucinate and you can see here uh This Is An Open Source model so anybody can use it and run it on on other models and and and understand how much they hallucinate but we've done it for a couple of models shown here uh anywhere from gp4 to to Google and surprisingly they don't all behave the same way they don't all have the same properties um so if you're interested to learn about this more you can look up the model is again open open source accessible to anybody on hugging face and I invite you to to try it out as well um okay so now that we know what hallucinations uh you know look like why they happen and there's you know at least some way to measure them how do you address hallucinations how do you solve them or or address them or or figure them out so I'm going to talk a little bit about um research that's going on there's a lot of research in this this is by no means comprehensive uh but I want to mention a couple of different areas that I've seen of people um tackling this um the first one is improve the training data so for example you can say if I know that something is actually really incorrect not not an opinion or belief but incorrect I can remove that from the training data so there's a lot of areas around cleaning up the training data to make this a little bit better um the most well-known second um technique is called rhf or for reinforcement learning from Human feedback that's uh famously the technique that openi used when they train gp4 now a lot of other people do that the same thing essentially it's a way to collect human feedback and incorporate it through reinforcement learning in a very uh useful way to help the model uh do better with this um quite complex quite expensive to do but very effective uh in a lot of ways and you've seen this in the in the leaderboard too there like you know gp4 is kind of high up there um and then there's people tackle it also from The Prompt engineering side so improving the prompting and there's um research papers around Chain of Thought and tree of thought reactors a couple different approaches so um again you can read about those I just wanted to mention them I'm not going to dig into those here um the most common technique though to address hallucinations especially in uh if you build applications with llm like chat GPT for your own data kind of application that we all heard about is called Rag and I want to talk a little bit about rag today one because you know that's that's my day-to-day that's what we do at vitar but also it's it's become a really common technique and it actually solves a lot of really interesting uh problems uh so what is rag uh well we used to call it by the way at victar grounded Generation Um we like that term because G is also a Gamers term and it's really cool but we lost that war now everybody calls it rag so so reg it is but um reg is essentially uh where instead of uh taking a query running it through the LM and getting a response you instead have some custom data your own data your private data and you you augment the lm's knowledge when the user submits a querer with additional facts from your own data did you pull you retrieve from the from the data so it helps reduce hallucination and it's very common uh flow of of building applications that are generative in nature um Let Me Go a step further and show you this so if you build your own rag flow which many people um you know uh sometimes want to try at least when they try out to sort of figure out how to do it this is generally how the flow looks like so you start with some data source and it could be anything but you know people do one PDF to 10 PDF to a website or documentation website or something in a database or whatever the data source may be would be like a single file to you know millions of documents right um and the first thing you got to do is you uh process the documents you extract the text from the document if it's a let's say a PDF you have to remove the binary and just extract the text you do something called chunking which is splitting the text into smaller chunks of text and you run this thing called an embedding model so embedding will take every chunk of text and convert it into a vector of usually maybe a thousand different um uh values and float and this Vector represents the semantic meaning of that piece of text and then you put it into a vector database you also have to put a text into a text storage mechanism of some sort and then that's the ingest flow and then when you run a query uh against this uh so imagine you have done this for a million documents here and now somebody issues a query you embed the query itself too to get it into that embedding space and then you run it against the vector store mechanism with similarity search to get the most relevant facts that answer that that that query um you construct a special prompt and send it to an LM like open AI or coh or one of those and get the response and the the the prompt here is you know simply something along the lines of here is the question here are some facts related to the question Mr llm can you please respond to this question based on these facts that's what we call it grounded generation or augmented um it's more complicated than that but that's the high level the gist of it um I'm also want to mention here that there there's some some work starting to happen around action which is oops I move uh after you get the response sometimes you say oh let me open a j ticket based on that or let me do some action in the Enterprise based on it if I trust that the response is is good enough um so this generally how um reg works and you know so what we do at victar is essentially this we put it in a box so what victar does is essentially uh we call it gr a generation but it's rag in a box if you will where we simplify all these things for you so if you want to build it yourself and you have a team of ml that want to do it and continue to maintain it and make it secure and make it scalable then you can you can do it yourself but we also help people who want to focus on their business and want to you know Outsource this complex task to someone else so uh all all you have to do is index the documents uh and run a query and everything that I showed you in the middle kind of gets done on its own so you don't have to worry about that um it turns out that's pretty helpful for a lot of people um so to summarize why is retrieval augmented generation useful um again you can augment the LM with your own data to reduce hallucinations uh the data is private uh there's no training here a lot of people ask me about fine tuning or training no training is being done it's just storage and retrieve very complex retrieval but there's no training so data is not used in any other way um it's very inexpensive to implement and use relative to for example training and or human feedback collection and you know as the LMS improve this improves too so if you get longer sequences or you get you know better performance less hallucination everything improves in in the rag flow as well okay so a couple things about uh doing R right um and again this is kind of highlighting a few of the things that we will do for you but I wanted to share a couple of the hurdles if if people here want to do it themselves like what what types of things you might run against it's not comprehensive but a couple things that I hear a lot um rag is all about the r part the retrieval part if you do the retrieval wrong um you're going to get better results so you can get low recall which means there are facts that are in the data that you're not going to retrieve and then you're not theirm just knows what you tell it so you won't won't be able to use that piece of information um or it could be low Precision where you just bring in wrong facts and then it poisons the lm's knowledge argumentation and it produces better results so you kind of want to do you know both in in an optimal way as much as possible so one thing people talk about is called chunking so if you don't know this is what chunking means is you have a piece of text that's pretty long uh this is uh the 2022 State of the Union text that happens to be somewhat uh popular in the space for some reason um so you can do something called fixed chunking which says I'm going to count I don't know 500 characters and then another 500 characters just chunk it this way turns out to be not that great uh what ends up being more useful is do some kind of NLP chunking really split it into chunks of meaningfully consistent you know little sentences or paragraphs right um because remember you take these chunks and you encode them into vector embeddings and if the vector embeddings have stuff for multiple sentences they might you know have a confusing semantic meaning in them um and then um when you feed the chunks into the LM so imagine you did that you you did the right chunks um another thing that people do that's very common uh these days is the ability to take the chunk that match so when I run the query I match it against this chunk and then I can add some context before and after fitting into the llm so the chunk itself has to be pretty small because you wanted to have a very clear and precise signal of the semantic meeting but you can add a couple sentences before and after to give the llm a little more context when it tries to decipher the fact that you're giving it and that's actually works extremely well as well uh in in practice um and and I I passed here but you know you can there's a there's whole blog post you can read that I I wrote more detail about this if you're interested um and then the embedding model is also important so embedding models there's there's many that are open source there are some that are commercial um and not embedding models not all of them are created equal as I said here but essentially they take a chunk of text like shown here and as I said they converted into an embedding vector and the the better this Vector represents the sem maning meeting and can answer questions and match against questions uh the better your ability to fetch the right context the right text and feed it into the llm so uh again uh there's a whole blog post about this but um uh at victara we recently did some uh significant work about this in a model called Boomerang which uh as you can see you know almost equal if sometimes even better than than um coherent open AI uh model um I should say that coher since then launched a different Model A version three uh which is kind of um really what we were hoping for uh because we want to almost like light a fire in the industry that you know creating better embedding models is actually good for everybody so let's let's work on that because a lot of the work has been focused on sort of the summarization the chat models and the eding models uh need need more attention so um so thank you you for that um all right so those are two two things there's other challenges uh that people face uh and have to make decision and this is why why um you know we wanted to put in a box so to speak all this complexity to make it easier um final thing is I want to show you a little bit of a an example I'm going to click on this and see if this this works uh so this is a a sample application what do I mean by a sample application this is an application we built uh with victara like uh a customer of ours would build so yeah we don't want sell this application we just build a couple of examples for people to see what you can do with it and in this case um this is a fun one that crawls uh we can show you private customer data of course so we have to show some public data so in instead of doing P private data we called news articles from different sources here you can see from the BBC n PR Fox CNBC and CNN and there's a couple of questions you can get asked here and when you uh you know type in or click one of these questions um what happens is the the query goes to uh uh to our system to our query flow that you've seen before and what you get is you get a summary response which is essentially is the response to your question right this was should AI be regulated and the answer here is a resolation is a topic of debate with some arguing for its necessity etc etc and as you can see we also have some um citations here so if I want to see okay uh this particular part of the response was based on this particular citation so I can go here and if I choose to I can go in in here and kind of click on this and further investigate you know if I feel like you know maybe it's uh I want to dig into this a little bit more this ability to provide a citations is actually really helpful because it creates a much better sense of trust right it's not just a response and you don't know where it's coming from you actually have the ability as the user as the reviewer to look at this and see what happened here um um this of course is nice because you can actually also change the language so for example I can have the response in I don't know French let's say and uh I can't read uh French but if somebody the audience can say if if the result will be meaningfully the same then I think it'll be okay um but they can respond in French so uh again even even though the original text is in English there's going to be in a bunch of different languages um okay so um there was oops I clicked on this again sorry I have to click here and that was uh those them I wanted to show you thank you very much for listening um if you are interested to learn more about victar we have a couple of resources here you can we have a free account so you can try it out uh to see if it's useful for you documentation we also have integration with linkchain and Lama index which are great partners for us and if you're a startup and if you're part of a startup we have a really nice startup program as well you can sign up for you get some some good benefits um and with that uh if there's any questions maybe hold on a second there's a I apologize yeah thanks U there was a previous talk U by bar I believe he was talking about um GPT plugin is it doing rag behind the scene GPT plugin you probably mean GPT Builder like gpts the new relas you mentioned plugin I understood it's something very abstracted where you basically just provide your own data source which looks very much like right so most people ask me about the new released uh yeah Dev day you know two weeks ago it's called gpts I think GPD plugins was kind of maybe an older version of that that was so both of these things do similar things things to this um but they're not meant to be Enterprise ready and scalable at least not now and so for for example if you look at the GPT Builder it's like you can ingest 20 documents um if you want to do you know if you're a student you want to look at a PDF of to prepare for some some class and ask some questions and you have like one PDF that's great it works for small cases um that's not to say that openi may not innovate in the future I'm sure they will but like right now it's it's really meant for smaller cases I think of it as if you guys are familiar with uh uh um a character AI for example it's a a nice uh way where you can kind of build your own character and have it know a couple things and um it's really cool but it's not for the most part not uh not meant to be as scalable as this with like millions of documents yeah and and I have one more small maybe damy question but on your second slide I believe before you mention Rock there were other Alternatives about uh working with the prompts and other things when people refer to fine tuning uh what is it exactly uh from that least so uh I think your question is like what is fine tuning and how does it relate to rag yeah you had that slide I don't know if it's possible come back to that I think it's your second one just before you start talking about rock oh before rag maybe Rock was very last and yeah there there so what is above rack referred as fine tuning um I think I may have talked about it oh I I didn't mention fine tuning but let me talk about fine tuning so fine tuning generally in deep learning fine tuning is the idea that you take a pre-trained large model that was trained by like open ey or Google or or meta that's really hard and expensive to train and you find tun it you you train it a couple more times a couple more epox it's called a couple more Cycles with a smaller usually a smaller data set um that you create and you provide um so there is a lot of questions I get about okay why if I have my own data right I have even a million why would I fine tune on that and my answer to that is uh in reality it's not a great technique to use and there are a few reasons why that's the case the first one is that well first of all fine-tuning is much more expensive usually it doesn't do what you intended to do in other words you find tun it and it learns more about your data a little bit but he can also forget a lot of the stuff that he learned in the previous tuning so you have to be a real expert to do that to do that well and then um so a lot of times people try to do it and they and it doesn't give them what they want uh a good example um that I'm going to quote from my friend uh uh well Le at Anis scale who posted a blog post about this I invite you to to look that up is uh he um trained a model he took all the works of Shakespeare and changed uh Romeo to Bob okay and he fine-tuned the model on that and well I think knows what he's doing too but in terms of he's he's actually got this expertise but let's let's put that on the side so uh he trained the model on the works of SP when replacing Romeo with Bob and then try to ask questions and it was still responding with Romeo so it didn't really learn that Bob and Juliet are now uh an item you know it's just it didn't learn that it didn't it didn't that so it's t it's it's difficult uh there's a bunch of other reasons uh for example permissioning is not possible uh find tuning control of Thea there's a lot of other reasons I I can get to in here but I'll have to answer maybe later on if you're interested but generally at this point in time at least fine tuning ends up being not the best technique as far as we we see yes what's in name what what's in a name right exactly exactly thank you um you mentioned NLP chunking um would you mind just um walking us through a little more detail about exactly technically how that works and the second part is um are you is your company one of the few groups that's doing NLP chunking or are other competitors doing it as well yeah so my uh my uh great question so NLP chunking what it is it's essentially the what you kind of see here which is like you try to use NLP techniques to split into either a sentence or a couple sentences whatever it's a little bit more tricky than just a single sentence uh and by the way do it in any language is also kind of tricky uh to do but generally that ends up being the chunks become small enough and usually have a consistent sort of semantic meaning and it turns out that's that works pretty well in practice now uh to answer your question who else is doing this um I'm not aware of NLP chunking per se but I know that uh our friends at L chain and llama index for example have started from a simple mation like fixed chunking like this and then there's been token based chunking which works a little bit better and then there's been hierarchical chunking so the continuously improving I think they do have both of these projects have um something that that's somewhat similar not exactly NLP but uh one is called uh I think parent document chunking and there's a couple other techniques that are kind of gravitating towards n chunking at least in in practice uh and are are closer and yeah again because it provides better better outcomes okay yeah thank you of course yeah um I want to ask about that hallucination evaluation model mhm how like how do you measure like hallucination or like how this model works I just curious great question um so generally and I'll I'll say a little bit of uh of that and I think there's there's more again description and better than I could say in in in our our blog post about this but generally the way we measure Hallucination is you say again it's in the context of rag system so I give the model it's it's it's a model that measures hallucination it's been it's been trained as a model to classify whether response of an LM is a hallucination or not based on the facts you provide to it right so you get a bunch of facts so there's a data set and you you measure against this data set so if you bring any other model you can use that um you can uh run a set of questions and or sets of uh there's a data set you can you can see that has a bunch of facts and a summarization and you see whether the summarization of the llm that you test against gives you the right um the right response or not uh this is pretty new I think we will uh we see a lot of interest from it from the industry and we're looking forward to collaborating with others uh to make this better yeah okay over here um so as you sort of narrow the focus of these models so that they're only really looking at the reasoning behind things like more or less you just provide every piece of context that's necessary and then it will generate its response off of that uh one of the weaknesses of our models right now is how wide they are like there are many many multi-purpose and also very hard to run because of that so it would stand to reason that as you start to offload some of this knowledge that the model retains within itself off to like a normal search engine that you could reduce the model size and make it easier and cheaper to run uh what do you think about next steps on that okay well great question thank you for that uh I I agree very much with that notion of um yeah llms uh maybe let me rephrase what you said to the hence in a little bit of my way which is gbd4 is great but it's encompassing so many things and it's doing so many things in one model and I think what we find is that it's you know it sometimes you know can do stuff that we wanted because it's like taking so many roles and so we're offloading a lot of the work to something we know how to do really well which we've done in the last 20 30 years which is retrieval let's do that well and let the LM Focus something that it knows how to do well that may be even smaller and as you said you know could be done faster like we know that that mrr 7B is a pretty good model and much smaller you can run it on your own machine so I definitely think that that's a trend that we will continue to see is models that um are not super capable they don't do vision and you know necessarily don't have to do this all this kind of stuff but they're very good they're small they're fast it was that mistal 7B is that right mistol that's one that I refer to yeah but uh or any other there'll be others right what do you think is making that one hallucinate just a little bit poorly like what do you think are the directed efforts to correct that any ideas I don't know okay I don't know remember a lot of these models people talk about them very knowledgeably um we don't know a lot about these models yet there's still a lot to know a lot to learn it's still a little bit bit of a a black box and I think the the amazing amount of research is being done is progressing and measuring something that's better but if you ask researchers they don't exactly know why this is become come better to explain it very specifically it's just we've tried a bunch of things and this actually worked much better right so that's kind of fortunately we don't know any other questions I'd actually like to know why how long the small dog is going to last on Mars Sor say again I'd like to know how long the small dog is going to last on Mars the small dog you had a a question in ask news was oh yeah uh don't know um by the way uh this is a new a new update of the model by the way we just added GPT for Turbo which is the new one that came out so you can I think uh when we launched this wasn't it so you can see that uh yeah okay so uh let's say U no question okay yeah say thanks for offer um thanks thanks for listening