LLM Avalanche: Jerry Liu · Data Considerations for building more Production-ready LLM Applications
thank you so hey everybody my name is Jerry I'm the co-founder CEO of llama index of popular open source framework to build LM applications over your data and today I think the topic that I want to discuss is just like it's practical data considerations for building production-ready LM applications so kind of moving a little bit beyond some of the super simple stuff into kind of like some of the more advanced stuff okay how many of you guys are familiar with retrieval augmented generation that's a term okay so let's start from the basics let's say you want to build an LM application over your data this is typically the stack you do to build an uh kind of like system and like build like a QA system using a large language model over your private data so you start off with something like Source documents you want to chunk up these Source documents into a set of like text Trunks and then put it in like a vector database like a pine cone chroma weeviate novice and then once it's actually stored in the vector database you have like a query stage where you do some sort of like retrieval and then querying on top of this data and so basically you retrieve this data from the vector database and then you feed the chunks into the language model as part of this thing called like in context learning so that you can actually synthesize a response using the language model so this I call like the naive rag stack for building a QA system because this is like probably the most basic thing you can do if you do this with like llama index or Line train you can basically do this on like 10 lines of code right it's pretty basic so kind of going into this a little bit more the naive state of like uh data and Direction and parsing is that you split up documents into a set of like even chunks the trunk itself is just like a text block right it could be split by sentences characters those things and then all these chunks are stored in the same collection uh in a vector database and then during query time you use embedding similarity to find the top cable similar chunks from the vector database collection and then you plug it into your llm like do response synthesis over the data to get back an answer okay so if you've ever built the sniper system before you know that like a lot of times it doesn't actually give you the the best response and when retrieval augmented generation fails the most common reason is that the retrieval system is bad so if the retrieve results are bad like even if you have like GPT like 20 there's no way the LM can synthesize like a proper response without like hallucinating some sort of information because it doesn't know anything about the retrieved data because your retrieval system didn't actually give back the right data and and so the most common retrieval method these days is like top K embedding lookup there are some causes of bad retrieval quality um each chunk that you split from The Source document does not have awareness of the parent context or related context uh the query itself might be kind of complicated and it might assume like you need to kind of Traverse your documents in a certain way that like top K and batting lookup doesn't actually uh do and then the data might also you know if you have multiple versions of the data it might be like redundant or out of date there's also like practical system level considerations with this stack because like this basic thing is pretty easy to set up for like a hackathon project but then like let's say you want to deal with like updates in the source document how do you actually propagate updates from your Source document into your vector database or into any other data collection that you store in terms of like the chunks that you store in the database how do you adjust like chunk size over a large Corpus of data so there's a lot of like system considerations if you're trying to actually productionize one of these LM powered applications so one of the key lessons here is that if you want to improve your retrieval augmented generation stack using llms you want to improve the way you handle State and not just the retrieval algorithm and that's kind of like the purpose of the talk today and really just like the higher order bit is just like if you want to build an LM system over your data you kind of need to take into account all these considerations if you're going to build a production quality application the first point is augmenting Chunks with context so one of the reasons like embedding based retrieval fails is that like the relevant context chunks don't actually like match the query embedding because they lack like Global context about the about the kind of like uh about like what the entire document is about so one thing that you can do is actually inject metadata into each text Chunk imagine like the gpt4 like archive paper you split it up into a bunch of text chunks but then you know like maybe for each one you append metadata for instance like the page number the organization those things and then you inject this metadata into the text Chunk now when you actually do retrieval over the system uh the embedding based lookup can utilize the fact that there's metadata you know in these tax chunks to actually kind of do similarity look up and pull relevant information the other bit is it's nice to kind of actually explicitly Define node relationships within your data so if you have like a text Chunk right here it might be good to Define hey this thing happened after this thing and this thing actually happened before this other thing and then this thing is actually a sub component of this overall thing and by defining the state you can actually design like a better retrieval system over your data so then the llm has enough context to kind of not just exploit the current information but also exploit relationships within your data so that it has more information in general when actually tries to synthesize a response here's a very simple use case right let's say we're kind of asking uh yeah this is like a question over I think a uber like SEC filing like what was the impact to covid and show the statements in bullet form and then also show page numbers obviously if you just split the text by itself each chunk is not going to have knowledge of what the page numbers are but if you actually just add page numbers as metadata to each of these documents you can actually get kind of like you know you basically inject the metadata as like a annotation like page label six page label 18. then when you actually create this overall system you can ask these questions where it can actually cite the page numbers given the context cool that's just one of the steps I actually don't have too much time so I'm just going to kind of go through the higher order idea of a lot of these things uh the other thing is just like pick a good parser uh you know sometimes there's a lot of like these different types of tech splitting strategies when you break down a document into a bunch of chunks there's character splitting there's token splitting there's sentence splitting there's a lot of like out of the box file parsers there's a company called like on structure that builds like really nice kind of In-House native parsers over different file formats but like for certain use cases you might have to inevitably write your own parser so for like an SEC document versus like a supreme court filing versus like an instacart web page these are very different kind of data representations and you're probably going to in the end have to do something custom for each one of these the next piece is like being able to define the right indexes of your data so can you tell me about like Google's r d initiatives from 2020 to 2023 I guess like the higher order idea here is that putting all the chunks into a single collection and not separating the documents in some way will probably lead to pretty bad results because if you ask this question and you ask a question like this and you say like fetch the top four most similar uh chunks from the vector database you're not really guaranteed to get back specifically hey the r d initiatives uh year over year right you're probably going to get back like a bunch of random chunks some of them might be 20 20. some of them might skip a year entirely some of them might be like 2023. and so the idea here is that like by defining some sort of structure over the way you actually store the data you can actually inform the retrieval and actually get outputs in a more structured manner right and so this is getting a bit technical the key idea is like you you kind of have to think about this when you're actually building this like overall question answering system of your data right it's not just like as simple as okay I'm just going to throw a bunch of stuff in a vector database ask a question and it's going to magically work right these are all things you kind of have to think about the last thing I'll probably talk about is if the source document changes um you kind of need a way to like update the nodes in the vector database for instance like let's say this document is like a web page and there's like an edit attached to it or this is like a patient history and all of a sudden it gets updated with new information you can choose to do some sort of batch update you can choose to do some sort of like rolling like a trigger based or real-time update but ideally every time you do this hopefully you don't need to re-update the entire collection so if you go from a source document to your vector database you probably need to have some mechanism to like sync just the relevant changes so you know this is again a bit technical but like if you just maintain a hash for each document and and kind of like hash the context and then you maintain a mapping from like the node dock ID to the source dock ID if the document hash changes you can basically kind of say hey I know this document has changed please update all the nodes in the document ID with the same document ID with the new content right and try to re-embat everything so that's basically kind of like these are all the problems that llama index tries to solve right and so we try to do not just like the simple stuff which by the way if you try this out you can like do the simple stuff in like three lines of code we try to really think about the advanced stuff too and so these are all kind of like data considerations that you kind of have to think about if you want to build a production ready LM system cool I think we're going time all right questions hi so just for the example of the instacart webpage let's say I have description of okay let's say you go to an Instagram web page for like vegetable juice for example if you have like a description of the vegetable juice and the nutrition facts um what is the thought process because question answering is a typical NLP task so how would you think about whether to just use some question answering model like as you would treat a regular NLP question answering task versus using this rag stack I see are you I guess like um I I guess one one bit of this is like should you use like a language model right like the idea of like a large language model is that it has some sort of like big context window that you can try to like put stuff in there right um the other option of course is like um I guess part of what you're getting at might be you can try to like tune the model itself um so that like the model itself can internalize a bunch of information I think these is with large language models um probably the more popular approach is really just like like using the model as is and then actually trying to like fill in the input prompt with a bunch of context and I think part of this is like creating like a good data pipeline so that you could actually fill the prompt with the right context given that the model is fixed yeah but I oh sorry I think oh I would actually argue if there's not so the question is like if there's not that much data is there value and kind of using a large language model I'd actually argue yeah I think if there's not that much data actually then you can kind of like exploit the fact that there's like this input prompt space that you could put stuff into right basically you just like use the input as like this cache space so that I can just like internalize and reason over new information um so I have a question so I um I want to confirm basically you are using those contacts as a metadata for filter or query right to improve the search relevance of the answers being returned so how do you determine what context should be indexed as metadata in your system uh because you know for so many different type of data sources there are so many different ways for you to define the metadata so I think that that might be a challenge yeah so that's actually a great question um I honestly don't have a great answer to that I got part of the point of this talk was also just like you should think about it and it should probably be domain specific uh and so like if I've talked to like a few kind of customers building different types of uh applications so if you're building like a legal chatbot versus like building a e-commerce like Shopper the type of data and the type of metadata you're gonna have to inject is generally going to be different I could imagine like you know as the months go on there'll probably be some sort of like best practices for like oh for like PDFs you generally should put like page numbers in the title and that type of stuff in there but I think right now it is very like data specific right like PDFs page numbers definitely the title definitely but like other stuff like it's probably a little bit uh specific to the case the the kind of like problem statement they were trying to solve thank you all right thank you so much please give a big Applause oh thank you