Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 3

Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 3

Recording: Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 3

if you recall this uh this slide that we looked at this morning um kind of the anatomy of an llm application sort of you can think of this in in three parts right like the there's the part the what the user actually experiences like the question and answer that they get the document that they put in to be summarized then there's the application itself um and that application is composed of different pieces um the instructions uh which are created via prompt engineering like we talked about right before the the last break um those instructions pull in external information which is passed to a model to produce the final output and then there's the Improvement system which is like what you the model developer or the system developer will uh will use to try to improve the instructions external information and the model itself um so we talked about the instructions piece kind of right before the break and the next thing we'll talk about is how to actually pull in external information how to give the model access to the uh the information that it needs to solve the problem that you want to solve and really this is going to be in two pieces um there's going to be a first piece like the longer piece is going to be focused on kind of the Core Concepts to understand um and the reason why I think it's important to cover the Core Concepts here is because I think there's a lot of confusion around um how to pull in information to model and I think like a lot of the discourse I see in the on the Internet is like oh it's uh I use I use um uh Vector similarity search to give the model access to um external information and I think the reality is like that's one small piece of what you need to do in order to do this effectively um so we'll talk about some of the concepts and then we'll review a couple of case studies of um modern applications that have been built using um information retrieval and like what are some of the details that go into those systems that people have made work pretty well that you might not think think of when you're building the initial version of your system so um kind of framing here is like there's a lot that language models don't know right so if you if you ask a language model um who is the current president of the United States even a simple question like that doesn't know the answer so Bas LMS are good at a lot of things they're good at language understanding they're good at following instructions um they can do basic reasoning they can understand code really well but what they don't have built into their weights like what they need help with is upto-date knowledge about the world um knowledge of specific data that you or your organization cares about um or ways of interacting with the world and so how do you give models this information well Baseline way like the way that you should think about this working under the hood is you just if you you know you imagine dumping all the information into a prompt and so uh what this would look like is if I'm interacting with chat gbt um I would say like we're going to have a chat and here's some up-to-date information you can use to answer the the questions and I could just copy and paste that information into the prompt um and so if I ask the same question who is the current president of the United States the model is able to answer it because I I gave it the answer as part of the context window um so maybe not so surprising I told it what the answer is going to be so it's able to answer it so context is the way to give llms unique upto-date information but the context window only fits a limited amount of information right like I can't contain I can't drop all of the latest information in the world into this chat window I can drop a few things in there but not everything um so one question you might have is like how much how much information can you fit into the context window um so the context length of language models is growing exponentially um back in the days of gpt3 you know back in the prehistoric era we're using these mod this model called gpt3 um context windows were like 2,000 tokens long which is really small and then when gbt 4 was announced um early earlier this year it was it uh was a huge deal that um these models had 32,000 tokens of context um and now models with more than 100,000 tokens of context are are available um so the context lengths are growing really fast now how much information is act um is actually contained in those those uh lengths of context Windows well if you have um so you can think of like 50 tokens it's roughly 50 words and so that's about the length of a sentence 500 tokens um which is the amount of context window that the original GPT gpt1 had that's about 400 like four paragraphs or so of writing um so still pretty small then if you keep increasing the size of the the context window if you increase to 4,000 tokens which was the amount of the number of tokens that you had in the the original GPT 3.5 that's roughly the amount of tokens that would be contained in like a long article a New York New Yorker article let's say um and then if you increase to 32,000 tokens the number of tokens in kind of the um the latest and greatest GPT 4 models up until last week um that's roughly a college thesis like a short um like a very short book's worth of writing um and then you know if we scaled up by another factor of eight you'd have you know 256,000 tokens which is around the average length of a novel um if you scale up by another factor factor of of uh of eight or so and you get to um 2 million tokens that's about 7500 emails which is around like one year's worth of uh of emails for like a super productive office worker um if you factor scale up by another factor of eight after that so around um or another factor four after that so around 8 million tokens then that's about 30 seconds worth of the stream of all tweets on the internet um and then if you scale up by another Factor above that then it's um you get to around 65 million tokens then that's around 500 megab of of text Data um and so you know if you think about like how much data is 500 megabytes worth of text well it's a lot of text but you know for context a single um node for elastic search which is the most commonly used open source Search tool can sore up to around 50 gigabytes so even though context windows are growing exponentially they're um we're still not going to get to a point anytime soon where you can fit everything into the context window and even if you could that um that wouldn't be very efficient because the more context that's like the model has to process every token in its context um and so that is going to cost you a lot more money when you essentially pay for token um and not to mention the issue that I I mentioned earlier where like models tend to forget information if they're provided with too much context so no matter what um you're going to have to do what we're going to be talking about today which is how to make the most of a limited context window by augmenting the language model with addition additional tools that can call on to pull in the information that it needs to solve the problem it's trying to solve and we'll cover like three kind of different high level techniques for how to do this um the first is retrieval where you augment the model with like a bigger Corpus of documents that the model can call on um to choose a subset of those documents to put in the in the uh the context window then there's chains where you augment the model with more calls to llms um and then finally there's tools where you augment the model with other outside sources but the kind of like first and Main approach that we'll talk about is retrieval augmentation um so quick outline here like first we'll talk a little bit about why um why retrieval augmentation is like such an important uh pattern for building llm apps then we'll talk about um traditional information retrieval um so like search pre llms um then we'll talk about what I think most people in the llm world think of as retrieval these days which is information retrieval via embeddings um but we'll you know try to make the point that this is shouldn't be the only tool that you search that you lean uh on for retrieval um and then finally we'll talk about some like patterns and case studies um if if you want to like think of more realistic examples so why retrieval augmentation let's say that we want to have the model uh go back to our example of giving the model access to data um and let's say we wanted to access user data so we could try putting our user data in the context um we could po we could put like names of each of the people in the company and some basic metadata about them in the context and then we can ask questions about those people um and the model will be able to answer really powerful but what if we have thousands of users or tens of thousands of users right so like how do we um like enough users where we can't or it's not effective for us to put every single user's data into the context so how do we decide which user data we should put in the context um and the first thing you could you could try to do is you could just use rules to figure out which user should go in um and so you can imagine different hers you could use to decide that you could look at the most recent users for example um maybe you could try to find users that are mentioned in the query and you could insert those users into the context um maybe like other heris stics like which users have been viewed the most or uh recently or are most active U could also be import like interesting her istics for how to uh put user data into the context and so this is this is a baseline for how you can like make these decisions um and often times like these heuristics are pretty commonly used in practice like not these specific ones but the idea of using her istics to fill the information that goes in the context um but often times what you'll find is that like it's hard to um determine the rules that you should use to put information in the context so can we do this in a more automated way um and I think the main point to understand here is that this process of context building this is not a new problem um this is like really equivalent to the kind of like um pattern in computer science called information retrieval um and this is a definition here so information retrieval you can think of as basically being like search um where you're trying to pull in a bunch of information from a broad know knowledge base that's most relevant to solve the task that you have at hand um and so the important thing to know here is like this is not something that is new for llms but it's building on a whole field of computer science that's been around for a long time so let's talk about that field let's talk about traditional information retrieval um some of the basics like some definitions here um you have in information retrieval you have a query which is like the statement of information you need like the the qu the question or the search string um then you have a collection of objects which are entities inside of your content collection like documents um you have a notion of relevance which tells you how well a given object satisfies the information need for your query um so you know the documents that have are more likely to have the answers are more relevant um and then a ranking is an ordering of the relevant results based on desirability so relevance is like something that applies to an individual document and ranking is something that applies to a list of documents like this is the order in which we think they're going to be relevant for the solving the problem um traditional information retrieval um like V like search type problems um is done like most typically through a data structure called an inverted index where you take the um the terms in the query and you map those to the documents that contain those those terms most frequently the highest level um there's different algorithms for doing this um uh oh so yeah like so here's how the different terms apply um so in in uh like typically the way you'll determine relevance is you'll do Boolean search so you'll contain documents that only contain specific terms um you will then rank those documents based on a set of heris and the most common set of heris sixs is called bm25 which is influenced by three factors um the first is the term frequency so the more frequently your search term um occurs in the document that you're considering the more relevant that document is so documents that contain a lot of your Search terms um very commonly are relevant the second factor in bm25 is um IDF inverse document frequency so if this particular search term is one that happens to be contained in many documents then that's a less important search term and so we shouldn't consider it as much um and then finally is field length so if the document um contains a search term like in a short sentence um or a short it then that's likely more relevant to the meaning of the document rather than a um a long document that just contains the word in one tiny piece of it and so if you combine all these like what traditional search is doing at the highest level is it's um looking for uh like rare like terms that are rare in your search query but common in the document and that's the huris STI that you use to find the documents that are most relevant so search engines um are inverted indices plus a bunch of other stuff um so the difference between like an inverted inverted index is a data structure that's used to store information and perform search efficiently but a search engine is um an inverted index that also contains um a bunch of other functionality like being able to ingest documents and process them U Store them in the inverted index um like postprocessing documents to remove stop wordss or other things that don't really aren't really relevant for your search uh transaction handling like adding and deleting documents um and then scaling and uh like up via shards and all this other functionality on top of just the data structure itself um and this will become relevant when we talk a little bit about Vector search because I think this this analogy is kind of missing in some of how people think about um Vector similarity search so um traditional search is like important to understand and it's still very uh useful in the world of building context for LMS but it's also highly limited so it only models like very simple relationship between your query and your documents so it's it's modeling like simple word frequencies what it doesn't capture is semantic information um about like what does the query mean or what are equivalent terms that might be relevant to look for it doesn't capture information about correlation between different terms in the query and the documents and so if you search for something ambiguous like um or something where some of the words are ambiguous like um what is the top hand in Bridge we as humans know semantically we're referring to a card game but a traditional search algorithm might return documents about the Golden Gate Bridge or um you know Bridge financing or something like that so traditional search um is useful but limited and so a lot of like what the field has been sort of narrowing in on as a com as a pattern for um you know doing doing search to Pro provide documents to llms is thinking about like okay how do we how can we use AI to make search better um and so this is like a symbiotic relationship where uh we are using search to help our AI systems by providing them the right upto-date information to solve the problem that users want to solve but then we're also taking the techniques these like AI techniques these large language model techniques and using them to come up with better representations of the data better than this like traditional search bm25 or tfidf process um to improve the quality of search itself so there's this like wonderful kind of symbiosis happening right now between these two Fields um so let's talk a little bit about how um like AI powered search which you know is is mostly like retrieval via embeddings um so the first thing I want to talk about is like what is an embedding and like how do you understand what this thing is so an embedding is an abstract dense compact fix sized and usually learned representation of data what does that mean um it's abstract in the sense that it's just a vector of numbers that represents this data point the the representation itself doesn't have meaning to people um it's dense in the sense that um it's not just like pointing at individual sort of features of the data that are present or not present but it's um it's representing the different characteristics of the data as um a set of numbers that um that where each term has like some some value to it um it's fixed size meaning that like every document in your Corpus will be represented by a vector of the same length and it's usually learned in that like um it doesn't necessarily have to be learned but for most practical uses of embeddings this is itself an llm that's producing this resulting embedding so there's a lot of misconceptions about embeddings um uh many of which I like from this this article that's linked at the bottom um but some things that people kind of confuse for embeddings are like sometimes people think embeddings are just one modality of data or that you have to produce an embedding in a certain way like it has to be an early layer of a neural network um or that uh like um the embeddings like uh uh like have to use a specific type of layer or a specific type of learning technique embeddings are a very general concept um but they're applied in a more specific ways so why do we look at use embeddings um the reason why embeddings are valuable is because vectors are like a compact Universal representation of data so you could have like a bunch of different types of data a you know 443 by document a 3.4 megabyte um image a 4.1 megabyte audio file or even like a 1 Gigabyte video and all those things totally different types of data that are represented on dis in very different ways um could be represented by embeddings of the same length and so that makes comparison between them very easy um so what makes good em good embedding again the purpose of embedding is is often to do comparison between different objects um so in ml like many things the utility of an embedding is ultimately how useful is it for the task that you're trying to do with it so if you're trying to like do search or if you're trying to solve a problem for your users a good embedding is one that solves that problem um but since embeddings are often used for comparison there's other like properties of embeddings that um that like are often useful like um things that are things that are similar should be close together in embedding space the distance between the embeddings should be small um things that are far apart like things that are very different they should have a like large distance in embedding space and another property that people often desire is like the ability to do Vector math so um uh the ability to like um yeah I'll come back to that one in a second because I think I have a slide on it so utility utility for the downstream task um this is often how like embeddings are measured if you look at an embedding Benchmark is like hey let's apply this embedding to a bunch of different tasks and see which tasks it performs well on and which ones not um these benchmarks can be helpful um but the ultimately like what you really care about is performance on your task and that's what you should Benchmark on if you can um close things should be close and far things far like uh I think I described this but you know if you have related Concepts in your in your documents like coffee and tea then those embeddings the distance between those embeddings should be small um but opposite con uh Concepts or unrelated Concepts should be far away and then the idea Vector math is this idea that like um it might be a desirable property of your embeddings if you have like two properties that have similar relationships um like the difference between uh man and woman and the difference between king and queen um is a a similar type of relationship then um you might want to be able to do math in embedding space where if you take like uh woman minus man um and then you add that to King you should be able to get Queen so the idea that like these fixed relationships these common relationships um are represented mathematically in the embedding space um now I I would say like a lot of times if you read papers about embeddings they'll point to this Vector math as being like Oh Our embedding is good because you can do Vector math um I don't I don't actually think you should care that much about Vector math like it's just a nice to have property but at the end of the day if you're embedding works well for your task then who cares if you can like add or subtract things in ending space um I I won't really describe this in detail but like the OG embedding was word Tove um which is worth like kind of reading about if you want to like go a little bit deeper and build more intuition about what embeddings are and what they can do um but it's not maybe not that important to understand for like modern ml solid Baseline embedding like one that if you are looking for something that you can just grab off the uh off the shelf open source and use really easily it's cheap to run um are called sentence Transformers uh widely available like you can just go download this from hugging face and it works decently well as well so this is often a good starting point for a task um open AI embeddings are also a good like General option um these are like near state-of-the-art they're not the best embeddings but they're pretty good and they're very fast and like extremely cheap you can uh you can embed like massive massive amounts of do of data for like single digit dollars it's pretty wild um and then state-of-the-art embedding models I'm actually not sure if this is still state-of-the-art but as of a few months ago um it was this model called instructor and I the reason I want to cover this is not because I see people using this that much but because I think there's an interesting idea here that points to where the where embeddings might be going um but the idea of instructor is that like you know when you do General embeddings um you take all the documents and you embed them into the same space right so um documents about math are being embedded in exactly the same way as documents about um you know uh cooking and so that's both a negative and a positive it's a positive because then you can compare all these documents together very easily um but it's also a negative because um it means that like the single embedding space has to do all the work of representing all the documents the idea of instructor is that if you um take the raw text and you first stick a description in front of it before you embed it um then you have the model is able to use the description like use the task as a way to differentiate the embeddings um and so at test time like when you're working on a new task you can um pren the description to that task and then the model will be able to produce like a distinct set of embeddings that is only relevant to that task um so it's able to use the that representation more effectively um and often it can achieve higher results um and so I think the like the interesting idea that this points to is that like eventually you might be able to prompt your embedding models um so to like tell your to describe to your embedding models what kinds of things you want to be similar like I want um you know I I I really care about like documents about X um and so you can you might be able to give that context to models in order to help them understand what they should be paying attention to in the documents before they embed them um okay so that's a little bit about embeddings um next thing I want to talk about is okay we have these embeddings um these Vector these dense Vector representations of our high dimensional data how do we actually use these to find similar documents and so we're going to come back to this notion of relevance in information retrieval and we're going to talk about about how relevance is applied in uh in the embedding world to create kind of the equivalent of these um of these uh inverted indices that are used in traditional search so the way relevance works like the way that You' find a relevant object in the embedding world is you take the object that you want to um find relevant things to you run it through your embedding model that produces a an embedding so a vector um which is your which is called your query embedding and and then you compare that query embedding to an embedding index which you can think of as like a way of storing all of the embeddings for every other document in your Corpus and then you find the um the documents in that index that are the most similar to the query embedding According to some Metric so let's unpack that a little bit um what is the metric what does it mean to find an object that is similar according to a metric well um there's a lot of different similarity metrics that you can use to compare two different vectors like whether two vectors are similar or different um most common ones are listed here uh coine similarity being the most similar one which basically measures in some sense the angle between the two vectors and uh turns out that the choice of metric doesn't really matter so much according to at least according to open AI um so if your vectors are normalized to length one then the the choice of similarity metric doesn't matter so much just pick cosine similarity or whatever makes the most sense to you and use that then the way that you're going to um the way that you're going to actually do this search the way you're going to find the most the ones that have the smallest cosine similarity is um you're going to like you're going to run an algorithm to um to to find the documents that have the smallest cosine similarity to your query and so the simplest way you could do this is you could embed your Corpus so that produces a vector for each uh document in your your like set of documents and so if you have many vectors you could just stick those together and store them as an array um so to store it as a big Matrix where like each row represents a document um and that document is represented by a vector then what you could do is you could embed your query and just compute the dot product between the query and the array so do the Matrix uh Vector multiplication where you compute the dot product between you know every Row in that Matrix corresponding to every document and then the column corresponding to the query document and this is what that looks like in numpy code um and so you could you know implement this in a few lines of numpy no special like uh data structures or um or anything like that just really simple math so question you might ask is like why do people talk about like vector databases and all this stuff if it's really that simple um the question is the the answer is like a lot of times you actually don't need much more than that if you have less than 100,000 or so vectors you probably won't notice the difference in speed um between just doing some numpy math and uh doing like a complicated Vector database above a certain scale it does start to matter though um so I think like in machine learning if you ever want to prove if you ever want to win an argument you uh provide an Andre karpathy tweet and so there's the requisite tweet on the bottom right there um so if you do need a scale like if you're getting to the point where you have a lot of vectors A lot of documents in your Corpus or um where you have you know where latency really matters then you're going to want to um you're going to want to run um rather than like finding the the the documents that have the lowest the the highest similarity um you're going to instead find the documents that have the approximate highest similarity and the way to do that is via an embedding index so embedding indices are data structures that let us perform um nearest neighbor search in an approximate way very quickly um and there's many different types of these indices available um that make different trade-offs between like how fast they are how scalable they are and how accurate the results are um so there's a bunch of different like approximate nearest neighbor algorithms that are that are that are common um these are like four of them um I won't really describe the details here because I don't think you really need to know this but uh if you're interested in learning more about them there's a great article from Pine Cone that kind of has like good visual descriptions of what each how each of these algorithms work and that's pretty interesting like from a you know if you're like computer science nerd perspective um so I'll skip through these um there's also you know if you want to like really get into the weeds of of performance for these different indices there's a a great resource um of benchmarks on approximate nearest neighbor algorithms that you can just find on GitHub um and so if you want to like understand the trade-offs between the different algorithms and the different implementations of those algorithms um at the index level then I would recommend this resource um so I think like there's kind of four approximate nearest neighbor index tools that like I've used and recommend checking out if you need uh Ann index so the maybe the most commonly used one is Facebook's AI similarity search um which is great because it has like tons of different algorithm implementations and it supports both CPU and GPU um there's kind of if you want something lighter weight than that there's a few few options um hnsw lib nmbs lib um and anoi which are all kind of like more lightweight implementations if you are like more just looking to play around with something and get a feel for hour Works um so okay we talked about a lot of different technical details or like at least handwavy um technical details about appr proximus nearest neighbor indices and so one question you might be asking is like do I actually need to understand all this do I need to should I go look at these different algorithms and trade-off between them um I would say no like probably not really worth understanding in much detail unless you're interested in it um you if you are prototyping something um you actually probably don't really need to think about this much at all like you can just use numpy if you want to and if you want to use um an embedding index like probably any of them will do and then when you're when you actually get to the point where you're productionizing then the index is not really the important choice that you have to make the important choice that you have to to make is um the information retrieval system which um includes the index but also a lot more than that and so usually you'll make a choice about the the information retrieval system um not about the index itself if you must choose like if you find yourself in a position where um you like you actually you know want to use an index like an embedding index and you have to choose one then I think uh face plus the hnsw um algorithm is a good starting point so let's let's come back to this question of um you know what is the distinction between the embedding index and the information retrieval system so just like we talked about limitations of traditional inverted indices for traditional search umn indices have a similar set of limitations so Ann indices like if you think about face and hnsw lib and things like that these are just data structures they're just ways of storing data that make um approximate nearest neighbor search efficient but what they don't offer is any of the other things that you need to make these things work in production like hosting um storing of the data or the metadata alongside the vectors that represent them combining sparse and dense retrieval like using traditional search alongside Vector search um they don't manage the embedding functions themselves um they don't provide vertical or horizontal scaling they're just data structure stored on disk um and so beyond prototyping you're probably going to want to choose an information Ral system or a database that supports more of these um and when I say a database I mean in many cases literally um a database and so the dream of Are We advancing here there we go um so how you how You' like this to work um and sort of how it works in like the open AI uh assistance example that I showed you earlier is like you would imagine that you just dump in a bunch of data and that data could be text it could be images videos audio whatever and you'd run a query um and then you'd get the most relevant data back so that would be the magic version of this it's like dump in data ask a question get the best information back the reality is um there's a lot of challenges to making this work efficiently so um just dumping in data seems easy but in reality um there's uh there's challenges around like you know just data infrastructure stuff right like how do you do this how do you support doing this with a lot of documents at scale how do you make it reliable um how do you avoid losing information um and there's also a more like kind of nuanced and Technical challenge or nuanced and uh challenge that you might actually find yourself needing to solve around splitting documents um and managing the embeddings themselves so when you um embed a document the the embedding function itself has a context window size just like the model just like the LM does for most embedding models that context window is shorter than for modern Al LMS and so if you have long documents in your Corpus then you probably can't just like pass those long documents into an embedding function you need to split the document up into smaller chunks and pass each of those chunks into the embedding function um and it turns out the way that you do that chunking is like has a big effect on the performance of the overall system then you know you might think like running a query is is a pretty simple um thing to do here but actually defining query language defining the acceptable queries is hard and then finally getting the most relevant data back um it turns out that to do this reliably it's uh involves a lot more steps than just um finding the most similar documents and returning those um usually there's pre- and postprocessing that's applied as well um so we talked about database stuff like scale and reliability as as a limitation um we talked about managing the embedding as a limitation so how do we how do we choose which embedding function we should use for different data types um what happens if we change the embedding function like open AI releases a new embedding model how do we update all the embeddings for our previous uh uh for that are stored in our index um we talked about document splitting like where the document is too long to fit in the context window um and we found we talked about the query language right so uh you know just finding the most similar documents is easy to describe but what if you want to filter on other metadata um or what if you don't have a query document in mind you just have you know a more abstract request um and then finally we talked about the search algorithm itself so like for example um if you if you have long documents and you split those into chunks then and you look for the the nearest neighbors then maybe all those chunks come from the same document is that good is that bad it depends right like maybe that's okay maybe that's the most relevant document and so you want all the information to be from that document but maybe um that's actually a false positive and so like it might be better to have a more diverse set of chunks returned so that the model can um have access to more different ways of of solving the problem so um the so dealing with like the structure of the data the the high the the relationship between the different um data that's embedded is is another thing that people spend a lot of time thinking about so uh the good thing is you know people are excited about Ai and so people are excited about databases for AI um investors are excited about databases for AI um and so there's a bunch of different tools like managed embedding databases that you can try or that you can look at um to to try to incorporate um into your use case and uh so here's like I would say probably like the five most common ones that I see getting used um and these all have different trade-offs with with one another so if you um are like looking for something really really simple to get started with and or if you want to make a bet on a tool that you that is like uh potentially going to get a lot more popular in the future because it's more maybe more AI native than these other tools uh then you know chroma is a pretty good is a is a decent bet um so chroma is uh incorporates like like I would say maybe one of the main distinguishing features it has is that it incorporates a lot of the embedding stuff like managing the embedding function into the database itself um it's not the only one that does that but it's it's one of the ones that does it pretty well um the disadvantage of using it is that I think it's like a little bit more immature compared to the rest of the tools in this category so this would really be for if you're prototyping or if you are making a longer term bet um on an interesting piece of technology uh milis is another commonly used one um that has like some prominent users at large companies um and it's it's pretty established and so if you're looking for something that is like works well for scale and you want to work with a company that has experience working with Enterprises I think milis is a decent Choice um pine cone is probably the most popular Vector database today I think the the reason why is because it's the fastest and it's the easiest to get started with um so if you if you uh you know if you if you just want to like get started with something that's managed very quickly pine cone is a good way to do that um it's also they've raised the most money I think this is I didn't update this column but I think they've raised more than 100 million now um and they have prominent users uh the disadvantage of pine cone is that it's like uh not very full featured compared to other tools in the category so it doesn't do anything to help you manage edings and it really just does like similarity search it doesn't help you with um the more richer like full featured search that a lot of these other databases do so two tools that do help a lot with this more advanced querying like if you want to combine traditional search and Vector search are vesa and we8 um so vesa was built at um Yahoo it's like a um piece of technology that's been in development for like more than a decade now um it's extremely powerful it's extremely fast and it's highly battle tested um Yahoo just spun this out in into a company a few weeks ago um so I expect to see this getting more popular over time uh I think it's like maybe the disadvantage is it's pretty heavy weight and hard to use but hope you know we'll see if the startup is able to solve that and then we V8 I think is a good middle ground between a powerful system like vesa and an easyto ouse system like pine cone um we V8 has a lot more of the full uh featured functionality that something like vesa has um but it was built by a startup and um it's very easy to use get started with um and I think also has some prominent users but I wasn't able to figure out which ones they are so um one thing to know about embedding databas is that is that you might not need one at all um One Direction that this field might go is that the embedding like the vector similarity search functionality is getting baked into a lot of traditional databases so if you use elastic search postgress redis most other kind of like commonly used databases today um you can now configure them to do a proximate nearest neighbor search just like inside of the database um and so you the nice thing is that you get all the benefits of the that those platforms already have um transactions acid wide availability scalability all those sorts of things with reasonable performance for approximate nearest neighbors uh the disadvantage of this is that like compared to a standalone database the performance is quite bad um and it these are not full featured Information roval Systems right so they don't work for like the most complicated queries are the highest scale um but for you they'll probably work so recommendation here is like I think when you're ready to move on from prototyping um I honestly would recommend just starting with a database you're already using um because it's easy you already have it and it's probably fine for your use case if you don't have one of those databases set up and you want to just kind of move on to something quickly then pine cone is great for Speed of setup um but eventually you're eventually if you push the limits of this information retrieval system enough like if you really have a need for accurate documents um quickly then um you're going to need to upgrade and so I think you should consider why like if you if you really want if the reason why you need to upgrade is because you need more flexible queries like just nearest neighbor is not enough you need more advanced filtering um then vesar or weate are good choices and if the reason why you're upgrading is ultimately because um you know the queries are fine but you need more scale and reliability then vesa and milis are good choices to consider so let me pause there any questions about embeddings uh uh nearest neighbor search nearest neighbor indices uh Vector databases information retrieval yeah uh go for it I'll I'll repeat the question do you think it's worth experimenting with different embeddings do you think that is a uh I guess I mean it could make a difference but do you think do you think that's something that should be prioritized in addition to the other parts of the information retrieval system yeah I I think so um choosing better embeddings can like is maybe the biggest lever that you have and can make the biggest difference but it's hard um because I think like one one failure mode that I've seen with a lot of folks is like for a lot of people building with AI right now you're used to using you know chat GPT or gp4 and so the mental model you have is that this is going to be like pretty good for most tasks out of the box um the reality is that like offthe shelf embeddings are not at the same level as off the-shelf LMS today um they're just not as good and so if you expect this to work out off like off the shelf for whatever task you're doing it's you're probably going to be disappointed it might work okay but it's not going to work super well there's still a lot of um in embeddings there's still a lot of room for improvement by training your own model um and uh so like you can get a big lift by training your own embedding model um and if you don't want to go all the way to training your own embedding model then there's simpler things that you can do there's simpler models that you can train that can help transform your embeddings in a way that's um in a way that can make them more useful so there's like if you look in the open AI cookbook there's one um idea that they suggest which is like basically just to fit um a linear transformation of the embedding model um to like adapt it for the task that you're trying to trying to do and so that's that can improve performance pretty significantly um and then in one of the case studies that we at um they share an example where they train a model to rerank the results of the nearest neighbor search um and so uh short the short answer is like yes choosing better embeddings has a huge impact um choosing better off the shelf embeddings probably won't have as big an impact as you want because there's still a lot of headro in customizing embeddings um you can go full customized like fully train your own embedding model for your use case it's slow and expensive like training models always is um or you can go halfway which is you can train like adapter models to adapt the embeddings to whatever task you're trying to solve I think that's kind of like where most people land in practice it's like use open open Ai embeddings and then transform them to be slightly more suited for your task and I guess like actually just one more thing on that um the intuition for why off the-shelf embedding models are not very good is because um since like open AI doesn't know the task that you're trying to solve or whoever developed the model doesn't know the task that you're trying to solve when they train the embeddings um the model is just trying to like it's looking at like tons of different data and it's trying to Cluster Things based on overall similarity of the documents and so depending on your task um similar and different might mean very different things so like um for example you know I think like one reason why a lot of these like just naive Vector similarity search systems don't work very well is because you're embedding um a query like you're embeding a question that a user asked and then you're trying to find similar documents by comparing that embedding to um the embeddings of documents and so documents and queries are very very different types of data right like documents are long they've got lots of information um they have structure like you know paragraphs beginning middle and end um queries are short um they're just a few words sometimes uh they might not contain very much information and so for a general embedding model that doesn't hasn't been trained to consider the difference between queries and documents um those those the the query and the doc the like query part embedding space and the document part of embedding space are very far apart from each other um and so if you think about like if you imagine on a whiteboard you've got like your queries here and You' got your documents here and so compared to any query all the documents are pretty far away um and relatively speaking they're all pretty close to each other um and so that means that like yes the more relevant documents still are probably closer um but you're using like a very small part you're you're using like very small differences between those uh those embedding distances to try to distinguish large difference in relevancy of the documents um and that means that you like you like any noise in the embedding function is like magnified um because all the documents are like about the same distance apart even though there there's some that are much more relevant hi um I was thinking about what you said about how Vector meth might not be important on your meting but I was wondering if that could have an impact when your query is asked in a negative kind of of perspective like um if you're asking uh Which documents are not about Youth and adult is not the inverse of Youth then you won't get the right return yeah I think um if you have like a negative in your uh query it's probably not going to be like none of the none of the techniques we've talked about would return um well I I guess the question is like what does relevant mean right like the um if you're trying to find documents that are not about something then saying like give me documents that are not about this is probably not going to work very well um unless you have some post-processing of the documents that are returned so um the vectors are not the vectors will be very similar like um the vector for not something like not uh not a turkey is going to be really similar to the vector for like a description of a turkey because um the model the embedding models don't understand the semantic context of the uh General embedding models don't understand the semantic context of the task that you're trying to solve um and so anything that talks about turkeys is probably end up being pretty close to one another um now you could get around that by training an embedding model that understands that the context of what you want there um yeah so uh toate short question the first one is about like when do you decide to go for fine tun or using a main techniques and the other question is about um how do you deal with um model you know when you have an image and you have a text and they have the kind of similar meaning can you Pro an intuition behind of this kind of system so I I couldn't catch that because of the echo like is it the first one was about um yeah the first one is uh when you to find in how you know fine tuning instead of so pretty much like when do you decide to go for fine tuning the model or just using a menting techniques such as chain or yeah and the other question is can you provide some inition behind of multimodal embeddings when you have an image and and you have a text and somehow they have have this same meaning how do you train these kind of models in order to you know when you project these two vectors in a the same space how they can have the similar similarities M um so for fine tuning we'll come back and talk about this a little bit more later but um in short I would say I would almost always start with information retrieval first um and then I would use fine tuning I would mostly think of people using fine tuning for different reasons other than like trying to inject information um um it's more for like trying to respect a certain structure trying to save cost things like that um but if you if you're really struggling to like uh follow the knowledge in your Corpus um because there's way too much and the informational part is too hard you could also try fine tuning just much harder um and then for intuition about how like multimodal embeddings work um you can think of it as like uh imagine a data set that contains pairs so like an image alongside a description and if you train a model on those things together um then the model knows that the image and the pair or the image and its its corresponding description should be close together um and that's a high level like how some of these models work yeah so looks like uh the embeddings are uh very important for the retrieval aspects right uh especially when I'm using open AI to uh generate some text for me uh so is is this a competitive Advantage for open aai to use their open AI iddings uh is it uh like I can use third party stuff you said the quality might reduce so this is how they're trying to make sure that we all adhere to their openi offerings they're pretty interchangeable um so I think the only I think with the um assistance API uh that's kind of why they're like if that ends up becoming good that's when they would it would be advantageous to use both because they can run both on the same infrastructure without a network call between them um but today unless you're using that assistance API which probably no one is because it doesn't really work um you are going to be making two Network calls either way like you're going to be first calling the embeding and then you're going to be getting the result back and then you're going to be calling the uh the llm and so there's no reason not to mix and match and open AI doesn't does not have the best embeddings on the market I think um some of the open source ones are better and um I've heard that the recent cohere ones are better as well um but I haven't tried them myself yet so yeah there's no reason not to mix a match and how do you uh Define a good embedding versus a bad embedding so a good embedding is it's very unsatisfying but um this is true in machine learning in general a good embedding is one that helps you solve your task so if your goal is to answer questions for users um then the best embedding model is the one that produces the best best answers for users whatever that means for your task um that means that like at the end of the day like understanding embedding performance is the same as understanding task performance um there's other heris STS about what makes a good embedding like things that you think are similar should be close together things that you think are different should be far apart um but at the end of the day that's also not very quantitative and so really like looking at Downstream performance is the best way to measure them about the the question is about practice so um open AI itself was suggesting to use embeddings as an alternate to fine-tuning so I wanted to get your response on that is it a good practice or not and then the second question to that is that is the purpose of embedding to get responses only within the embedded you know content or is it also broader to the base is llm to which we are embedding yeah so um I agree like I I don't really recommend I think people reach for fine tuning too soon like I don't think you need fine tuning most of the time um and fine tuning is much much harder than all the stuff that we're describing um so I don't I've trained a lot of machine learning models um I don't really recommend people do it very often um it's a great tool in the toolkit but I I think I would try other things first um embeddings being one of them and then um embeddings are usually used to like when you have a corpus of documents um and you don't have room to put all of them in the LM at the same time to help choose the most relevant ones to put into the LM and so it's it's uh typically used when you have like an external body of knowledge that you're trying to search over so you mentioned that um the fundamental problem is that the space of the query the embedden space is different from the embedden space where the actual documents live it doesn't it make embedding basically not suitable for this retrieval augmenting so it um the so it does it does not make embeddings not suitable for retrieval augmentation because um embeddings inquery like documents in queries do not have to be far apart in embedding space um and so for companies that put a lot of research and effort into this they design embedding models where um they're trained in such a way that the queries and documents are close together um the what it does mean is that most off-the-shelf edings that were trained without information retrieval in mind um are limited in how useful they will be for information retrieval without additional post-processing um but pragmatically they still have some value it's just that they're not like you don't get perfect results out of the box all right let's talk about some uh some case studies um so the first one I want to talk about is co-pilot co-pilot arguably the most successful application of large language models maybe other than chat gbt um certainly the one I've used the most or maybe the second most um so how does so for co-pilot and again like for those who are not familiar with it co-pilot is like a agent that says alongside your coding in um vs code IDE and it make suggestions for you for code that you're you might be trying to write um so how do they pull the relevant documents into the co-pilot so they use a combination of retrieval postprocessing and ranking um retrieval is they use mostly heuristics like they look at the most recently accessed documents in your vs Code system um and they limit that to the same programming language that you're currently programming in so if I'm programming in Python it pulls the 20 python files I've looked at the most recently um then it post-processes those documents so it takes um the like my current cursor position in the IDE and it looks at things at uh at things like what CA comes before the cursor um and what comes after the cursor and it finds uh Snippets of the 20 candid documents using similarity search um then it takes those Snippets and it sorts them um using some heuristics and then it puts those heurist those sorted results into a prompt to produce the outputs so it's a very heuristic system the nearest neighbor similarity search is not like a big Vector index or anything like that it's just um you know uh a like very kind of local process that's done that's wrapped in a ton of tics and I think that's part of what makes it so good is the heuristics that they used um another like common pattern this is probably the most common pattern that you'll see for these applications is the retrieval augment augmentation question answering pattern the way that this works is you um you how take a user's query like when was our company founded you compute an embedding you find similar documents um in your index and then you dump those documents directly into a prompt um so you say like hey qt4 answer the question based on the following documents um where those documents are the ones that are the most similar to the query uh so again we talked about this limitation already but um the main limitation of this is that your your nearest neighbor similarity search has to be good like the if you just dump the top handful of most relevant documents into the prompt then you have to be pretty confident that um the context that you need is going to be in one of the top few embedded chunks so you're placing a lot of emphasis on how good your embeddings are if you do things this way one way that you can um get around this limitation is you can uh do this in a loop so like if you um rather than you know only providing a handful of examples to fit in your context window instead you could try to answer the qu you could take the top 10 instead of the top three um and try to answer the question based on each of those documents and have a model decide which one of the documents is most relevant to answer the question so you can have the model process the first document decide if it can use that document to answer the question if it can't use that document to answer the question then it can look at the second document um and then the third document and so on until it finds the document that is the most relevant this can get around the limitation because you're splitting it into multiple model calls um but obviously have the trade-off of like cost and latency because you're calling these expensive llms multiple times um so this that we just described this process of like taking a um uh like rather than trying to dump all the information in a single prompt instead um letting the model process the information multiple times um to get around context window limitations this is a simple example of a chain so chains are um you know like sometimes the the best context that you can have for your LM doesn't exist in your Corpus right so like sometimes you need to pre or post-process the data um before or after it's retrieved from your document Corpus um and so and oftentimes the way that you'll do that pre or postprocessing is via another llm so the idea of change is that like um to to produce the context needed to answer the question or solve the problem your user wants to solve by not only pulling in relevant information but also processing that information so that it's more useful for the downstream LM uh there's many many different types of chains that people build and this is like kind of a whole rabbit hole that you can go down um there's some like common patterns like here's some like I guess example patterns to understand this better um so the question answering pattern that we described before you can think of this as a simple chain where the first step in the chain is you embed the question then the second step is that you find similar docs to the embedding um and the third is that you pass those docks into a QA prompt and you run that QA prompt one way to get around the limitation of uh queries and documents being far part is to use um a a type of chain called like hypothetical document embeddings which is kind of like a neat trick that works um okay not like not super well but can be sometimes helpful which is you take your question and then rather than passing that question directly into the uh into the embedding and finding the nearest neighbors instead you say like okay well if questions are in embedding space are really far apart from documents in embeding space what if I just use an llm to generate an imaginary document um that it thinks could be helpful for answering this question and then rather than passing the question into the embedding model instead I pass this imaginary document into the embedding model um and I perform similarity search with the imaginary document instead of with the raw query um and then run the rest of the the question answering chain the same way so that's a more complicated chain where rather than just uh you know passing the question in directly instead we pre-process the question before it's passed into the embedding model you have experi um so so typically the way I'll see people do that like so the question is around like can you just um if you have examples of questions and relevant documents can you use those questions and relevant documents to build better embeddings um usually the way people see people do that is to train an embedding model um so like the like state-of-the-art embedding models like if you have a really mature search use case um the models will be trained jointly on questions and relevant documents uh so that when you pass the question in um then it's it's the model is explicitly trained to find documents that are similar to questions rather than just um trained to find how all kinds of texts are related to each other um another type of chain would be like a summarization chain so um you could take like let's say that you want to summarize a big big Corpus of text you want to summarize um like summarize everything on Wikipedia for me the way that you could do that is you could take every document in your Corpus you could Summarize each one of those so summarize every document on Wikipedia and then you could pass all of the document summaries into another summarization prompt and you could summarize the summaries and get a global summary back so again this is like a a chain that is getting around a context window limitation of a model um because you're just uh you're you're breaking the task down different to different parts Each of which fit in the context window um so the most popular tool for building chains of models is called Lang chain uh it's one of the fastest growing open source projects of all time um it's both Python and javascripts um I would say the strength of Lang chain is that they have a lot of different chain examples that you can pull from um to get started very quickly um and also just generally like having a framework is sometimes helpful I have found in practice most people I talk to who are building production systems end up building their own chaining framework um because Lang chain has a lot of limitations and uh it's there's not a lot of you know you don't need to write a lot of code to build your own Chains It's like literally just passing llm calls to each other um another common tool that you'll see by the way is uh llama index which is the difference between llama index and Lang chain is that um llama index is like Lang chain is just a general purpose chaining framework where it's like you can build any of these types of chains in Lang chain and they have lots of examples of crazy different types of chains they've got um document loaders and uh just basically focused on making prototyping really easy um and then llama index is focused more on information retrieval and so it's more focused on these types of chains where the goal is to um you know take documents from different parts of the Corpus um pre and postprocess them in different ways to do better information retrieval than just nearest neighbors similarity search alone any questions on [Music] chains when you change and you're doing summaries of summaries of summaries how well does that work it can work surprisingly well yeah I mean all these things are have limitations right um but often times like I I think what chaining is most useful for in my view is like when there's some inherent like logical structure to the problem um you can ask the you can tell the LM in the prompt to follow that structure like you could say you know so like let's say that you had had a document that would fit in the corpus um and you want to summarize the document you could say hey LM summarize the first paragraph then summarize the second paragraph then summarize the third paragraph write down the output and then summarize all three of those things um you could do that all in one prompt but um if that if the logic is that crisp um then often times breaking it into smaller prompts uh allows the model to follow it more reliably um so there's there's really two reasons for it like one is if the model's having trouble with the task and you want impose some structure to the task to make it easier then like sometimes splitting it into multiple calls is more reliable than putting it all in one prompt and then the second reason is if you're trying to get around like big context window limitations which um if you're doing that then like yeah I mean it can still work well like as long as the um the base as long as each individual task works well then the overall thing can work well too okay I um you talked about three types of augmentation the retrieval uh the chains and the tools right and uh it's maybe not a fair question but you sort of hinted at it with your demo earlier how durable will this skill set be uh getting good at this will it will it be good to be good at this two years from now yeah I don't I don't really know I mean um I guess you could ask the same question about you could have asked the same question about deep learning um like when we were teaching this class first in uh you know 20 2018 2019 um to me it was like already felt obvious that models were going to not everyone was going to need to be able to train models um and so you could argue that like yeah learning how to train models really well in 2019 was not a good skill to learn um but on the other hand it's kind of like models have gotten so good so fast that the overall size of the pie has grown a ton as well and so if you learn if you got really good at training models in 2018 like you're probably doing pretty well for your career um and so maybe there's a similar thing here where like if your goal is just to make is just to use this technology then um yeah you can wait a couple years and it's going to be a lot easier than it is today um but the people who are really good at it the people who are good at the lowle details are still going to be really valuable um if the size of the pie grows like the one of the most one of the most impactful people at open aai um early on was this this guy Scott Gray who was basically just one of the world's best GPU kernel programmers so do you need to learn how to program GPU kernels to build machine learning applications hell no no no obviously not um but does open AI need to be good at building GPU kernels it's really really helpful for them to be good at it and so there's not a lot of people in the world who are good at it and so the people who are good at it are are really valuable um so for my own understanding um just you were talking about the context window problem is the reason that we're doing the the um the chaining so that we can compress the information from the documents um into a smaller into a a smaller um context that we can feed to the the llm that is lower down in the chain is that a correct understanding yeah that that's one common way of using chains I I think um I I think of it as kind of like there's two reasons why you'd use it the one is um if you have more data than you can process um in a single LM call then you can split into multiple calls process things independently and then combine the results that's like one reason that you could use chains and then another probably more common way that you would use chains is if you have like some inherent like structure or logic to the task like um I want to process each of these paragraphs independently and then combine the results um then splitting it into multiple LM calls can be more reliable than uh packing that information into the context into the instructions because models can get confused by long instructions but does it still works if for instance you wanted to do um Le let's say you had a multi-step thing and the steps were dependent on each other like you do this step and then you do this step and then you do this step would chaining um help with that because then you don't have to make like one enormous prompt like do this series of tasks is yeah exactly um yeah I think like you could do that all in one prompt and I think it's good Baseline to do it all in one prompt and it might be fine um but I I found that like models the the longer the context gets like both the inputs and the outputs the more likely it is for them to sort of lose their train of thought or become confused um and so if you're working on a task where like there's many steps or um the there's a really specific set of rules that the model has to follow to do all the steps um and you find it starting to get confused of like oh it you know you told it do Step A B C and then BCA and this complicated set of things and you find it getting lost then why not just embed that logic directly like write that logic directly in code and then have the model just understand what it needs to understand to do the simpler task uh I have a question about productionizing so you you mentioned that you can start with Lang chain and lots of people like we included we started with L chain prototype but what is a step to move to productionizing can we just roll L chain um you know up in production or at which point do we start to customize and roll our own yeah I think it's very it's like highly individualized to different organizations people do uh roll Lang chain into production um I've seen people do that too I I think it's what I've observed is that like at a certain point so the the strength of L chain is breadth like there's lots of document loaders built in there's lots of chains built in um the strength is not so much depth like they they don't necessarily have the best prompt for your use case um or the best chain for your use case and so what I find is that like the path that most people go down is like um exploration first like oh let's try lots of different chains for our use case or U maybe even try lots of different use cases because we don't know what's going to be most helpful for a company yet and then when they narrow down on one then a lot more of the what they're spending time doing is iterating um on the prompt or the chain um trying to improve performance like both task performance as well as like system performance and then oftentimes what folks do at that point is they'll end up just kind of rewriting their chain just in code um because they find that like L chain subtractions start to get in the way or just because they don't like 90% of it is not that useful for them so they want to simplify but not everyone does that yeah hi I had a question uh I think it might be something you're going to talk more about later but um you did mention and it's just something that's been happening that context Windows have been getting really large and I think it's kind of misleading because you know if you put more context into a prompt it's not it's not necessarily going to perform that well and so I wonder if you're aware of any research or just general her istics in terms of uh you know like when thinking about passing context into a prompt uh what's a good amount to not or what's sort of overdoing it to the point where you're not going to get good results Beyond just like just testing yourself yeah yeah I think um well the punchline is like testing yourself is like the way like machine learning is inherently empirical like um people don't always know why things work or don't always have like the best mental models for what's going to work and what doesn't work and at the end of the day the way that really great machine learning folks work on problems is that they are like relent like ruthlessly focused on measurement um like they said benchmarks for themselves they spend a lot of time looking at data they try a lot of different things they're not and um they're they're more like the the best machine researchers I worked with it's more like they had a high throughput of trying things than they necessarily had more accurate ideas um and I think the the challenge with answering that question in general is that it depends a lot on what's in the data um and what tasks you're trying to do with it so I've found that like models you know if you're doing something like summarization models do a pretty good job at summarizing long documents like you can pass you can fill up the context window and have the model summarize it and most of the time it'll do reasonably well um but then for things that involve like nuanced understanding in comparison of the details of the document that's when they tend to struggle more so if you have a long set of instructions the model will sometimes forget about one of the earlier instructions or if two instructions are like you know not even contradictory but like could be read in opposite ways then the model will not always have a good nuanced way of distinguishing between between them um or if you're asking model to answer a question based on a document and there's multiple facts in the document that are relevant to understanding the question then sometimes the model will miss one of the facts and just focus on the other ones those types of things um yeah but measurement is the way to answer it ultimately all right let's talk a little bit about tools so um I think like the point of the lecture so far is basically been how to build your own search engine um so one question you might have is like why should we build our own search engine why can't we just use Google um um and you can so this there's a chain that I would call the I'm feeling lucky chain um where you take a question then you search Google for the answer to the question you take the top result you pass the top result to a summarization prompt and then you return the summary and so the key Insight here is like um another way to give llms access to the outside world rather than maintaining a corpus of documents um or and indexing that Corpus of documents and building a search engine yourself is just to give access give the LM access to tools that already exist to interact with the outside world um and so you can actually just let the model use other apis um and and do things that way so there's some papers that like kind of initially um push this idea forward I think the one of the earliest ones was based on Bert like a very old small language model um which was called giving Bert a calculator where um what they would do is they would uh like they they gave birth the ability to call a calculator instead of trying to do math itself and they found that it was able to do math much better might as much as you might expect than LM doing math on its own um one of the kind of like more ambitious use like uh applications of this is web gbt which is um which is a browser based question answering uh uh approach from open AI which is much of which has been built into their like browsing plugin now and then I think like one of the papers that first got people really excited about this idea in a general way was a paper called tool assisted language models um where they kind of provided a general framework for like giving a model tools letting it call those tools and pass the result back into the model um the uh let's see yeah um a couple other Pap papers here but I'll skip through this um I I think like maybe the more relevant thing to understand is like what is a tool a tool is basically any API or any um other piece of code that you can call um and the way that you can use it there's at a high level two kind of common patterns for how to use tools um tool you can use tools like just like any other part of the chain um so you can use them deterministically you can say like okay the first thing that you do after you receive this query is you pass this query into the Google API or into a search API and return the result back and put it in a specific place in the prompt um the second way that is common to use tools is you can let the model choose when to use the tools so you can let the model um say as its output um I would like to call a tool pleas and then you pass the input the the result of that tool call back into the model to use for whatever it needs to do next um and so the the I'd say probably the most successful example of that second pattern is uh open aai plugins or chat GPD plugins or like open AI function calls um the way that these work is you just provide a spec of the API that the model is going to call um that spec needs to contain a description the description is a prompt um it's a natural language description of what this tool is going to be used for um the model uses this this uh API spec in two ways the first is it takes that description um it passes that description as a system message to the model um as part of the system message and then the model can uh then knows about the existence of this function and can decide hey rather than returning a result to the user I'm instead going to ask for some more information from this tool um and the way it decides to do that is by looking at the descriptions of the tools and reasoning about like hey would any of these be helpful for me uh to to answer the question better um and then the second way that it uses the API spec is that it uh then produces an output for the model um it produces an output for the tool that is formatted in the way that the tool understands so if the the API expect uh spec expects you know three inputs the model will produce those three inputs um then what happens behind the scenes is that the the client like chat GPT or the application that you're building is responsible for taking the inputs that the model produced um calling the API Gathering the results back from the API and then sending them back to the model um so then the model sees the result of that function call um of that API call and can use that resulting information to try to answer the original question um any questions on function calling yeah uh can you elaborate a little bit more on how the model chooses to invoke a given API like example let's say in a application we have a bunch of microservices with a bunch of API and we try to metadata in the plugin but uh how do we train the model saying that hey if if there's a question regarding billing or something use use the API from the service if it's from some other service like call some other API like how can you like train it expand so it's um at least with open AI you can't really train it um what you can do is you can prompt it um and so there's two places in the prompt that you can that you can describe how to use these functions so the first is in the description of the function itself and that's kind of the way that is encouraged um and so the way that works is like let's say that you have a function for getting weather um you might write a description that says like uh you know this is a function that will return the weather at a given location um and then let's say that you have a function that's you know Returns the current time um you might describe it as like this Returns the current time as of right now um and then those you can think of those as like um as part of the overall prompt that is being used the the part of that prompt is a description of all the tools that the model has access to um those descriptions come from this API spec uh and they're the descriptions I just chose and so then when the model is like reading through those instructions it's know it knows it's trained to um think about whether it should use uh whether it should whether it might be able to answer the question better by calling one of the tools given the descriptions that you provided of the tools um you can also help you can also help with this process by like putting some information in the prompt itself um like in the prompt that I showed earlier for uh for um question answering I said like hey always use this this uh this um information retrieval prompt um but you could also provide more nuanced information right like you could say like if the user asks an ambiguous question then always call at least one of the functions um or things like that um just on that how do you to not do something like for example asking the recent sort of billing information for the next quarter which is sensitive right uh in your previous session you talked about if you say something not to do for LM it might start thinking about that yeah um so in this case how do you sort of ensure that it does not do something coming from a privacy kind of standpoint here where we are ensuring that it's not calling certain set of apis if certain rule based value have not being met yeah it's it's really hard to um ensure that the model will not do something um like generally if via prompting alone so so usually what people do is they'll do that in more of a rule based way right so if you if you give the model access to an API but you don't want it to call the API for you know information from the past month or something you would just apply that you'd rewrite that as deterministic logic um or maybe if the logic is really hard to write deterministically you would have a totally separate llm call that doesn't have access to the full context and its only job is to determine whether this is an acceptable call or not um so you could isolate the llm calls in that way and that might make it more reliable but usually it's heuristics yeah now we we talked about how this how the models will predict that they will use some kind of tools but do you know how this will work from the engineering perspective Ive um I know we can't see what open AI does but I suspect something like a special kind of tokens being spit out of the model then the generation will be stopped the tool will be executed but do you have any more information either from openi or when building your own llms locally yeah so for openi it's pretty easy because of the um like the chat Paradigm um so basically like the model can just can um return has different return types like it can return a answer or it can return a request for a function call um for more like open source models um the implementation is trickier uh like if you're just if you just literally have a Transformer like a next token predictor um and so usually what you'll use is like stop tokens um so you'll have like certain tokens that the model is supposed to Output when it wants to call a function um and then you'll deterministically you'll look at the stream of outputs you'll look for that uh that that specific sequence of tokens when you see that sequence of tokens you'll stop generating outputs you'll instead call the function um you'll append that to the context and you'll pass the whole context back into the model to keep generating uh so for like retrieval augmentation uh how do you take the call that for a particular query the answer is in the documents or whether it is about like the previous responses itself like for example like if I'm writing a chatbot for Relevant documents and I ask some WR the response that you gave above or like U you know related to the context of the uh session versus like from the documents like how do you decide in those cases or how do you like yeah so the these techniques are not are um complimentary they're not mutually exclusive so um a common pattern for retrieval augmentation for question answering is just deterministic it's like no matter what the user types in I'm going to search first and then take the documents and I'm going to determine this basically just dump those documents into the context and then respond to the user um but that's not the only way that you can do this um the other way that you can do this is you could and I think makes a lot more sense for more of a conversational agent is um you could give the model access to the retrieval uh system as a function and so then the model makes the determination for whether it needs to run a search in order to answer answer the question or not um so that's sort of the Paradigm that open a ey is pushing um and uh I think it the limitation is that like like the model has to get multiple things right then it has to call the right function um and interpret the result correct correct call the right function with the right parameters and interpret the result correctly um whereas if it's deterministic you can have more things that you know you know are supposed to work um but the advantage of it is exactly what we described right like if you have a complicated Behavior it can be hard to write the rules and so you can just let the model decide all right I want to make this a little bit more concrete by covering a couple case studies um so this first one is a case study that was published recently by a company called shortwave um I've never used this product before but it's basically a um like AI powered email tool um and uh seems pretty cool but the the reason why I thought it was um relevant is mostly just because they uh they have like a really nice in-depth case study about how they built this so the way that um shortwave works at a high level is the you take the user's query um then the model does some tool selection um so it chooses which functions it's going to call based on the user's query then it retrieves the data from those tools um it answers the question it performs the question answering step then there's some post- processing that's applied and then it Returns the the answer back to the user so let's dig into some of these steps um the tool selection steps uh the the um shortwave team has like provided a bunch of tools to the model um some of these tools are deterministic and ones are themselves llm calls um so there's a calendar tool which is deterministic it just looks up a schedule information from the user's calendar but then there's also things like a compos tool which um uh which you know like takes uh like runs through a prompt to actually compose an email um or summarized tool which like augments the the what the the current context with like Specific Instructions to uh provide a good summary and so the the team has designed these um tools to help help with like common tasks that their users have common things that their users want to do in their application um the uh like one one of the most uh one of the tools that they as you could imagine put the most effort into designing is the Search tool um and so this is a description of how the Search tool Works which I think is interesting because it gives a little bit of a window into how much more complicated these like high quality search tools become than just like nearest neighbor search so in their case what they do is they take a raw query from the user then they perform a step called query reformulation which we'll come back to in a second um then they take the reformulated query they run a deterministic um feature or a a l based feature extraction um in order to pull out some relevant features that they're going to use to perform like a traditional search um they also compute an embedding of the reformulated query and do a vector similarity search then they use all their documents that they retrieve from both of those types of search um to like um pass into a reranking model which I'll come back to um and there's multiple reranking steps that are applied before the uh before the like ordered fragments are returned um to uh like to to like whatever the search algorithm is using or like pass back into the prompt um that's produced by the the Search tool so to double click on a couple of these steps um the first one that's interesting to look at is query reform ation um so the reason why they have a query reformulation step is because the user's query doesn't always contain all the information that's needed to understand what their intent is um so in this example the most recent question the user asked is like what about Johnny right so without understanding the previous context of the conversation it's impossible to know like what are the most relevant um uh emails to pull in relation to Johnny but if you understand the previous context um then you understand that it's about Johnny's flight to Phoenix um and that allows you to reformulate the query to when does Johnny land in Phoenix in order to pull documents that are relevant to answering the the intended question that the user meant to ask so that's the query reformulation step the first step before they actually do the search um for the search itself um there there's a combination of like a vector nearest neighbor search as well as a heuristic based search so they take the reformulated query and they extract a bunch of features from it um so they extract a recency bias like how much they want to prefer recent answers they extract some keywords some named entities um some date ranges if they're relevant to the question um as well as the embeddings and they pass all of those things instead of just the embeddings are just the features into the document retriever um the uh then so that that's kind of like a highle of their search process right so it's um it's a it's a a customized search engine that uses both heuristics extracted from the documents that they know are you know helpful for answering the types of questions they want to answer as well as kind of the raw embeddings that they can use for just like pure semantic similarity um but then once they take the results back um once they gather the top results from the search engine rather than just spitting those results out in the order that they came back from the search engine they apply apply a post-processing step to rerank those results um before they pass them back into the model um so they reorder the top results passed by back by the model and that's a really common pattern for these information retrieval systems so the way that they do it is they um have two different phases of reranking so they first have a bunch of heris that they use for the reranking they have the raw similarity score but also things like the target date ranges um the names and the people from the query um any labels or tags for the emails that are present or not present in both um and then they also drank things like emails that are tagged as promotions or um things like that that they just would expect to be less relevant so all these like email specific heuristics that they use to reorder the search results um and then they take the top results from that reordering and they perform a further reordering step based on a custom model um so this is a purpose-built model that is trained to take the uh query and the relevant documents and spit out um like a a relevance or an ordering those documents that they can use to um further refine what the most important documents are to pass into the model and so this is a custom trained model um it's a lot simpler to train than the full embedding model because it's just taking the um the relevant documents and uh an outputting a score um but and it's uh slow to run so it's only running on the top handful of examples to just like further refine the ordering so that's case study from short w W um this is also an interesting case study from the open AI assistant API I've I've not been nice to the open AI assistant API today but I think it'll get I'm confident that it will get good um and so there's an interesting slide from their uh uh Dev day when they talked about like what they techniques they use to improve the retrieval augmented Generation Um now we saw an example earlier of the retrieval augmented generation being really really bad so but I we'll take them out their word that um in some use cases it's really good and so you can see that they started with just like retrieval via pure cosine similarity as the Baseline that they considered um and uh they achieved 45% accuracy which is seems pretty bad um then they tried another number of other things like the hypothetical document embeddings that we discussed earlier um I don't know what uh what ft embeddings are but whatever that is they tried it and it didn't work um and then the main thing that got them from 45 to 65% is they ran a bunch of experiments with chunking and uh different embedding models so um choosing different ways of chunking up documents choosing different embedding models can make a big difference um then the further the next accuracy bo uh bump came from having a reranking classification steps so um instead of just taking the results that you get from the similarity search and accepting them um instead doing what we saw in the last case study where you reorder them based on the results of another model and then finally they got sort of the final accuracy boost by um doing prompt engineering on the on the different prompts that are part of this process um adding in tool use and um performing query expansion so uh you know like reformulating the query to produce better results so um I think the the interesting thing here is not so much like the specific techniques that they used but it's really two things um one is just to give you an example of the types of things that you will probably need to try to go from an okay retrieval system to something that's really good um and then the second is like if you notice how they present this information it's with a um an eye to the accuracy of the overall retrieval algorithm at each step um one thing that opening ey does really well is they're obsessive about measurement and um evaluation of everything that they develop at every step along the way and so the way um what we'll come back to a little bit later this afternoon the way that I recommend developing systems like this is by building an evaluation or a measurement harness early on in the project um so that you can try a lot of things and quantify the impact that those things have um rather than just throwing you know throwing food at the wall and seeing what sticks so um I think just to wrap up this section um LMS are a lot more powerful when you collect them connect them to external sources of data um when you do that you can do a lot with just rules and hers rather than needing to build a search engine um so the the canonical example of that is co-pilot right where like most of co-pilot is just heris sixs it's it's not like some complicated fancy algorithms as your knowledge base scales up um and you have more and more documents in your Corpus you should you should um maybe move away from heuristics or consider moving away from heuristics um but also consider moving away from just thinking about this as Vector similarity search and instead you should start thinking about this as an information retrieval system that consists of vector similarity search but also other techniques like reranking um heuristic search and some of the other things that we looked at in the different case studies um as you start to like have more and more complex reasoning and um start to hit token limits you can consider chains as a way to break up the logic in a deterministic way um and then finally you know if you want to have access to a much broader body of knowledge um and use existing a like information access patterns rather than building your own um you can use tools um or function calls as a way to give the model access to a wider set of information so um couple other resources I'll point to um in the slides to go to learn more and um yeah let's uh let's take a uh call it 20-minute break