Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 5
Recording: Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 5
[Music] so um I'm going to show you an example of building a bot um to do Q&A on the FSL Discord um based on the information uh that's contained in previous FSL classes and so this bot is running live in the Discord now um so if you go to Discord and then you go to to the ask fsdl bot Channel um you can play around with this as we talk and like try asking it questions and see what works and what doesn't work um using this SL ask command um I also posted in the um llbc uh sbtb 2023 General channel the channel for this class um a link to the GitHub repo where um you can follow along the code that we're going to talk through and this is also like if you don't have this link you can just go to GitHub slthe full stack and it's the ask fsdl repo there so um what are we going to talk about here um so the basic architecture of this system is it's kind of like a simple implementation of the reference architecture that we've been talking about for most of the day um so we're going to start with different sources of data and these sources of data come from like different um data that we have from previous classes so it includes um uh the like text on the website on the FSL website um it includes the uh the Youtube videos like the recordings of the lectures and the transcriptions of those recordings um so like all the different sources of information that we have about um about like what's happened in previous classes and so the first thing that we're going to need to do is we're going to need to take all of that information um all that inform like raw information and we're going going to need to store it somewhere um in a format where we'll be able to do stuff with it later um and so we'll run an ETL process on that raw data to put it in data storage then once we have the raw data we're going to need to embed each of those data points um store in a vector index um we will have that Vector index um interacts with with a serverless backend um that serverless backend what it does is it takes the questions that you'll be asking in Discord um which are processed by a Discord bot server um and it combines those with data from the vector index and a prompt um for the base llm to produce a response um that response goes back to the Discord bot server which then forwards it along like formats it for Discord and forwards it along to the Discord Channel um so that you can see the the answer and the tools that we'll be using for each of these steps um again these are just examples of the many different types of tools that you could use here um but we'll be using a tool called modal for the ETL um modal is like a serverless backend tool um that makes it really easy to turn python functions into serverless functions that you can run on um other infrastructure so it's helpful for many things um among which is like paralyzing um kind of computation expensive tasks like ETL then we'll be storing the data in um mongodb um and they're their uh their um hosted option mongodb Atlas we will be embedding that that data storing it in Pine Cone um our serverless backend will be again hosted in modal um our Discord bot server will be hosted on AWS and then the llm that we're going to use is going to be powered by open AI so again one's uh individual choices of many um and again you can try this out on your own ask fsdl bot channel in the Discord so play around with this and then if you want to follow along in the code it's uh github.com thee full stack and so um this is also didn't want to do like a live coding thing today because I It's just tough with a group this size but um this is you should be able to run this on your own um if you want to like try this out and make changes to it um and explore the code base a little bit more so with that um let's take a look at what's going on here so I will start with the data um so I mentioned the data was like a representation of each of the the different kind of sources of information that we have um that kind of make up the the body of knowledge that is the the um uh that is fsdl and so the way that we've stored this information is we've stored it as like um kind of man FS files that um keep track of all the individual um types of documents and information um but are not the raw data themselves um instead they're just pointers to that data so showed this example before but um we have the URL for where the raw lecture notes are um are are stored alongside the um the individual lecture examples um so these are the the different lectures that are stored um as like part of the uh fsdl Corpus but then we also have other sources of data here in the kind of full bot that we didn't talk about this morning so we have references to llm papers that include the title of the paper the archive link um as well as different tags that are relevant for like what type of queries they're going to be used for um and then we have the IDS of the videos themselves um and so this is like you can think of this as like a manifest file um and the reason why we do it this way is that like typically um not really best practice to store large amounts of raw data in GitHub it's uh um if you want a version control things it's helpful to use GitHub to just store like pointers to the data um but then store the data itself elsewhere so that's the data piece um the next part of the the um if we go back to our diagram here the next part of the pipeline after the data itself was the ETL process so how do we take that raw data um download it transform it and store it in our data storage and so let's look at the um the ETL code itself and maybe we'll focus on the um the specific example for markdown um so this is a good point to kind of like tell you a little bit about modal which is the tool that we're using for ETL um so again modal is like a a really easy way to manage serverless compute in Python and the way it works is it kind of treats it's like the um infrastructure that you're running things on as like basically just like python objects so um in this case we have is we have a modal image um oops so we look at the Shared code here um we're defining a a modal image and this modal image is uh an image that uses um a container um that's based on uh dbn slim um upu uh uh distribution with this particular P version of python with the following packages installed Lang chain um and mongodb and so what this does is it defines for model um Hey whenever I use this image to run a function this is the um the docker container that's going to be running behind the scenes to actually um provide the the the resources to run this um and so the uh the um let's see um so if we go back to our ETL process for markdown we start with the um this base image that we defined that had those two dependencies those two python dependencies we installed some additional python requirements and then we Define a stub um which is a stub is like a combination of the um raw image alongside of like anything else that we want to provide to that computer when we run it um so in this case we give it a name um we specify some Secrets like some API keys that we want to pass in um as well as some mounts so like some local files that we want to attach to it and then stubs in modal are things that you can just run um and so whenever you run um a stub um it basically um takes care of translating um how to actually run that code on the physical infrastructure in a way that where you just don't have to think about it it's just like running a python function U but modal knows um what are the requirements for that computer and the the image that's running on that computer in order to run this function correctly and so in this case we have a um uh like a simple sort of like coordination script that we're going to Define um as something that we run locally on this machine and what this does is it um loads um like all of the uh the Json um that we have that defines the Corpus of markdown files that we want to read it has the logic for how you combine these URLs like if you remember we had a base URL and then a stub so how do we combine these things into the ultimate URL that we want to download um and then we uh we take each of these documents um and we map this uh two documents function on them so this two documents function um what this is doing is it's downloading the text from the URL it's splitting um the splitting up the headings and then it's um basically taking these these uh these documents um and adding some additional metadata to them and so the key thing to understand here is like basically you know without understanding necessarily all the details about how this ETL Works um what this is doing is it's paralyzing the process of taking each of these files that we have stored somewhere else and preparing them to store in our document store um so it's it's uh splitting them up it's handling the metadata um it's putting it all back together in a way that we can put it in the document store and each of these um kind of ETL scripts works pretty similarly so we have one for markdown we have a different one for PDFs um which you know processes the PDFs uh differently and we have a different one still for videos um which takes these uh videos and gets a transcript for them and extracts the um extracts like kind of the the uh the text from those transcripts to use for um ultimately for embedding so any questions on like kind of the data or ETL piece of this um guess sort of gloss through some of the details here but I feel like this is maybe um one of the less like LM specific pieces to understand here cool um so next thing let's talk about the vector index um so let's see let me sorry uh yeah a question so for this test uh for the mongod DB we have to set up somewhere in the cloud for the Atlas uh can you say that again no for the ETL process uh it needs to store in a data store mhm is it mongodb Atlas yeah that we have to configure still is it or yeah so so you still need to configure mongodb separately um and so we I think we have that documented um somewhere here yeah maybe under setup um so there's um there's a set of instructions here that walk you through if you want to run this on your own um connecting how like how you actually set up mongod DB and connect it to the the this fsdl repo got it okay thanks um and yeah so the it's it's worth noting that like under this setup folder there's a set of instructions here if you want to run this on your own how you set this up from scratch um and so this involves kind of updating environment uh file preparing your python environment um setting up your managed services like modal uh open Ai and mongodb and then um and like once then once you have those things set up then you can actually run the ETL which downloads all the data processes it um transforms it loads it into the document store creates the vector index um runs the back end so that you can interact with it so this is you know if you want to if you want to follow along or do this on your own um this is the set of this is the right starting point the set of instructions to follow um okay so let's look at what's going on here when we um make these documents um uh storable like um indexable in the vector index so if we look at the make file um what we're looking for is the uh make Vector index command um and so what this is doing is it's running this um this file from the app or or this this command from the app using modal so modal run just says hey um I want to run this using modal which allows me to run this on the cloud in models infrastructure using the environment that I specified um and we're going to run this create Vector index uh command from this file um and we're going to point it to the specific database that we set up and the specific collection within that database where we want to store the stuff so these are configuration level things that you'll set up when you're um when you're building this application for the first time um that are specific to your instance of mongodb but the logic is contained in this app.py file um and so if we search here for create Vector index then we can see that this is a um a function like a modal function so this is something that we can call and it'll run it on um the image that we defined above um so again it's a it's a it's a uh a dbn container with um some additional python requirements installed including um face for Vector storage and similarity search um mongodb clients gradio for simple web uis um open AI requirements like that um so on that machine when you run this command what will happen is you will um connect to this database um you will pull the collection so again this is defined in your your in your um in your environment you'll download the documents from that collection you will then prepare these documents for the vector store and we'll go into a little bit more detail about what this file about what this command does but basically what you're doing is you're um taking these documents which might be large um you're splitting them into chunks um and you're keeping the metadata alongside each of the chunks so you can store alongside the um the chunk what you know all the relevant information about the document that contains it um then you uh like we then then we basically create an index in our um in our embedding vexter store and we um uh we index each of these chunks um and then we save this embedding index locally um so again simple example so we're following a simple pattern here of like creating and storing index locally um but in production you might be using a vector database uh pine cone or something like that to do this any questions on what's going on here um I believe if I remember correctly this we're just using the um like local uh face for this yeah um but you could swap this out for a pine cone if you want to do this in more like production oriented way other questions mhm uh can you can you say that again so face is both embedded model and the vector store right so face is just the vector store um face is just the the vector index um and so when we run this function um uh so we're running this um embedding engine um which is can actually run these embeddings for us and so we can look at that what that does in a second okay go ahead is that model model is a cloud agnostic so when you run the kind of like a function into the image of the container you can actually like tell model gcp or AWS or somewhere else MH yeah so so model like you can think of it as like um it's like AWS Lambda or something like that it's like a serverless um kind of function environment but um designed to be like really easily used from a python environment um and so when you what's actually happening behind the scenes when you run a function like this is it is um building a Docker container that contains all of the um requirements that you specified all of the secrets that you provided things like that um it's running that Docker container on a remote machine somewhere um it's running the the python code in that Docker container on a remote machine somewhere and then it's transmitting the results back to you um or back to the calling process um in a way where the the calling process doesn't really need to know where the code was executed cool um okay so let's keep going here um let's dig into this uh this this vexor um thing here right so we we're kind of glossed over some of the details here where we um ran a function called get embedding engine um that is provided by this vex or um uh uh part of the code base and then we use that embedding engine to create a vector index so what's actually going on here um so let's look into this Vex store file and um if we look at the uh the create uh G embedding engine function then um what's we can see what's going on behind the scenes here is we are just creating a um Lang chain class that wraps the in this case the open AI embeddings for uh text um embedding ada2 and um this is just like a a lang chain wrapper around the open AI embeddings um that makes it easy to like call those embeddings on a chunk of a document and so this you can think of this embedding engine that's returned as just like a wrapper around the open AI embedding function um and so if we look at how this is used we pass this embedding engine into this create Vector index function so we created a vector index um using a uh an index name the embedding engine that we just got as well as the chunks of text and uh metadata associated with those chunks of text so what is what's actually going on there when we call this create Vector index um so again this is this create Vector index is wrapping some Lang chain functionality and uh in in particular the Lang chain functionality that we're wrapping is we're importing this wrapper that Lang chain has round face um and then what we're doing is we are um looking at all of the um files that we're trying to embed um and we're creating the face embedding from these raw texts um by passing in the underlying documents as well as the embedding engine and so what's happening here is that the Lang chain wrapper around face um takes care of the work of running the embedding engine which is just again a thin wrapper around the open AI embedding function on each of the chunks of documents and then in indexing those in face so face is not running the embeddings uh Lang chain is running the embeddings um but Lang chain has provided kind of this nice wrapper functionality around face that makes a consistent Behavior between different Vector Stores um for how to get documents into the vector store for given embedding function and so the nice thing about this is um so the not so nice thing about this is it obscures a little bit what's happening uh under the hood here which is like we're actually calling the open AI embedding function on each of the documents and then we're taking those chunks um those embeddings alongside the metadata and we're indexing them um in this face uh Vector index but the nice thing about it is it makes the behavior of how you do this consist relatively consistent between Vector stores so if we wanted to just um if we wanted to switch Vector stores and use something like pine cone then we wouldn't have to change too much code um we could mostly just uh import a different Vector store and change some like basic configuration about it any questions on how we're like taking the documents from the document store um embedding them and storing them in the embedding index uh can you talk a bit more like tokenization and um because there's a prepare function because we need to chunk embeddings uh sorry chunks the documents and add met so how what's your intuition behind choosing the like how to tokenize like chunk size and like overlaps yeah so how to tune the chunk size um so unfortunately it's pretty empirical like it's uh it doesn't always work the same way for every task um I can show you what we did here um so we um if you remember we called this function called prep documents for Vector storage um and uh the way that this function works is it's um importing again a wrapper from Lang chain and the wrapper from Lang chain that we're using is the recursive text splitter um and so we create this recursive text splitter um that uh uses a chunk size of 500 with an overlap of 100 um and uh and then we basically um split the text for each document um into chunks of around 500 with an over AP of around 100 um why chunks of 500 and overlap of 100 there's no real like theoretical reason for why you would do it that way it's more just like you have to try a few different things and see which seems to work best for your use case um trade off being like if you um like if you have smaller chunks then there's more to search over um so the likelihood that you get the wrong thing back is higher um but then the chunk itself contains less information and so if you get the right thing back then um that it's it might be easier for the model to process or cheaper for the model to process at the end um but it might be losing also some surrounding information like if it's just you know the if it's if you know in the limit if you just returned like 10 tokens or something um then you would just be looking for like keywords and you wouldn't actually understand the meaning of the keywords in the context of the broader document um so I generally think like probably better to air on the side of using larger chunks um with some overlap rather than um trying to make your chunks too small and like how much do you invest into optimizing this so just like try a little bit and move on kind of yeah um I think like I would probably just pick something that seems reasonable to start like you know chunk size somewhere between 500 and a th000 um and then what I would do is like I would follow um like as you debug the system um when the system fails you you're going to want to look at why so if um you look at a failure and you look at kind of the execution trace and you see that like actually I don't think I could have figured this out because the model doesn't really have like this chunk of information that it got um seems like it doesn't really contain enough for me to really answer the question then that might cause you to think like well let's see you know maybe one hypothesis is that increasing the chunk size could help the model avoid this kind of of failure and so then you could try like increasing this chunk size parameter um rerunning your evaluation and looking at the results again and seeing like okay did the model get the answer right um if not why not like was I wrong was it actually not the fact that the model didn't have a big enough chunk of information um or maybe like when you increase the chunk size it cause some other problem go ah no not necessarily so it means that any like um information that you need from this document should be contained in in the chunk of size 500 but um overlap between chunks can also create problems like you could have a um some information in the document that where in order to understand the information um it it the model needs to look at two chunks next to each other um and so that's where the overlap comes in because if like a sentence if like a if a complete thought or a complete piece of information starts in one chunk um but then is finished in the other chunk if there's no overlap between the chunks then that's going to happen a lot but if there's overlap between the chunks then it's less likely to happen because you get um you have the um like that information is contained in both chunks um and so the more overlap you have the less likely is that you have like a uh something that's important for the model to understand that is you can't understand because it's not fully contained in one chunk um but the trade-off is you're creating more embeddings um because you're duplicating some of the information right and what happens if a fact uh well particular piece of information is longer than this overlap it means that uh this fact is still split between two chunks chunks between any overlap at all yeah uh what will the out outcome be well the outcome would be um you might retrieve a document that would be relevant but the model can't interpret it correctly because it doesn't have all the information it needs to interpret it and that kind of failure happens all the time with these systems like the chunking is I think one of the most um one of the easiest things to get wrong about these systems and one of the hardest things to tune and like if you think back to the open eye example um you know if you remember there was like that that chart where it showed like the increases in performance are trying different things and at first they were like 45% and then the first jump was like 45 to 85 or 45 to 65 um and they tried three things right it was like uh one was hypothetical document embeddings another was something else and then the third one was um tuning the chunks and the only thing that actually worked was tuning the chunks and it had a huge increase in performance so um this is a place where you just you know mostly need to be like pretty experimental to see what works for your for your document Corp thanks so when you have the overlap uh there is no restriction that you need to pass just one of the chunks right you could send it like the top three or top four and even if the answer is not there towards the end of the other chunk it should still be able to retrieve from The Continuous part right there is no limitation on that yeah um so the the trade-off is like with smaller chunks it's more likely that you're going to get an incomplete thought in one of the chunks but then the chunk itself takes up less of the context so um if you have small chunks then that means you can pull a lot of chunks and pass all of those chunks into the into the models context um and so that means means that like at the limmit like let's say that you have one chunk um let's say that your chunk size is so large that you can only fit one chunk into the context then that means that you have to your your retrieval system has to be really really good um because it always has to get the right chunk it always has to get the right Chunk in the top as the top one um on the other side like imagine that you have many you can fit many chunks you can fit um 100 chunks then that means that the right answer just has to be in the top 100 so so it's putting a lot less burden on the information retrieval system um but there's a lot more like it's a lot more likelihood that a chunk will not really be able to fully answer the question cool um okay so let's see where were we here um let's go back to our uh our make file actually let's just ground this in looking at the setup again um so again like you know skipping through this this is just your setup um but the first thing that we did is well after we extracted the document and put it in the document store then we created our Vector index and so again what was happening there is we called the appropriate Lang chain wrappers to pull all the documents from our document store um chunk them into chunk sizes of 500 with an overlap of 100 um embed them using open AI embedding model um and store them alongside the metadata for the document in our Vector store which in this case is just a copy of face running locally um and we save that index to disk so the next thing that we will do is um we will um actually run the back end um so let's look at what we're doing when we're actually creating this backend web service um so what we're doing when we run this backend service is we are um calling this uh task called run backend model and um what this task is doing is uh we are running a modal command um either serve or Deploy on this file app.py and so let's look let's go back to this app.py file which defines a lot of our application logic that's running in the back end for modal um and let's look at what this task is actually doing oh apologies something I um oh uh sorry this is this is actually just a modal thing um so we're either um serving this as a uh um as a like a a local thing where it changes to local file will trigger an update or deploying it so it's something that other folks can use um but either way it's running this like this full app um and so like the kind of front end to this is the web service itself and so this is we're telling model um I want to run this as a container again that is running this particular image um and it's connected to this particular Network file system but one of the cool things about model is that you can also um take this running container and expose it as um a like expose a web endpoint on this container so that Services outside of model can interact with it and um model makes it really easy like you just Define this additional wrapper here um which is you define a uh a web endpoint with a git method um and so what this service is doing is it's taking our Q&A chain um and is exposing it to queries via this web endpoint so it's uh like taking a query which is the request that we're going to send to this to this server um it's passing this query into this function called Q&A and then it's returning that answer so what is this Q&A function doing um so this Q&A function is well oh like one like fun detail to know about this Q&A function is um again if you look at the modal definition we have here like one of the um pain points about using serverless functions is that like when they uh when um you know if you're constantly calling the serverless function you can latency is really low because the uh container is constantly running but if you are not constantly calling that that serverless function in a Lambda or something like that then what happens is um it'll spin down um and that's an intentional benefit of serverless functions it saves you money because if people aren't using it you're not paying for it that's like one of the core benefits of serverless but um it creates problems sometimes because then when you call it again um the system has to spin up the the container again and that takes time um because it has to load all the information from dis it has to find an available machine and so you can uh in in modal you can specify a parameter called keep War which says hey um don't get rid of this container even if no one has called it for at least one second um and that means that you're going to have like if you have uh requests that aren't happening very frequently um you'll have a lot lower latency for those requests as long as it hasn't been more than a second um so what let's come back to this Q&A function and let's talk about what this is doing so again we're taking this query that the user passed the question that you want to ask about fsdl and we are um running this through our Lang chain uh and so we're loading from Lang chain um a loader function to load um this default question answering chain and uh we're going to call this model we're going to call this chain with a particular model as the back end and in particular it's going to be this chat open AI model which we'll talk a little bit more about in a second so what's going on here is we are recreating our embedding engine um and again if you remember this is just a wrapper around the open AI embedding function and the reason that we're um creating this again that we're sort of reinstantiate this wrapper around the Open Eye embedding function is because we want to um be able to embed the query and then use that to find similar documents from our Corpus so uh what we're doing here is um we're doing exactly that we are finding the sources that are the closest to the query um in the vector index and we're returning those along with their closeness scores um and the way we're doing this is that like the the uh Vector Index this like kind of um Lang chain wrapper around the vector index provides just an easy to use function called similarity search with score which um takes the raw query um embeds it using the vector um uh embedding engine and then finds the three nearest neighbors so um and we get as a result is like the sources the chunks that we're going to use as well as the scores the similarity scores so then the next part of the chain is what do we actually do with those similarity scores or with those with those chunks um and those similarity scores so next thing we'll do is we'll instantiate this um this Lang chain wrapper that wraps gp4 with certain hyper parameters which in this case we're using a temperature of zero and a Max tokens of 256 um and then we are instantiating this uh question answering with sources chain which is going to wrap a lot of the functionality of um you know taking the the uh the question the chunks documents and uh wrapping them in prompts in order to answer the question um and then we are running this chain this chain that we just loaded on our input documents like our source documents as well as the question we take the result and then we just return that back to the user so most of the work here is happening in two different places the first is like actually calling the um embedding store to pull the nearest neighbors and then the second is this uh QA with sources chain um which is taking as an input those input documents that we pulled as well as the query um and then it's passing those into a prompt that it sends to gp4 um and uses to answer the user's question and so we can pull up Lang chain and look at what this um this QA chain is actually doing um but I'll pause here first and see if there's any questions yeah mhm yeah there'll be quite a bit of latency there uh well try it out how's the how's the latency look H say how how's the latency look on Discord I did not try but I'm asking uh because the chunk size sorry uh the sources could be of any arbitrary length right mhm uh and also another question I have is uh so we are assuming that the question uh because you're looking at the nearest neighbor right um so the question must be related to the source and you're getting all the sources related to the question so how accurate is that similarity score going to look like yeah I mean I think um like I think the the there's no way to answer that other than just to tell you try and let me know what you think um it's a the it's it works surprisingly well but this is a pretty naive implementation of this um so like I think you know a next step for this would be to look at the the queries that um users have had in Discord um figure out where it's working well where it's not working well and then figure out what part of the system we need to tune in order to make it work better um so is it that you know we're retrieving three documents and each one of those documents is a chunk of 500 is that we normally retrieve the right chunks and our prompt is not right or is it that we the more likely thing which is that we don't always have the right information in the context because our information retrieval system is not working that well um so the first thing we'd want to do is diagnose that and then the Improvement that we'd make to the system would be dependent on the diagnosis that we had cool um okay so let's take a look at Lang chain um let's see if we can find this in GitHub um so Lang chain um it's constantly changing so I hope I remember where this is actually located um but one of the ways this is organized is have a bunch of things in Chains um and one of these chains is the QA with sources chain and so if you remember we had this function called load QA with sources so let's look in this QA with sources chain and um uh let's find this function that we calling and let's look at what this function is doing so um one of the things one of the unfortunate things about Lang chain um is that it's doing something pretty simple but it's like wrapped in a lot a lot of um layers of misdirection so it's going to be like a little bit harder to explain this than I think it should be um but yeah let's see if there's a better place here where we can I don't think I'm going to be able to I don't think it's like the most efficient to walk you through the whole code um but let's see yeah uh there's a lot of code here to do something very simple yeah so here's like an example of what's going on under the hood just to like I think the most interesting thing to look at is just the Bas prompt itself um because this determines the logic for how the chunks and the question are combined to um to to produce an answer um and so like the simplest version of this is this uh question prompt template and so this is um well okay so I think in this case this is a version of this prompt that is um meant to be performed over uh when you have more chunks than canit in your context just as an example of like one of several variants that they have here um and so the way they're doing this is they have this question prompt template which says um hey I'm going to give you a piece of a long document and then um what you are supposed to do is determine whether this text is relevant to answering the question so here's the context like here's the piece of the long document and here's the question and then the model responds with any relevant text um so this is something that's meant to be applied to each chunk of the text in independently um then once it's applied to each chunk of the text independently each of these results are pass to this combined prompt template and what this combined promate template is doing is it's um like combining these extracted pieces of the text and creating a final answer and you'll notice here that like if you don't know the answer just say you don't know don't try to make up an answer just like a common um uh instruction that you'll find in like many many prompts uh and they also have like a couple of few shot examples here um just to like provide the model with um additional context like probably I would guess for the purpose of formatting to make sure that the model um knows understands the way that it's supposed to format things and then finally the question and summaries are uh are passed in here so this is like um I believe in this chain they have like different ways that you can run this chain um they have the map reduce prompt which we just looked at um but then they also have the stuff prompt where um you just basically have a single prompts that says um hey take the the following extracted pieces of this long document and create a final answer by like reading each of these pieces um and so rather than applying this map reduced logic in the chain where it processes each of the chunks independently decides whether they're relevant and then passes those along to a final prompt instead in this case what it's doing is it's just taking all of the um all of the results and the question and producing a final answer and um there's a bunch of kind of logic in here um that is designed to have like a bunch of reusability between the different chains and prompts that they have but under the hood all it's really doing is it's taking these prompts it's filling in the prompts with the context and the question that you're answering that you asked um and then it's taking that filled in prompt passing it to open Ai and returning the result to you in a standardized format um so maybe this gives you an idea of like why a lot of people don't Lang like Lang chain that much um how difficult is to explain what's actually happening um under the hood here but it's it's a still useful tool if you want to like try out a bunch of these ideas quickly any other questions on this yeah so we we use um we return three chunks each of those chunks is size 500 tokens um and then the prompt that we're wrapping it with is like not very much it's I don't know I would ballpark this at what maybe another 100 tokens um so we're using probably around uh 1600 tokens plus the response and the response is capped at I think we said 256 so it's going to be around it's gonna be under 2,000 tokens for for each one yeah any other questions where you basically execute task in or in an orchestr fashion or is it just like a software where you just tell like do step a b and c and on um Lang chain is a library um that just wraps the vector database and prompts functionality or vector database and llm functionality um and then includes a bunch of examples of how to combine those two pieces of functionality into to a chain with a specific predefined prompt um and so what it's useful for is like there's probably around 50 or 100 prompts like 50 or 100 chain examples in here um and so if you don't really know what kind of chain you want to use you can find like you can find a lot of useful stuff in Lang chain um a lot of useful ideas of different chains you could run um but the functionality is pretty thin because it's just um it's just python code that wraps those two thing like that wraps the vector database and the llm um combines them with some logic and a prompt and then calls those other services and so you would need to take care of Hosting this and um doing any like infrastructures out of it yourself got so in this example the computer is happening in the model or the docker container where you're running this right so in this example the computer is running in the modal like the docker container that modal is managing um and then it's running this uh Lang chain Library which uh manages the the calls out to these other services so the calls out to mongodb and um uh face and um opening eye yeah so um these prompts are all like structured as F shot so they have like um they have some instructions like hey here's what you should do and then they have some examples and I think like I suspect that really like these examples are more useful for like ensuring that the model responds with consistent formatting than anything else um because this is a pretty General prompt right so you wouldn't really expect the model to be able to understand how to do question answering well just from three examples um so we're using Gantry for like logging all of the inputs and outputs um so that we can use those for improving the system after that to the whole system yeah totally different question uh I'm unable to access the Discord link uh do do I need an invite for that um yeah so there's a invite code that I had in the slides here is this the I can't access that it says uh it's invalid link or maybe I'm not invited so H um was everyone else able to use this worked okay um are you maybe already in the Discord is that possible okay um well I can send around like I can try to make another link and send it around via email as well so folks um it's uh so folks can stay in touch with each other and there's a bunch of other like channels and stuff um that are pretty active where people are talking about LM apps cool well that was sort of like a um a quick tour of that codebase I would encourage you to like try running it on your own and try to make some improvements um I think there's a lot of low like it's int like intentionally a pretty simple implementation of this Discord chatbot um just to show you like kind of how simple these things can start um but there's a lot of room for improvement like tuning the chunk size um trying out different prompts trying out different chains um uh like adding some more advanced information retrieval techniques um that could probably make this a lot better and so I think it's a good starting point for trying out some of these things on your own um but I guess to kind of wrap things up um I think like the the main thing that I want folks to take away from this is you know it's really cool time to be working in AI because we have access to this incredibly accessible and highly magical feeling technology um but um llms may feel like magic but they're not a silver bullet um and in particular it's never been easier to build prototypes um with llms but it's just as hard as it's ever been in ml to build like really consistently working production systems and there's a few reasons for that the first is it can be hard to get LMS to do exactly what you want um it can it's easy to get them to do something reasonable but it's hard to get them to do what you have in your mind and so we talked about different ways of achieving that like prompt engineering uh task decomposition and fine-tuning as ways of specifying to the llm what exactly you want it to do um llms also you know they know a lot about the world they're trained on the whole internet more or less but they don't know everything that you want them to know they don't know about your problem they don't know about the things that happened yesterday they don't know about your emails um your company's documents and so we talked about like a um a set of techniques that you can use to give llms broader knowledge of the world U by augmenting them with tools and information retrieval um and uh and giving them giving them ways to pull in other data to use to answer the question um and then finally it's really hard to go from a prototype to a reliable production system there's all kinds of different things that you can try um how you Chunk Up the documents what is the text of your prompt um uh like how your information retrieval system works all of which there's no right answer to how to do it it's it's all empirical it's all just based on like um what is needed for the specific task that you're trying to solve and so we talked about a more systematic approach for building up from a simple application to something that works well for your use case um that's grounded in these ideas of llm Ops and uh a process for doing test driven development where you um try to ground the changes that you make to your system in measurement um by running evaluations um rolling out your system to a broader set of users um capturing the feedback that you're getting from those users using that not only to make your prompts better or your uh your whole system better but also to make your evaluation better um so you can create this sort of like positive flywheel effect over time um and we put it all together into this like kind of what's emerging I think is like a standard um anatomy of an llm powered application where the user's request is processed by an llm app that um is under the hood just one of a base model from open AI or anthropic or maybe one that you're hosting on your own um but that model is wrapped in a lot of logic um that logic includes the instructions that you provide to the model in your prompt or your chain of prompts um as well as the way that the model can access external information and um as a developer of this system you will want to build um an improvement system around it that you can use to understand what's working and what's not working um and use it to like make changes to your llm app to bring it closer to what your users want um and we talked through the different pieces of this system as well as looking at like two simple examples of concretely what a system could look like um one that works pretty well today in the Discord bot and another that works really terribly today but um might be the easiest way to build these things in the coming months which is the um the uh open AI agents API and um yeah that's most of what I want to cover today um happy to pause here um answer any additional questions that folks had or um you know talk about whatever else is on your mind about building LM apps is it a uh reasonable practice to say when you get a prompt question from a user Q&A uh and then you find find a nearest neighbor find a bunch of chunks so that's sort of deterministic so next time around you get the same question uh is it reasonable assumption that I can cach the results and then serve it instead of going to open AI each time uh and what's your thought because I know going to open AI each time also gives you different results uh but if the user is already satisfied with the current result you might always pick it up from the cash yeah I think caching is a very reasonable thing to do as long as you're aware of the tradeoffs um which is like uh if you update your um embeddings or you update your prompt you need to remember to invalidate the cach um and you know if you want the results to be non-deterministic like if you want the user to be able to ask the same question multiple times to get different answers um You probably don't want to do that but I it's pretty like I see most people doing caching and yeah I can save you significant amount of money well basically like what you just mentioned I don't know if you saw that in the open ey dep day they actually discuss about this kind of new functionality they are planning to release the seat when you can actually specify the seat of the model so the model going to be a little more deterministic rather than probabilistic so how do you think that this could potentially be implemented in a model that is more a probabilistic model um how the caching can be implemented how how you are thinking about the seat you heard that he they're planning to kind of if you specify the seat the output that you're going to get should be the same every time that you call the is that is possible or how do you think that this can be implemented in this kind of systems are you are you asking how a seed can be implemented for systems other than open AI uhhuh um it's surprisingly tricky um that's why I took open AI so long um I don't know like you know you you just need to make sure that you're controlling all the sources of Randomness um usually there's um sources of Randomness um that like pop up at different points in the sampling process where you're um you know normally like when you're sampling an output from an llm um the like what you you might think you might do is you might at each step select the most likely token um but it turns out that selecting the most most likely token at each step um does not produce the sequence that is most likely um instead what you want to try to do what ideally what you'd want to do is you'd want to select the most likely sequence of tokens um but that's computationally infeasible because there's a massive number of possible sequences um uh because sequences can be pretty long and the each token in this in in each element of the sequence could be any of your tokens so it's a huge explosion of possible sequen is and so the way that people um typically do this is through sampling um and different sampling processes using like techniques like beam search um to like do an approximate way of searching over sequences and that's where a lot of the randomness comes from so um never really tried to make this work in open source LMS I'm not sure how much of it is like built in versus things that you need to uh uh be careful about but generally like I think with sampling from um deep learning models I think the things that people get wrong or like um often times you need to set many random seeds like you need to set your numpy random seed your pie torch random seed um and like one way I've seen this go wrong in the past is like you set one of those random seats but you forget to set the others and there's like some additional Randomness that you're not controlling for yeah so temperature zero um also does not um guarantee deterministic outputs um but and also I think for evaluation I would set your temperature to whatever your temperature is going to be in production um because the characteristics of the outputs will be different with different temperatures and so I think the seed is the right way to do it because you want to you want to have like an output that looks like production but it just doesn't change um and if you set if you change your temperature then that's will make it more deterministic but it will not make it but it will also make it different than what you're you're going to see in the real world um I think um I would check with them on that it's it's generally available as of maybe a month ago um so I think it's It's relatively production grade I haven't seen people use it for like large scale systems um I think where I see people use it more is like for like data science kind of tasks where you don't you're not necessarily like running your whole application back end on it but you're um running some of your ETL logic or like something that you need to paralyze but you're okay with it failing sometimes um so more like data science workflows instead of like what most people consider production workflows um but I don't know that that means you can't use it for those workflows uh I just don't haven't seen that as much yeah I mean it just totally depends on what your company does um so normally I see model being used either by like startups or like um data science teams or ml teams within companies um for stuff that they want to have more within their control and then for you know bigger companies if you're building like a web application um your company's probably making a decision about how to build that independently of how your building your llm systems and so mostly companies will like piggy back on the existing infrastructure that you have for that with with temperature zero where is the IR Randomness coming in from um yeah it's a good question I was hoping no one's going to answer that because I don't actually remember um but try it it's not it's not it's not deterministic yeah okay one more question how much does for ask deal how much do you guys pay for it to be hosted right now with open ey being used like very little I I want to say it's less than $10 a month oh okay thanks and it's like people use it it's not yeah is temperature zero about Randomness or is about a kneeling um temperature zero controls uh the sampling process um so it is the effect is that it higher temperature um means that the model does not concentrate as much on the most likely tokens it samples more broadly from the set of available tokens um which makes the outputs more random but setting temperature to zero does not make the outputs totally deterministic um in the Lang chain can you chain like multiple llm like models to run an response and if that's the case do application developers have to worry about what kind of embeddings you're using because if one llm uses a type of embeding like in open in this example and if the other one uses L's like like do you have to like ensure you have the different embedding models stored in your vector database um yeah so Lang chain does let you swap things out pretty easily that's one of the advantages of it it's like if you want to try out both um opening eye embeddings and coher embedding settings or something like the they have the wrapper code that they make around them um makes them not totally interchangeable but more interchangeable than they were if you're just calling those companies apis directly um and that's one of the advantages of it I would say um but in terms of compatibility like yes you can't really compare directly embeddings from one provider to another um or even from one model to another so you need would need to re-embed your entire Corpus in order to run that experiment maybe we'll take one more uh so uh Aurora and all these databases they are coming out with all this Vector indexes right so is there like a real use case for pine cone and all the newer ones uh because I see Aurora is like one solution that you can use in production systems where you can put your data sources slash you can also do vctor yeah um I think like so the decision tree I would have is like um do you actually need an embedding database at all um in some cases no like you can just do um you can just store your embeddings on disk um and then compute the embedding distances manually by hand which is not that dissimilar from what we're doing in this example um instead we're using an embedding index called face but we're just storing that on disk and we're running all of the similarities locally uh so the question is like when would you ever need more than that um and the answer is like um well one reason why you might need more than that is if you want your search to be more complicated than just Vector similarity so if you want to have other information retrieval functionality um another reason why you want might want to do more than that is for performance or scale reasons um performance scale or reliability reasons so if you know um storing a vector database on disk um on your machine is uh not going to be very good for um reliability scale parallelization things like that um and so you might want an actual database for that um yeah so like if your index doesn't fit easily on a single machine um yeah my question was mostly not on the on memory itself uh my question was mostly this newer generation database companies like pine cone vv8 all these companies what is their business case While others can easily uh adapt like Aurora and mongod and all these people can adapt and provide Vector searches um natively I would say and that way you can also minimize let's say in the demo you had a mongod DB and a pine cor so if you actually go run it in production you have to monitor both mhm all right operationally it's expensive mhm all right so while if you have just one database you can just do everything with it yeah I think um you can just have one database for a lot of cases like you could just um you can run Vector similarity search in um in um uh postgress in like most kind of commonly used form databases and so for a lot of companies I think that's the right decision um but there like I guess the business case would be um any every technology shift has led to a new form of database um you know uh itself you can make the case for like why did why should exist um when you can just use postgress for everything You' use for um Well turns out that people use databases for a lot of stuff and uh and a lot of like really really core development tasks depend on databases and so um databases themselves are highly highly specialized to the tasks that people use them for um so the argument that I think you would make about using something like pine cone is um well we are going to do be doing a lot of vector DB operations and so we should use the tool that's best suited for the job um and that ends up being that could end up being cheaper depending on your use case um than the operational cost that you pay by now having more than one database that you need to maintain uh for a test driven development section are there any existing resources of Frameworks that are available on the Internet or is it still very new and very like still still very new yeah um I think there's like beginning to be some resources on evaluation um so opening ey has an open source evaluation Library um there's a number of startups that are working on llm evaluation um including us uh we're building some stuff around this but in terms of like the entire test driven development framework um I haven't really seen anything yet um but I think it's coming and it's something that you can do on your own like you can uh that it's it's not that like no one is doing it um a lot of like a lot of companies especially the best companies follow a process like this um but there's just isn't really something that has been you know uh packaged up and systematized and one of the reasons for that is because the details of doing it depend a lot on the thing that you're trying to build cool um well let's wrap here um yeah really appreciate you all attending the class um hopefully you learned something from this um always really interesting to hear um questions people have and uh and stuff that you all are working on as well so um we love to stay in touch uh I guess you all are in the Discord or feel free to message me on platform of your choice if you uh weren't able to make it in the Discord and I can get you an invite and um yeah um thanks for uh thanks for being a good [Applause] audience