Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 4

Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 4

Recording: Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 4

so I'm going to um just ground us again in this slide that we've looked at a couple times now um the anatomy like emerging structure um that a lot of these llm powered applications have um and so we talked about you know the what the user sees um the language um user interface that the user interacts with um and when user interacts with a system like a an llm powered application behind the scenes um usually that application consists of a few pieces there's the underlying model like you might expect from open AI or anthropic or wherever you get wherever you purchase your models today um then there's the instructions which is the prompts um all the fun prompt engineering magic that goes into that um or the chains the the um the more logical sequence of prompts that you can use to tell the model what to do and then more often than not these days those instructions also interact with external information that you can't fit directly into the instructions um and that's via tools it's via retrieval systems um Vector databases or more generally functions um so that's kind of the the structure of the llm app um and that's what we've covered so far um the next thing that we're going to talk about is um let's let's talk a little bit more practically about how you get from a prototype of all these systems working together to something that works really well in production um and what we're going to cover is the like what I think of as like the Improvement system for the llm application um and uh and we'll talk about some of the pieces of that now oops struggling here um so we're going to talk about a process for building and iterating on language model powerered applications um that you can think of as being like analogous to test driven development um but to start off with we're going to cover how do you actually choose which model you want to work with um there's so many models available today closed Source open source um and everything in between um how do you pick which one you're actually going to use so tldr um and this is kind of my opinion at to take but what I encourage people to do is um well like first of all just understand that there's no best machine learning model there's no best LM the best model um depends on your use case and in particular the trade-offs in your use case between um things like the quality that you're able to get out of the box for your task the speed like the latency with which you're able to produce results which has a big impact on the user experience um cost obviously um and then how like hackable or extensible the model is um because you know even if you can't get good out of the box performance the more customizable um the more you're able to actually squeeze more performance if you're willing to put enough effort in and then um lastly like data security and license permissibility um but I think the short answer is like I almost always encourage people to use gp4 to start um and there's a pretty simple answer for why um which is that like I think um generally you know for AI applications in general and llm applications in particular the first thing that you need to do is you need to um basically prove whether the T prove to yourself whether this task is feasible um and prove to your organization whether anyone actually cares if you solve it um so if you if you are um if you're working on a new task then most of the time I wouldn't really think about cost upfront um I would start by making sure that we can do this task with no constraints and that if we do it people will care um the easiest way to do that is with the most powerful models that you have available to you which is for almost every use case is gp4 um there's some exceptions to this like if you're working on an environment where you know this needs to run an embedded device or um or something like that where yeah it's probably not like the feasibility question is not something you'll be able to answer by using gp4 but in most cases I think you start with gp4 make it work and then you make it cheaper or make it faster um so one question you might ask or one question I get a lot is like do you want to use a proprietary model or do you want to start with open source um so the short answer is proprietary models are better um they are higher quality today um and they're much much easier to use and interact with um serving like hosting open source models introduces infrastructure overhead it's a huge pain um there's good tools available for it now but they're not that good it's still going to create a lot of effort for you and your team um and so I uh and it's also going to be more expensive um not in the limit in the limit open source is cheaper like if you're willing to invest a lot in the infrastructure side but for you when you're starting out um it's going to be cheaper for you to pay open AI run your models or pay anthropic to run your models than it will be to run your own open source models that's because they have cheaper access to gpus and they have invested a ton in like making serving and training really efficient um so proi proprietary models are better um and so you should use those um unless you really need open source so why might you need open source um I think the main reason like the reason people don't really talk about but I think in my mind like the most important reason is that open source models are much easier to customize so if you're doing something like if you want the maximum flexibility to experiment and to try things and to push the envelope a little bit of what these models are capable of then um that's a good reason to use open source um another reason that people kind of site for using open source is to like like own the whole stack myself or to like respect data security um I think that's in some cases it's a decent argument but I think it's like a very very overrated argument honestly um I think um most of these like llm providers um have you can sign contracts with them where they won't touch your data they won't train on your data um they have pretty good security practices um if you're worried about open AI security practices you can you know deploy a mo your model in your own Azure environment or on AWS with anthropic or something like that um so I think like owning the stack or like data security in my mind are not really like it's an argument I hear all the time for using open source but I think it's a pretty weak one um and uh uh yeah and like the way I would think about this is yes you want to own the IP of your AI powered products yourself but um owning the IP like to me it's like um arguing that you need to use open source um because you want to own the stack is like saying oh we want we're going to build our own we're going to build our own um data warehouses because we want to own all the infrastructure ourselves it like just doesn't make sense if a company invests in like syncing all this money into building something better to um go try to replicate that yourself um unless that's really where unless that's really where you're going to create value um as a company in most cases the value that you're going to crew is much more to do with the application that you build around the model and the data set that you build around the model than the core model infrastructure itself um other good reason to use Open Source by the way is that in the limit it is cheaper and so if you're very cost sensitive then um you and you are willing to invest a lot upfront in driving cost down then you can achieve a lower like per unit cost um using open source um I think one thing that's important to know in this in this field of like uh llms which as boring as it seems is open source licensing and the reason for that is because um you know typically when we think of Open Source our mental model is like oh this is just free code I can use for anything I want to and there are some models that operate that way um under licenses like Apache 2 but you do have to be careful in the open source um model world because there's also um a lot of models that are released under restricted or non-commercial licenses where you know you can use them for research um but you like they actually either explicitly prohibit commercial use or there's kind of terms in the licenses that are ambiguous about whether you can um use it for commercial use or not so just a thing to pay attention to um because in the open source llm World a lot of people want to say they're open source even if they're doing something that is not actually open source um so you know like one of the big uh factors is for considering which LM use is the performance like the task performance and just like we discussed with um embeddings in information retrieval there's really only one way to know which llm is going to work best for your use case and that is to try it on your use case to measure the performance of that llm on your use case um you will find all kinds of like General benchmarks that people release about llm performance um and those can be helpful for building like an overall sense for Which models are you know overall better than others but um just because one model is better on a benchmark than another model does not mean that it will be better on your task and so task performance is the only thing that matters in the end um so again most projects should start with gp4 um this will give you proof of concept about feasibility um metaphor is like you know in if you're um doing software engineering work typically you prototype in um like the best practice would be to prototype in the highest level language that you can um to reduce development time um and avoid premature optimization and then uh for whatever things that end up becoming critical path for your code um things that are relatively mature that are performance bottlenecks then you might reimplement those and see or rust or something like that um I think of building with LMS in a similar way like you prototype with the highest level language possible which is gp4 and then um as your stack matures you can start to replace parts of that gp4 stack with smaller faster cheaper models whether they're open source or just smaller commercial models if you run into problems with cost or latency you can consider down sizing um both GPT 3.5 and CLA um are really good choices for this and are pretty comparable in performance um and if you want to go even faster and cheaper like I don't really see a huge difference between those models but it might be pretty task dependent um I would say like of the like of the tiny models like the small options for the models um uh anthropics seems to be like the one that's trained in the most modern way um and yes open source is a viable option but it is definitely a lot more work and so if you're focused on building applications I wouldn't start there um and yeah I would say the the most the best options today are uh the 7 billion parameter model from mistol and the Llama 2 variants but there's new um open source models being released every week and so this is probably going to be a out ofd recommendation very soon um if not already in the week since I made this slide um okay so you have a you have a um you have a model now um now the question is like how do you actually work on your prompt um how do you improve your prompt over time in order to like achieve the best performance on your task um so as you work on your prompts and as you work on your chains how do you save your work like how do you um keep how do you um keep track of progress over time so why are we talking about this um well I I think there's like an interesting analogy here to where we were with traditional deep learning you know the old school deep learning uh pre- llm deep learning back in 2015 when I got the field so what it felt like to work in deep learning at the time was I would train all these models and every single time I train a model I would write down the hyperparameters in a spreadsheet um and I would do that because otherwise there's no way for me to remember what I had already tried and what I hadn't tried and then when I train the model I would save that training file on um as like a just a file on my laptop so I would have it for later um but there was no way for me to reproduce experiments or share my work with a team or anything like that it's just like I would train the thing and I would have a record of what happened and that's only available for me and that's basically it um now if you do traditional deep learning if you're training models from scratch or fine-tuning models um you have access to amazing tools where every single time you run model. Trin you automatically get a full log of that experiment um contains all the hyper parameters so you don't have to write them down it's sharable it's comparable and it's fully reproducible like you could run the exact same thing again and produce the same model um today I think prompt engineering um feels a lot like traditional deep learning did in 2015 so um every time I change my prompt what I'll do is I'll play around with it in like the open AI playground or some other sort of model playground um old prompts like changes the prompts I made in the past are lost to time you know they're just uh they're buried somewhere in a spreadsheet or in um in your your chat GPT logs and there's no way to reproduce experiments Share work with Team or anything like that um and so I think the question is like what is the equivalent of the tooling stack that we'll have for that we have now for traditional deep learning what is that going to evolve into for prompt engineering um I think the the answer is still the the jury is a little bit out there so um you know why was why was this so impactful in deep learning like why um why did this like accelerate the the uh ability for people to build deep learning models so quickly um the reason why was because in deep the Deep learning world you constantly need to go back and check old experiments um so uh the reason for that is because experiments take a long time to run so when I was training models you know the models would take anywhere between a few hours and a few weeks to train and so I I was constantly needing to go back and look at models that were I started training a long time ago so it's really important to have a record of what I had done um and also I often ran many experiments in parallel like I would run a hyperparameter sweep um I would you know run train on like uh 256 gpus at a time because it's the nice thing about working at open AI even back then we had access to a lot of gpus um and so you know being able to I couldn't keep track of all that out of my head um and it's way too easy to end up repeating yourself for like getting lost in a loop without having track of uh what you were doing before but prompt engineering today doesn't really have the same Dynamic um so you might argue that prompt engineering like maybe isn't in need of the same level of uh tooling around experimentation as deep learning was the reason for that is that you know in contrast to deep learning experiments are quick it feels more like writing code than it does like training a model right like I I can just add a line of to my prompt and rerun it um and it's pretty much instant that I get the feedback back it's not like I have to wait days or weeks to see the result um I also usually do experimentation sequentially and most people I know do experimentation sequentially um like you try one change to a prompt you run it you see if that has the desired effect um if not then you make another change um and you look at one result at a time much like you would in like a reppel um when you're writing code and most of the time today for most people this experiment experimentation itself is pretty limited like you are um I'm not like in in the Deep learning world you're trying all these hyper parameters you're running these sweeps trying a 100 different variants of the same experiment but in prompt engineering you know often times the experimentation is limited by the ideas that you have about how to change the prompt or how to change the chain um rather than just like running a sweep over some hyper parameters and so we often we usually try many fewer things in prompt engineering today uh so I would like you don't really need to pay as much attention to prompt management today as you do to as you did to experiment management um but I think there's something that's going to change this which is that um imagine that you had a way to just automatically evaluate like a change to a prompt like is this change to a prompt better um that would unlock our ability to um search over prompts um or run many changes to prompts in parallel and see what the results are like kind of like what we were talking about in the when we talked about um the prospect of prompt engineering becoming automated and so when that that comes to pass um then I think we're going to start experimenting a lot more with the prompts that we have um and uh and so I think that's going to increase the need for specialized tooling around managing prompt engineering experiments today um I think there's kind of like three main ways that I see people doing this so level one is like do nothing just you know make your prompts in the openi playground you know copy and paste them into a text file like the ones that work um you know share them in slack with your team uh save them directly in your code and um that is usually actually good enough um honestly especially if you're building a prototype um but this obviously hits a limit when you're building applications or when you're collaborating with people you need a it it becomes valuable to have a more systematic way of managing these prompts level two like what I see most people graduate to is just track prompts the way that you track the rest of your code um just track them in your GitHub repo and um I actually think this is kind of what most people should be doing like just um you know save your prompt as a text file in your GitHub repo or even save it as a string in your uh like python or JavaScript file um and just Version Control it and collaborate on it that way um I do think like it's slightly better to save them as a text file than as a string in uh in you know python or JavaScript because um I think it makes it easier to like uh share those with like non-technical folks on your team and then level three which I think is like becoming more common but still not very common is to track prompts and a specialized tool for tracking prompts um and I think if you want to be able to run many evaluations in parallel or if you want to be able to decouple prompt changes from deploys like be able to change have a non-technical member of your team change your prompts um without redeploying all of your code um then this can become really useful so I do expect to see more people use these over time as uh as as evaluation gets better and more consistently used um okay so what you know what should you look for in a specialized prompt tracking tool um I think there isn't really like a standard that's emerged here yet um you know a lot of the like traditional experiment management tools from the traditional deep learning world have offerings for prompts now weights and biases Comet mlflow um there's startups that have specialized offerings around this um I I think uh you know like the cloud providers are um building stuff for this I'm not sure how mature it is Microsoft has a tool called prompt flow um so there's no standard that's emerged for this I would say like some things that I would look for if I was if I was building a tool like this is um I would want to be able to decouple prompts from git like I'd want to be able to manage like have a source of Truth for prompts in git but be able to have um like test changes to prompts and have non-technical St stakeholders test changes to prompts without interacting through git um because I think one in a lot of organizations that I talk to um one of the biggest pain points is that like often times the people who are best suited to write the prompts are not the software Engineers um not the ml Engineers either right it's like the people who care about the task um and everyone's use chat gvt at this point right pring writing a prompt is not something that is uh restricted to people who have Technical Training um and so I think a really desirable property of a system like this would be to be able to involve non-technical stakeholders in a way that doesn't um compromise the the quality of the rest of your code and so you know one piece of that is that like you should be able to um it should have a playground component to it right like the non-technical user should be able to interact with things in the UI um and uh yeah um so these are some of the things I think would be like helpful properties to have in a system like this and so it's worth paying attention to this space um to see like if you if you need some of these things now um to try out some of the tools um and if not to just keep an eye and see if one emerges as a standard so um again coming back to my recommendations here I think you should manage your prompts and chains and kit and it's uh it's worth checking out some of these other tools if collaboration um or automated evaluation become a big pain Point all right um any any questions so far um the next topic is like one of the the meteor ones which is around testing and evaluation all right yeah um I don't have a specific recommendation for that um yet but there's there's a bunch of tools that are trying to do this um yeah Lang chain has one um and a lot of the tools in the space like a lot of the starters in the space have offerings around this um so you could yeah you could you could try some of the ones out I think they're all like relatively immature but um um yeah cool okay um next topic is testing and I think like I I think I kind of realized this the last time we taught this class but certainly since then have come to believe that this is like probably the single most important part of everything that we're going to talk about today um because I think a lot of the problems with LMS and prompt engineering today are that like um we're we're trying to make we're trying to like machine learning does really well at um optimizing measurements like if you if you have a number and you want to make that number go up or make that number go down then like the whole field of machine learning has evolved over the past like n decades to be have all kinds of really good techniques for making number go up um but if you don't have number to go up then machine learning is kind of useless I think that's a little bit where we're at with a lot of these LM applications is sort of a lack of measurement so testing or evalu valtion is about how you measure whether a new prompt or a new model is better than the old one like a change that you make is is an improvement or not so the reason why this matters is because you're never going to get things right to start out with you're just not LLS make tons of mistakes um the better that they get the fewer mistakes that they make but in they're always going to make a lot because um they never are going to understand exactly what you want specifically out of the box and to make matters worse just because the new prompt that you made or the new chain that you made looks better on the handful of examples that you have like that you ran it on your laptop does not mean that it's actually a better prompt or model in general it's super super common to see like you improve on five examples but you get worse on five other examples um and so if people rely on your model like if you if you have users um then there you um the relationship that you have with the users of your AI system is a relationship of trust where um users are trusting you to maintain among other things to Main maintain performance on the things that they care about and I suspect that like a lot of what's happening right now in the AI power tool world is like in early in the year when everyone was really hyped about these tools a lot of these things got a ton of adoption and um really crazy usage numbers off the bat um but we're starting to see some of them tail off and I think the reason why is because you know you interact with these things and they feel like magic but then as you start to get a feel for what they can and can't do um the trust that you have in them starts gradually starts to erode um and as that trust erods then you're us churn um and so your job as an AI application developer is to build and maintain trust with your users and evaluations are a really critical way to do that um so let's talk about how to do this for LMS and I think I want to ground this a little bit in um how to how we used to test machine learning models the the old school way you know like um all back in the day in like 2021 uh so in the way that testing and machine learning model uh in machine Learning Works in general is um you start with a data set so in before llm M you would always have a data set because you would use that data set to create your model after llms um often times you start with a prompt instead of data set but in Old School machine learning you'd have a training distribution and You' sample two data sets from that distribution one is your training set and then the other is your validation or your evaluation set you compute your metric like you compute your accuracy on each of those data sets and the difference between those two numbers would give you a measurement for how much you're overfitting to the training data so if your evaluation accuracy if your validation accuracy is much lower in your training accuracy then that means that you know you don't have enough training data or you're overfitting to that training data then you would sample some additional data from production like from the actual data set that you want to test the model on um and that would form your test set um and you'd compute your accuracy on your test set um the difference between the accuracy on your test set and your evaluation set tells you how much um you're overfitting to your EV validation set or um how much domain shift is hurting you right like how much the difference between production ction and uh and training distributions are impacting your performance and then finally um you'd continuously measure accuracy on your production data over time and the difference between the test set accuracy so the model the accuracy on your test distribution when you initially train the model and the accuracy on your test distribution um today um that difference tells you how much your data has changed like how much drift has affected your model performance so that's how we used to think about testing models now why doesn't this work for llms um the first first of all you don't actually have access to the training distribution right like opening eyes is not telling you what's in their training data um and even if you use an open source model where uh then most of the open source many open source models don't have open data sets um even if they have open models and open data sets like let's be honest you don't really know what's in the pile data set like there's so much data there just even understanding what's in that data set is a massive effort into it on to itself so you don't have access to the training set so it's um so it's like really hard to do to do any sort of measurement on the training set um then the other big difference is that like in the traditional ml world we worked really really hard to make sure that our training data and our production data were similar um because you know machine learning only really gives you guarantees when you're eval when you're running their model on data that looks like the data it was drained on but in the modern like World of llms your production data is always different than your training data no matter what right like you're never um because you don't really have control over your training data set and you're always trying it on some new task so this Paradigm doesn't really make that much sense anymore um another big difference is that um which is not always true for LMS but it's often true for LMS is just the generative Paradigm So In traditional machine learning you are doing most of the time things that look like classification um so you have a set of predictions that you're making um and then there's a a real answer to the problem that you're trying to solve like you might predict the bunch of images of cats or dogs and then there's a way to know objectively whether the image is a cat or a dog and so you can objectively measure the difference uh in the predictions that the model made and the real answer uh to to compute traditional machine learning metrics like accuracy but in generative your prediction might be something like this is an image of a tabby cat um and the label might say this is a photograph of a cat right so is this a good prediction or a bad prediction that sort of depends right it like depends on what you're going for your task it depends on you know what um what you consider to be a good answer um so what metric do you use like how do you actually quantify the difference like the the um the accuracy of a model in this Paradigm it's another really hard challenge um and then finally you know a lot of times in generative AI like you are building systems that are pretty General that are meant to work for a lot of different tasks so if you have an accuracy of 90% that's pretty good um but then if you break down that accuracy among the different things that your users care about and you find out that actually we have a great accuracy 95% accuracy for questions about startups U but we only have a 17% accuracy for questions about physics is this a good model or a bad model well it depends right like if you're if you're really just designing the system to be good at answering questions about startups you're probably pretty happy with this but if you if you know that your users need to know about physics as well then you definitely wouldn't be happy with this model um so it's hard to summarize this like diverse set of inputs and tasks with a single number um yeah so to summarize you know these models are trained on the internet so there's always drift um but drift also doesn't really matter in this world um it's qualitative so it's hard to quantify success and there's often times A diversity of behaviors that you care about and so it's so aggregate metrics don't tell the full story um so how should you think about testing LMS um I think there's kind of like two core questions that you need to answer the first question is what data do you evaluate on um and then the second question is what metrics should you look at to do the evaluation so the key thing for building a data set is that this needs to be a data set that is specific to your task so good evaluation data set is um like just really really closely coupled with a good definition of the task to begin with like if you have a perfect evaluation set um that's in some ways you can think of that as like a perfect description of the behavior that you want the system to have and so how do you actually do that it sounds really hard the good news is you don't have to do it all at once like you don't have to build a perfect evaluation set out of the box um what you can do is you can start incrementally you can pull in your language model to help you build this more quickly um and then you can add data incrementally as you roll out to a larger and larger set of users um and lastly I think there's there's hope that there might be a way to do this more in a more Auto made way in the future so we start incrementally so like if I'm writing a prompt um and let's say I want to write a prompt that helps me write short stories um usually what the way I'll start out is like I'll start out by just typing what I want into chat GPT and just playing around and seeing if I can get chat gbt to do the kind of thing that I want um in general and then once I find something that seems reasonable I'll usually move um from that to starting to have a templated prompt where um now rather than just asking it for a short story about uh dogs and then LinkedIn um instead I'll have um the ability to just pass in a different subject that prompt by filling in the template and uh and then what I'll do is I will um evaluate ad hoc so I'll just try out these different subjects one after another um and then the way I start to systematize that is as you find interesting evaluation examples then you can just collect those evaluation examples and organize them into a small data set um so then once I have that small data set of interesting examples rather than just trying out things ad hoc um instead when I make a change to the prompt I'll run that new prompt on all of those interesting examples that I've flagged in the past what are interesting examples um struggling here but I'm assuming that you can see that um there's like two things that make examples interesting um one is examples are interesting that are difficult for the model to do so if you're playing around with your model ad hoc and you find like a place where it fails that's a thing that's worth saving because you'd want to make sure that like um any Chang that you make to the model in the future or the prompt in the future will um perform well in that hard example and then other interesting examples are just ones that are really different right so if you you know if you're if you're like find yourself writing a lot of short stories about dogs and cats and things like that and then all of a sudden you have this idea pop into your head that maybe users want stories about LinkedIn that's really different and so you might want to record that one um because the model might perform very differently for those types of stories so hard examples and different examples are interesting in order to do this faster you can use your llm to help um so l are pretty good at generating data um they're not very good at generating like super diverse data but they're pretty good at like just generating a bunch of examples faster than you might and uh so you can write a prompt like the one below to help you generate test cases and this can be helpful for just bootstrapping having like multiple things to try out your prompt on then um as you as you go you want to start to like expand this data set and try to make it more and more representative of the task that you care about um in lock step with how you roll the model out to uh to your user base so um you can use signals like what do your users dislike um what do your annotators dislike um even what does another model dis like as well as things like um what are outliers you know relative to your current evaluation set um or topics that you hadn't considered and things like that and so you can do this like as you increase your user base like if you have um you know if you if you're just working on this yourself and then next you roll it out to a couple of your friends or a couple of your co-workers then you want to look at the the way that they're interacting with it and collect back any examples that you didn't think of um that are hard or or um different from what you're currently testing on and incorporate those into your test case then as you roll out to your like Alpha users you'll do the same thing except at more scale and then as you roll out to the broader user base you'll continue to do that but again you know with much more data to consider um and so you know what like you might be wondering as you look at this is like okay is there a way to like we have this these high level ideas of like Gathering hard data and Gathering interesting data to fold into your evaluation set um is there a way we could do this more automatically um there isn't really today but I think that there is a uh more I think that there's a possibility that there's a more quantitative way to um evaluate the quality of an evaluation set um so the intuition here is like um imagine that you have like production data that's represented by this sort of blue space um this blue like data distribution and then you have test data like your current evaluation data points that are represented by the kind of dark blue points um and so like test coverage for machine learning models um an intuition that you can that you could have is like um a high test coverage uh evaluation set is one where every data point in production is pretty close is pretty similar to some data point in your evaluation data set um and so in contrast a low test coverage evaluation set is one where there's a lot of data points in production that don't look anything like any of your evaluation data points um so you could imagine formalizing this by trying to find um the data points that are most different from your evaluation data and then incorporating those automatically into your evaluation use cases um slide issue again here well okay if I remember correctly what the slide says um the uh the question is like is this enough like if you just found the just look for the the examples in production that are most different from your evaluation data and Incorporated those automatically what would you miss the main thing that you'd Miss is like a notion of the difficulty right like you um there's a lot of data that your model is just going to automatically do really well on you don't necessarily need as much of that data um to evaluate your model on but on the other hand if there's a bunch of data points that are really similar but your model struggles with all of them uh you might want to collect more data like that so just an idea that I think is like um one of the ways that this is going to become more systematized over time so the next thing that's hard um if you remember is like even if we have a good data set to evaluate on how do we quantify performance um so the uh like I think of this as kind of a decision tree so if there's a right and wrong answer to the task that you're trying to do then you're doing something like classification or regression you're not doing generative and so you can just use the normal evaluation metrics that you would use in the regular ml world and your life is a lot easier if there is no correct answer but you have an example or you have a handful of examples of good answers then you can use metrics um that match those reference answers to the answer that the model generated um and ask the question of like okay is this how close is this to the reference answer so if you don't have um a correct answer it's a really good idea to try to have some at least a subset of your examples where you have a reference answer like something that you want to compare it to but if you don't have that um You can compare to previous answers and you could try to assess which answer is better or you can compare to human feedback um and say like Does this answer incorporate the feedback that a person gave um or you can just look at static metrics like you can just say is this answer good or not um and the key thing here is that like all of with the exception of the first branch of this tree here um everything else um these are all metrics that are defined by having an llm look look at the outputs of the other llm and assess whether those outputs are good or not so in a reference matching metric you have an llm look at uh the the output that your model generated as well as the reference answer and you write a prompt that says something to the effect of like does this output you know contain all the same information as the reference answer or um is it meaningfully the same thing as the reference answer or is it saying something different um if you're asking which is better then the way that you write a metric for that is you'd write a prompt that takes as an input um both of the two answers and says like which one's better which one's more accurate which one's funnier whatever way that you want those things to be better Etc um and so the key idea here is using llms to evaluate other llms sorry I'm like I can't see what's on my slides which is makes it hard to it's like it is showing up there right um H oh there we go okay sorry for the lack of uh presentation mode here um so I I think like what a lot of people say when they hear of this idea of like using llms to assess the quality of other llms is like wait a minute that seems kind of sketchy right because it's like well if we don't trust the llms well enough to um just trust their answers then why would we trust them to check another lm's answers isn't that kind of circular and um I think that's true but um two things to keep in mind one is empirically it kind of works well like you can um LMS are better at assessing other llms than most people are most people that You' pay to do it um and the second is that like just because you have an automated evaluation setup like just because you have llms checking other llms doesn't mean that you shouldn't also have people doing it um and so the mental model I have for the right way to do this is like you should think of your LM your automated evaluation as kind of like the fast eval process um that is verified by slow eval process which is having humans do QA review and so think about it as like different processes if you're a developer and you're working on prompts and you want to like try to find a better prompt or you're trying to fix a particular issue with the model then you run the fast automated eval um because it doesn't make sense for you to pay people to read the outputs or for you to read 100 outputs yourself or a thousand outputs yourself every single time you add a word to your prompt um so you should run the fast evals and just see whether um you know first pass whether the prompt looks like it's better or not but then if you're deploying to production if you're going to put this in front of your users then you should have a QA process a governance process in place that involves people looking at those answers um which can tell you like hey are we um how much do we trust the automated evaluation and uh and uh and give you a better sense of whether this is you should be comfortable putting this in front of your users okay so that's kind of quick overview on testing and evaluation I'm curious if there's any questions on that yes yes um it's it's one of the hardest and I think the most important pieces of this whole puzzle because I think it's like okay imagine if you could do this really well if you just had a reliable way of saying like prompt a is better than prompt B then I think a lot of the like nonsense around prompt engineering will just go away um because then we can optimize it and uh I think also you know even looking Beyond kind of the engineering context of like how uncomfortable and hacky it is it feels to do Prim engineering I think a lot of what's slowing down LM adoption in um Enterprises where I think it's going to make maybe the biggest difference is just like it's really hard to trust this thing in front of your users and so if we had a way to Quant to quantify um that like we know what the balance of performance of this model are then it's a lot easier to make those kinds of organizational decisions yeah just a quick question on Scope when you say prompt you mean all of the last hour of what we've been talking about not just not just to yeah not not just the promp I think what I mean is like the um the whole um the whole like LM application yeah so so the the before I me guess the from the rag changing the prompt onward is what you mean yeah and the chains Etc MH uh so when you say um if you have a matching previous answer uh and you want to compare the previous answer to the one that's generated so how do you compare with another llm or is it's more um context or do you want to are you referring to uh comparisons yeah I I think like maybe to expand on this um you could um so the simplest thing you can do is you can just write a prompt that says how good is this answer effectively like rate this answer on a scale of 1 to five it's like the simplest possible way you could do this um the issue with that is that like the llm doesn't know what a good answer or a bad answer is necessarily like because it doesn't know what you care about in your task um so a lot of what people end up doing or a lot of what makes these effective is the same thing that makes prompting effective like you you need to make your evaluation prompt more specific to the thing things that you really care about um and one way to do that like one hack to do that um more easily is to give the model more information about what a good answer for this question looks like so if you provide a reference answer you still need to think about like what aspects of the reference answer you want it to keep because it's not it doesn't need to be exactly the same as the reference answer it could be worded differently um and depending on your task you might care about different aspects of it right so like um maybe you know if the question is like it's really important for it to get the facts right like then maybe you you'd tell you'd say the model you'd tell the model like hey make sure that the the answer here contains the exact same set of facts um as the reference answer but it's fine if it's worded in a different way or maybe what you really want what you really care about about the reference answer is the tone like um if it's a customer support like make sure that the the the answer matches the tone of the reference answer um and so no matter what you need to think about like what aspects of the outputs do you do you need to have in order to have good performance um and then the reference answer can give you like additional information to point the model to to help it understand what you mean by that so in the current state of art um do people automate this or people are manually intervening uh to do such validations so people um who are doing this today are most uh manually writing prompts to do well okay so most people are not doing any automated evaluation most people are doing manual pure manual evaluation um some people are doing automated evaluation where they have models evaluate other models um most of the people who do that are writing their own prompts to do it um there's a growing subset of people who are like uh you know um using tools to do this um like at Gantry we're building tools to help companies evaluate performance of models um um performance of LM applications and so um for certain use cases like for retrieval augment generation we have evaluators that work extremely well out of the box where you you know might not need to do too much of your own customization to um to like make a model that works well for this use case um for other use cases um that are like further away from things that are standard people still need to kind of Define for themselves what good performance looks like for the manual evaluation um I'm trying to do I'm trying to ask the model to provide its own reasoning uh when asks a question uh when answers a question and also provides the quote with citation is it um how trustworthy is that information for example if you're just focusing instead of looking at the answer but that's the actual reasoning and like what quote what reference it's referencing yeah I mean I think it's like um that's probably a pretty reasonable evaluation to make work pretty consistently um because like evaluation is just prompting it's you know it's kind of it's um it's basically like uh uh self it's it's like self-criticism except it's applied after the fact um so anything that you can do easily with a prompt you can do easily with an evaluation and things that are difficult to do with props are also difficult to do with for with evaluations hi I had a question about uh some terminology when you say evaluation um I've heard people use it interchangeably with experimentation and personally I find it pretty confusing and I try to sort it out and I couldn't really get a straight answer just from doing some research of like sort of which term is uh the best for whatever you're trying to describe so I'm wondering do you use it interchangeably or do you think there's uh distinction between the two when you think of evaluation versus experimentation I think it's different um I think um evalu I use evaluation and testing pretty interchangeably um and then experimentation to me means something different uh if you're talking to someone the machine learning World experimentation means trying a bunch of different architectures and prompts and stuff and seeing which one works and so experimentation relies on evaluation but it's a different like an eval you can evaluate a single prompt but experimentation means many um and if you if you're talking to someone in the product engineering World experimentation means like uh running a different variant in production on the subset of users and seeing which one the users uh prefer okay cool makes sense I also have another sort of related question um do you think it's useful at some point or at all to try to incorporate some notion of statistical significance when doing evaluations like if you're let's say if you have 30 question or 30 cases right and one variant gets 15 of them right the other one gets 16 of them right like do you think it's useful to yeah basically make a little bit more robust than just looking at the number and comparing the the two number numbers yeah I do but I don't see people do it very much um for two reasons one like I think mostly the phase that most people are at with these things are like still pretty early and so you're just trying a bunch of stuff um and so you know if there's not a statistically significant Improvement then um you might as well pick either one um it's not like there's a high switching cost early on in a project um and then the second reason is because it's expensive so in order to incorporate sta statistical significance I think for LMS what you you want to do is you'd want to incorporate the non-determinism of the L llms you want to generate multiple outputs for each um possible input and uh just most people aren't willing like Mo a lot of people have trouble justifying the cost of doing evaluation to begin with um and so you know in order to run like the evaluation like 10 times effectively to get the meaningful stats Sig numbers just most people don't want to do that thank you cool all right um let's talk a little bit about deployment um actually I'm going to kind of gloss over this a little bit uh so if you're using apis then this is not very hard like you just call the apis from your front end basically it if there's a lot of other logic like if you have complicated prompt Construction uh complicated chains things like that then you might want to isolate the logic as a service in your back end um or potentially even multiple Services um deploying open source LMS is a whole other thing like deploying open source LMS is where this gets complicated and it's worth learning about if you get on this path um but it's a bit beyond our scope for today uh some references down at the bottom below if you want to learn more about this and the other thing to know here is there's tons of Frameworks that are emerging um to do this uh that different trade-offs between ease of use um throughput latency uh things like that I would say the probably the two most credible are Ray any scale and M um Ray is what we use at Gantry um a lot of people use Mosaic uh hugging face and startups like base 10 a little bit more oriented around ease of use rather than performance but also good options and there's just a lot here to explore so uh check out the resources below do your own research if you want to um if you need to deploy open source LMS um next topic is okay now you've gotten this thing into production how do you improve like how do you make things better um so like one way you can do this is through some of the techniques we talked about impr prompting and training like you can do self-critique you can ask the LM to assess its own answer and if you have a high budget for cost and latency you can do that before you return the answer to the user um there are libraries like there's one called guard rails AI which is relatively popular that help you do this in a more systematic way um other things that you can do are um again a lot of the techniques we talked about in The Prompt engineering lecture you can sample many times and choose the best option um different ensembling techniques um and so all those techniques are relevant in production if you can pay the if you are willing to pay the cost and latency um to do so so once you've deployed um next thing that you're going to need to do is you're going to need to tell you need to measure whether this thing continues to work so what signals should you be monitoring um in addition to monitoring system performance metrics if you're especially if you're self-hosting you're also going to want to monit monitor model performance metrics um the most important one to measure is outcomes and and user feedback uh so if you have you know if you can tell how your users are using the output of this then that's great um even things like thumbs up thumbs down feedback are useful if you don't have outcomes or if you want like more detailed signal than just outcomes then model performance metrics are also really relevant to measure um so these can be these automatic llm performance metrics um or they could be more traditional ml metrics if you can use those for your task if you can't measure any of those things then um you can uh you can look at things like proxy you can look at proxy metrics like um you know uh did the result have the same subject as the question things like that um and then you can also measure the types of things that typically fail in production um which tend to be things like uh like you know um uh the model refuse to answer the question or um the model like uh you know said something offensive like things that you like common failure modes that you can write tests for you can monitor those things as well I think like one of the most important ones to get right if you're building a product around this is to gather feedback from users um and so what is good feedback good feedback needs to be easy for your users to give you because users don't really care that much about how much you improve your llm um so it needs to be easy for them um but it also needs to be high signal for you right so uh the best kind of signal you can get from user feedback is if that signal is part of the user's workflow so if the user um has to do something with the output of your model in your products where you can instrument that and see if they actually did the thing like uh did they copy this and paste it somewhere else um or did they edit it before they did that like those types of signals um the uh like one common pattern for this that I think is really powerful is the accept changes pattern so like in co-pilot if you've ever used it it'll make a suggestion for you and then if you click if you you know hit enter or whatever it accepts that change and so that's a really meaningful signal because that means it's useful for the user the the probably most common pattern is the thumbs up thumbs down pattern so it's just you know if you use chat GPT or whatever you can give a thumbs up or a thumbs down for the model's response um I think like otally I've heard from most people that barely any users will do that um it's a really really sparse signal and it's not very informative when you do get it because you don't really know why or um whether the user is actually uh representative of other people's preferences so um sometimes people will allow the option for users to provide longer form feedback once they've provided the low friction feedback and that's sometimes useful for determining whether this is feedback you should incorporate or not um we talked about this this question of like measuring what actually goes wrong with llm so in practice I think the most common things that I see honestly the most common is UI stuff which kind of uh uh tough to tough pill to swallow as people who are developing AI applications but um I've heard from folks that have deployed like some of these llm applications to massive number of users that people will click their thumbs down button and give them feedback and the most common type of feedback they get is hey I don't like the UI that you built for this thing um rather than the models outputs uh and latency is like an especially important problem here like users hate it when things take a long time um so you'll get feedback on that another really common one is incorrect answers or hallucinations so if the model basically makes some stuff up that's a really big issue with these systems today um but it can also be long-winded answers like if uh of you know for some use cases users want something that's really concise um and models uh some models tend to ramble that's a common issue um you know the the kind of negative side of reinforcement learning from Human feedback is that like the positive side is that it's helped models constrain their behaviors to things that are more acceptable for people the negative side is that models um I'm sure you've all seen this tend to dodge questions like um they'll say uh having a lot of laptop issues today um they'll they'll say like I'm sorry as a large language model trained by open AI I cannot answer that question even on things that are pretty innocuous and so that that tends to really rub users the wrong way um but then there's other things that you might want to watch out for like prompt injection attacks not very common but maybe something that you care about not leaking your prompt to your users um or toxicity profanity things like that so that's kind of like some of the signals that you might want to monitor for after you've deployed um and then lastly you know once you've deployed you're going to start to gather this data from your end users and this data is invaluable because it tells you what are the tasks that your users actually want to do not the ones just that you imagined that they were going to do um and so this is the the key point where you can start to like really improve the performance of your system by incorporating this data effectively so how can you use user feedback um you can use your user feedback to make the prompt better or you can use it to find to in the model um using user feedback to make the prompt better is much faster and much easier and is like most of what you should do so the way you do this is You' find themes and user feedback that are not addressed by the model so if users are giving you a lot of thumbs down um or a lot of written feedback you can look at like are there any patterns in where this feedback is coming and you can just do this manually just look at these data points um a lot of people I know who are building llm applications will literally look at every single negative feedback they get every day even if it's hundreds or thousands of examples um and then they'll build mental patterns for where this thing is working and where it's not um you could also Imagine doing this in a more automated way with llms though like you could ask an llm to tell you what are the themes and feedback that I got and then you can adjust the prompts to a account for these themes like if the model tends to fail in a particular way you can do some prompt engineering um you can add additional context to make the model better at answering those types of questions where you're seeing a lot of failures this also feels like a place to me that you could probably automate um if you trust your user feedback uh an llm can probably do a pretty good job of um understanding the themes in that feedback and then making changes to the prompt to adjust those themes but um the kind of like heavyweight option you have for improving your llm is to tune it um so there's a couple of different types of fine tuning that you might have heard of the first is supervised fine tuning and this is mostly useful if you want to adapt the model to a specific task and it's just really not reliable enough through prompting or in context learning and then like I think if you accumulate enough data like if you have hundreds of thousands of data points of like input output pairs then you can start to consider this as an option for things other than just um you know like improving on something that's not working the other like main reason I see people uh doing fine tuning is to reduce cost uh so if you have a relatively constrained task even if gp4 can do it it's probably not the cheapest way to do it and so once you've collected enough examples then you can start playing around with fine tuning as a as a cost reduction mechanism but generally speaking like I think this is more of an optimization than it is uh something that's core to building the application second type of fine-tuning that you might see people do is uh fine-tuning from Human feedback which the most like kind of well-known example of this is reinforcement learning from Human feedback and I would say this is like uncommon um there's only a handful of companies that do this on their own it's uh much more complex and expensive to do than supervised fine tuning um just purely from a technical perspective it's just a difficult thing to do so the way that supervised fine tuning works is um you uh basically can um update your uh like you can you have different ways of um basically fine-tuning like some of the weights of your model um or all of the weights of your model so the simplest thing you could do is you could like update um all of the uh like all of the layers of your model you can just finding the whole thing but what's more common to do is to um keep most of the weights of your model Frozen and only update like a small fraction of the weights uh because it's cheaper to do and it's um it's faster and it can avoid overfitting um one like kind of extreme place that this has gotten taken to is a tech set of techniques called parameter efficient fine tuning um and so this is what if you're going to start fine tuning on your own this is what I would look into to start um the most like I think probably best known technique in this category is called Laura uh low rank approximation and um the uh the reason why this works so well is because um you again like you're just you're not updating all the 10 billion parameters that your model has you're only updating a tiny tiny fraction of them and so it means you don't need as much data um and you don't need to like um you don't need as heavy weight of infrastructure to make it actually work so quick intuition about how lowo Works um you have your pre-trained weights and remember your pre-trained weights are like the set of matrices that you're doing like a bunch of linear algebra um billions and billions of times in order to like transform your input to your output in Laura what you do is you rather than updating every single one of the weights in that Matrix instead you learn um an adjustment to those weights that's parameterized by um a low rank approximation of the weight Matrix um so instead of taking this full like n bym Matrix that has n * n n * m parameters instead you take a smaller number of parameters um and you transform them in a deterministic way uh to be the same size as the output um and so you're learning a tiny subset of parameters that are applied to the output in a deterministic way and yeah oh this is a cool I forgot about this yeah mhm yeah it's um it's Bas it's basically basic mathematics um yeah and and it works surprisingly well um I would say the limitation with Laura is that um and the limitation of fine tuning on a small amount of data in general is that like mental model I have is um like the um the meat of what the model can do is determined by the largest data set that it's trained on and so like the core capabilities of the model mostly come from the initial pre-training on the whole internet and then the additional finetuning that is done on top of that is like a way of refining the behavior of the model like constraining the behavior of the model from that um just this is just an Intuition or like a mental model and so the supervised fine tuning constrains the model to like follow instructions um in a a way that we would expect and in a similar way if you're if you're fine-tuning especially on a very small number of examples um usually the model will learn how to replicate the surface level patterns in that small number of examples not the deeper like um uh reasoning or harder to capture things as part of that um and so there's a paper from Berkeley maybe six months ago that showed that um uh reinfor like models fine-tuned with reinforcement learning from Human feedback perform much better on human evaluation benchmarks than models that don't have rhf but they actually are um some in some cases less factually accurate than the models that humans rate less highly um and so the phenomenon that's occurring is that like uh rhf makes models sound more convincing um but it doesn't actually make them more correct uh but it turns out that most of the time when people are doing evaluations that's what they care about is uh how convincing the model sound um so that's the thing to be careful about with fine-tuning is that like if you are fine-tuning a small amount of data you can you can get surface level things like structure patterns things like that pretty easily um but you're not going to probably teach the model new capabilities um unless you have a massive amount of data so reinforcement learning from Human feedback um again it's a more complicated way to do tuning um the way this works is you collect demonstration data um then that demonstration data um is uh well I think the critical part starts as step two here so you collect comparison data where humans say I prefer output a or I prefer output B like this answer is better to me than this answer um and then what you use that data to do is not to train the model directly but to train an auxiliary model that's used to predict human preferences um that auxiliary model can be used to predict prct human preferences on data um that was not seen by the model um when that that was not seen when that auxiliary model was trained um and so you use that model as a signal um to train the base model using um the class of like set of technique called reinforcement learning um and so the reason why rhf is so difficult to do and why so few companies do it is because one it's just a lot more complicated to train two models and two because reinforcement learning in general is like incredibly finicky um and it's difficult as difficult it is as it is to train models and do that reliably um with supervised learning it's like much harder to do it with RL um so recommendations on fine-tuning um You probably don't need to do fine tuning honestly um at least not right now like at least not in the initial phases of your project uh I think people reach for this too soon in their projects because I don't know why I guess it sounds more cool or interesting or you feel like you're doing something that's more technically sound or um or where you're building more of a core technical advant Advantage um but I think the reality is that like most people don't put enough effort into their Baseline which is prompt engineering chaining and information retrieval um those things are way easier to do they're way more reliable and they're going to be way cheaper for your company um good reasons to finetune I think one of the best ones is cost reduction so if you do have a cost issue you can often mimic gp4 performance with a much smaller and cheaper model but um again cost optimization is an optimization so don't do this until you can solve the task uh speed Improvement similar thing um and then I think like kind of non-optimization use cases would be like if you're having trouble getting consistent structure to the output of your model um or if you have a ton of data and there there's just nothing you can do that will make the retrieval system work um those are those are good use cases for fine tuning as well uh Laura like and parameter ficient fine tuning is what I would reach for first it's uh it's cheaper it's more reliable it's more accessible um but it is ultimately also a little bit limited um and then I I probably wouldn't touch rhf today um unless you are like a big company that's going to be putting a lot of effort into into trying to make this work so I guess conclusion I want to just kind of wrap up this section by talking about like how to pull all these pieces together into um hopefully a more sane process for building llm powered applications um a process that mimics test driven development in software engineering so we start with a core workflow around prompt or chain development um we start with a BAS LM we choose that llm we start with gp4 and then maybe we we uh adjust it later then we develop a prompter a chain we test that prompter chain on our current evaluation data set and then once it we're like happy with how it performs on that that evaluation or those set of tests then we deploy it um and once we deploy it we start to collect feedback from our end users and those end users depend on what stage we are in in the project when I start on a project I am the end user like I'm the one I'm collecting feedback from and then I'll move on and I'll collect that feedback from my friends or my close colleagues who I'm working on this project with um then maybe I'll roll it out more broadly in our organization um then I'll start to roll it out to Alpha users beta users and in each of those phases you'll do the same process of collecting user feedback um that Fe user feedback results in interaction data like what did the users what was the result of the users using this application what did they give us what feedback did they give us um and we capture that interaction data and start to categorize into themes um those themes are turned into additional test data um that we can use to build a richer and richer test set that captures more and more of the behaviors of how we want this system to perform in the real world um and it also is captured as an improvement to our prompt or our chain um and then finally like once I've you know um exhausted the utility I can get by just making improvements to my prompter or my chain or my retrieval system then um I can run a similar kind of outer loop to this workflow that involves fine-tuning where I take the same interaction data I'm getting from my end users and I turn that not into test data but actually into training data um which I can use to fine-tune the model and eventually update the base llm and then this process repeats um at the different phases of the workflow um starting with yourself or your development team as the users um then rolling out to um the rest of your team as users and then finally with your customers in the loop as users as well so I think if you apply all the lessons from this um discussion then we can hopefully move from something that feels like kind of very hacky and and uh ad hoc to something that starts to feel like more of a systematic um process um that is grounded in in uh in like measurement and data collection