Test-driven development for large language model applications.

Josh Tobin: LLMOps: Test-Driven Development for Large Language Model Applications

Recording: Josh Tobin: LLMOps: Test-Driven Development for Large Language Model Applications

foreign [Music] which backs how we build products with llms and is also built into our products which is an analog to test driven development for large language models so what I'm going to do is I'm going to kind of talk you through what that actually means and kind of how could you actually even test large language models and then I'll show you a demo of how you can do it in gantry so a little bit about us off the bat uh just introduce myself my co-founder Vicki was a engineer at open AI in the early days built basically all their infrastructure we're backed by some awesome investors um but more to the point so I want to kind of ground Us in what it feels like to build applications with this crazy technology today in June 2023 so how many have built something with an ll1 before like an app or a demo or some sort of some sort of product yeah so vast majority of folks right and why is that right why is everyone experimenting with this stuff well I think one reason is that the gap between your idea and having a prototype has never been shorter so um in 2023 if I have an idea about something I want to build like let's say I want to ask a question of my notion database as a slackbot well all you have to do is just Google that and there's probably going to be a lang chain template for it right so I can just download this code and get started in like 15 minutes later I have a demo of asking questions of my notion database it's amazing right something that you couldn't even imagine doing two weeks before that and so I think a lot of us when we try this and get the first demo done really quickly get kind of the wrong idea about what the rest of the process is going to be to go from that initial demo to Something Real like an actual product that we can that our users can rely on so I think in our minds the way this works is you know we we built this whole thing in 15 minutes and so the rest of this is going to be easy too right we're going to um we're gonna get our link chain demo and then we're gonna do you know some stuff and then eventually in like I don't know a week or two we're gonna have some really powerful app maybe it's even AGI right um but as you all have like if who have played with this are probably familiar with it's this space in the middle which is you know what do you do after you have that demo which is really where all the difficulty is in building applications with AI today and the reason for that is that you know llms are these incredibly powerful Primitives it's this almost like this alien technology that's landed on Earth and it's unlocked all of these things that people you know were really difficult to build before even six months ago but um there's a lot of challenges in building with these models as well so you know I heard about this term prompt engineering and so you know I heard prompts engineering and I kind of thought that maybe this is going to be something that felt like engineering but I actually think you know calling it engineering is really you know at best like let's say generous uh it's more like prompt hacking and then once I built the that like prototype I you know I played around with it a little bit but I had no way to really know if it was actually working it was actually solving the problem that I wanted to solve and I kind of came to the conclusion like is it even possible to test these things right how do you know if this application is going to produce the right output so the upshot is building demos with llms and this has always been true in ml but it's especially true with llms is a long way from building reliable production systems so if our goal is to build an actual application that's powered by this technology we need something more than just um you know the the tools to build that first hacky demo um so what I want to talk about today is what we think is potentially a better way a philosophy that you can use a methodology that you can use to build to go from that first demo to something that actually works um and what this looks like is kind of a lot like the test driven development process and traditional software so imagine that you have built that initial Lang chain demo or llm demo you've uh and you're ready to test it so you give it to your friends um your friends start to give you feedback on how this thing is actually doing like hey did this answer the question correctly um is it uh is it spewing nonsense or is it saying things that are actually sensible then as a developer you take that feedback you use that feedback to understand what are the opportunities to make this application better um so you iterate on the application and then once you have a version that feels like it's better it seems like it's better to you what we want to do is we want to systematically evaluate whether that's actually true whether we've really made progress on this application or not um and then if you have then you bring your team in to approve those changes you deploy the new version your users interact with that version and you start this sort of virtuous cycle all over again of gathering feedback generating new test cases from that feedback improving your model and then validating that the improvements actually worked so this is kind of the the subject of what we're going to talk about today and so the first thing I want to do the next thing I want to do is talk about kind of I think the the sort of hairiest question to all this which is how do you actually measure like how do you evaluate models how do you measure whether the new version of your prompt or the new model is actually better than the one that you had before um so why is this difficult or why does this even actually matter right so um if you've used olms before you know you can you can get these like amazing first results right where you ask it something and it does something really cool that you didn't think it could do but then as you play around with it more you realize like hey these things make a ton of mistakes and so the implication of that is just because the prompt the new prompt that you wrote or the new model that you picked performs better on a few examples um the the few that you Cherry Picked to test it out does not mean that it's better in general you can make a prompt better you can make a model better on the you know handful of examples that you care most about and it can get worse in some other way in a way that's really really hard to predict and really hard to understand so if people rely on your model if you're building an actual product with this model not just a demo then they're trusting you to maintain your performance on their task the thing that they care about um not the thing necessarily that you as the developer care about so that's why this matters um so in order to understand like how to actually do this I want to kind of ground Us in uh how we used to test models you know back in the day when um when deep learning was like sort of the uh the technology that everyone was using to build state-of-the-art AI systems so the old way that we thought about testing machine learning models and how many of like you know done like trained models and done sort of more traditional ml stuff before okay very small fraction um interesting uh so the way that this worked is you had um you had some data that you trained your model on and that data came from some like abstract distribution of data you could think of this as like all of the possible points that might have come from this uh this you know this data set that you train your model on and uh within this this training distribution you have your actual training set so this is the data you fed into train your model um and then you had your evaluation set so you held out some data from this training distribution to use to measure the model's generalization Beyond just those data points that you trained on and so um you then would take your your um your metric like let's say your accuracy or whatever metric you care about for your use case you'd measure that metric on both your training set as well as this held out data that you didn't train your model on and the reason that we do that is because the difference between those two numbers the difference between the performance of your model on the data it was actually trained on and the performance of your model on data that looks like what it was trained on is a measure of how much your model is overfitting to those individual training points then there's often a gap between the data that your model is trained on and the data you actually need it to work well on in production so you can then look at um you know two similar data sets from your production distribution you can look at your test set um and then you can look at your actual production data from you know your end users interacting with the system and you compute your accuracy again on those two uh on those two sort of data sets and the difference between those numbers also tells you something about how the model is doing so the difference between performance on your evaluation set so this is held out data from your training distribution in your test set which is data from you know what the model is actually going to look at in production that's a measure of how much domain shift there is so how different is the data your model was trained on from the data it's ultimately going to be evaluated on in the real world and then lastly the difference between performance on that sort of held out data set from your production distribution and your you know the data you're currently seeing from your end users live in prod that difference is a measure of drift so how different is your performance now than when your model was trained so this is kind of how we think about evaluation in the world of traditional ml next I want to talk about why this doesn't actually work for llm so like why aren't we just talking about this same framework for evaluating Outlets first um okay you're um most of you are I'm guessing using a model from openai or anthropic or cohere or maybe you're using an open source model but even if you're using an open source model I'm willing to bet that very few of you have actually like downloaded the training set that that model was was trained on right so you don't actually have access to the training distribution these models are trained on the whole internet but you haven't actually looked at the data that's in that trading distribution then um one implication of that is that the production distribution so the data that comes from your end users for the tasks that they want to solve always looks different from the data your model was trained on right your your um if you're building a uh an app to ask questions from your notion database there's no way that openai trained their model on mostly data that looks like that right the data is much more diverse so in traditional ml uh you know so the implication of this is that like uh it doesn't really make sense to think of evaluating large language models as you know taking a big sample of data from a data set that we know about and Computing metrics on them but in addition to that the metrics themselves are really different so in traditional ml you might have a metric like you might be trying to classify whether an image is a prediction of a cat or a dog and the way you do that is you look at your predictions you know which ones are cats and which ones are dogs and then you look at your labels so which ones are actually cats and which ones are actually dogs and then you compare those things and you compute a metric like accuracy that says how often was the model right about this prediction in generative ml like in a lot of the things that you all are doing with large language models um your outputs like your predictions might be a sentence so the sentence might say this is an image of a tabby cat if your label says photograph of cat how do you how do you tell if that's correct or not right is that a good prediction or is that a bad prediction maybe it's a good prediction right because it's actually even more specific than the label but maybe it's a bad prediction because you don't want this model to to return the the specific type of cat that this is you actually just want to say that it's a cat and so the metrics that we use the numbers that we use to compute to tell whether models are working in generative AI are ambiguous so it's hard to quantify whether performance is working well or not and the last reason why this framework doesn't work very well for llms is that um oftentimes when we're building you know in traditional ml we're usually building a model to solve one use case like I'm building a model to classify images of cats versus images of dogs in llms you know the the general assumption is that you're Building Technology that's pretty general purpose you want this same model to be able to answer questions not just about um not just from your notion database but also from your slack and from these other data sources and not just questions about like payroll but also questions about uh sales and marketing and accounting and so um the diversity of data is is much more diverse in many generative ml applications than it is in traditional ml and so the problem with that is that let's say that you have a model where you have questions about startups dogs food and physics and your model is like really really good at answering questions about startups and it's pretty good at answering questions about dogs and food but it's terrible about answering questions about physics is this a good model or is this a bad model how many think this is a good model or couple I anyone think this is a bad model okay that people don't have an opinion um the question is obviously it depends right depends what you need this model to be able to do um and so it's really really hard to summarize the performance of a language model with a single number because the types of things that we're asking to do are very diverse um so just to summarize why doesn't traditional evaluation work well for llms well they're trading on the internet and that means that there's always drift your evaluation data is always different than your training data and so Drift actually doesn't really matter um it's qualitative so if you're generating text as an output it's difficult to come up with a number that measures success and there's a diversity of behaviors that you often want from these systems and so looking at aggregate metrics is just not that useful um so I think the way you should think about testing llms is there's kind of two hard questions that you need to answer you need to answer the question what data are we going to evaluate this model on and then you need to answer the question what metric are we going to compute on that data so let's talk about those um first question is what data do you evaluate the model on um I think there's uh and the summary is that there's like kind of four steps here so first thing that you want to do is you want to start incrementally um then you want to get get some help from AI add more data as you continue to roll this thing out and then uh I have kind of like an idea that I want to throw at you of something that I think might help make this more principled but it's a little bit ill-formed so we're going to start incrementally um and what does this mean so this means that you know as you're starting to work on your model you might be like writing your first couple of prompts it's okay to start by evaluating your model in a really ad hoc way so let's say that I have a you know prompt template that's like write a story about a given subject um I might just you know play around with this and you know try out a few subjects that I think might be interesting for it to write a short story about and um and you know just see what the model does in response to those to those inputs as I find inputs that are interesting in some way so where the model does something wrong or where the model you know does something really different than I saw before then I'll start to organize those examples into this like small data set of examples um and so now rather than having to type in dogs or LinkedIn or hats every single time I make a change to my prompt now I have this like little data set and I'll just run my model against all those examples and I'll just see the outputs side by side in parallel um quick aside what does it mean for an interest example to be interesting I think usually either that means it's hard so it like caused your model to do something bad in the past or it's really different like um hey I realize that I actually need this model to work well on you know LinkedIn posts as well as uh as well as marketing emails and those are very different or maybe not um second piece of advice here is user LM to help so um oftentimes like we don't actually have like this data set of examples that we can start out with um and so the nice thing about using llms is that they're really good at generating text and so maybe not so surprisingly they can help you generate test cases here's an example of a prompt from this um Auto evaluator Library which uh takes like um generates some question answer pairs for a QA environment and I'll show you an example of doing this in Gantry as well in just a second um so we've started like with this really simple data set we've used our LM to help us bootstrap more examples and then you know I think the most critical part of this is to build your data set incrementally so as you roll out to more users then add more data to your evaluation set so what data should you add well you should look at things that your users dislike maybe things that if you have annotators that they disagree with or they don't like um you can look for data where another model disagrees with the model that you have in production you can look at outliers relative to your current evaluation set or um you know underrepresented topics in your data set so you know these kind of break down to this category of data that's hard for your model so things that people don't like um your model didn't do a good job you know according to the people that are looking at this thing or data that's really different than what you currently have in your evaluation set so it might be helpful to see how future models perform on that okay um one thing you might be feeling looking at this is doesn't this feel a little arbitrary right like doesn't it feel kind of weird that we're just picking examples that we think are interesting adding them to an evaluation set isn't there a more scientific way to do this um so is there any way that we can quantify how good our evaluation set is so that we can progress towards having a better and better evaluation set I think there is um I'm not going to cover the details of how we do this in this talk but if you're interested come talk to me after there's some math involved but the high level idea is there's we can formulate a notion that uh that looks something like the traditional test coverage metric but for AI and the way that you would want this to work in in principle is you would want to have a metric where if your production data has a lot of data points that are really really far away from anything in your test set then that should be like a low test coverage data set right because you have a lot of production data points that don't look like anything in your test data set on the other hand you know even if you don't have that many test data points but there's nothing in your production distribution that is really that far away from any of them or from from at least one of them then you would want that to be a high test coverage data set so I won't get into the math here but this is some intuition about like how we might be able to quantify whether your evaluation sets are good or not um so yeah um maybe last thing I want to talk about on evaluation metrics is you know how do you pick or on evaluation is like how do you pick the metrics that you want to use um so like here's a flow chart that you can walk through um the first question you might ask yourself is is there a right answer to this problem right like if you're predicting whether something is a you know is a cat or a dog um there might be a right answer and if there is you can just use the same evaluation metrics to use in traditional ml if not then the next question you can ask yourself is do you have a reference answer do you have an example of what a human or like a human that you trust would say in response to this and um if there is then you can formulate metrics that involve matching the generated example from the llm to the reference example from that trusted human um if you don't have access to that or if it doesn't make sense for your problem then you can look at comparing the new output of the model to a previous answer generated by an older version of a model maybe one that you have reason to believe is at least okay and so you can formulate metrics that answer the question of like which one of these things is better and then finally if you don't have that so if you um if you don't have like which means that it's it's going to be really difficult for you to um to compute an automated metric on top of your system then you can uh you can look at previous examples of humans that provided feedback on older examples from the model and you can compute metrics based on like whether the feedback is actually incorporated into the model or not so one point I want to make here is this uh traditional eval metric that's like that's like just some math that you can do like accuracy you can just can you know compute a number using uh you know uh python scikit-learn whatever you want the rest of these are all metrics that involve using a large language model to evaluate another large language model um so the way that all the rest of this tree works is you ask an llm like you know which one of these answers is better or uh you know did this answer incorporate the feedback from the previous answer um so you know really surprising fact that llms are actually reasonably reliable evaluators of the performance of other albums and this is especially true if you know you're using a smaller llm in production and you're willing to you know pay the big money for gpt4 to evaluate it yeah offline um okay so I'm going to skip through this and I'm going to come back to um like let's try to make this a little bit more concrete so let's talk a little bit about Gantry um so Gantry is basically we're building a product that helps you run this process this test driven development process um on your uh like for your own applications so I will go through the details of this slide but basically high level this whole workflow that we talked about like finding feedback on your model um using that to iterate on the prompts evaluating it systematically uh proving the changes and promoting that to production we have features that sort of line up with each of those I'll show you what those look like in our UI um but it's also worth noting that like there's sort of ways to do each of these things that I'm going to show you in code as well um and lastly like a couple principles that we've had when building this so uh we're really big Believers in the value of human feedback like entry like it's uh you know at the end of the day um ultimately whether models are working it's a question that you're asking like are they solving the problem for your end users and so we really believe in incorporating feedback from end users um we're also really focused on collaboration so we found that a lot of these llm power products are getting built not by Engineers but by interdisciplinary teams of Engineers product managers other stakeholders so we're trying to help those whole teams come together to build this stuff and then lastly and like maybe the one that's most fun is um you know we we looked around at a lot of the tools in the LM op space and you know it's kind of funny like like a lot of you are building LM power product features You're Building these like amazing you know magical product features for your end users that are powered by this brand new technology to solve problems that they couldn't solve before um but the tool is that you're using may not actually incorporate that same sort of um like modern AI first approach and so we are like looking for as many ways as we can to incorporate AI into the process of building AI um so let me show you what that looks like and Carol do you mind like yeah cool um so you can write this prompt in code and upload it but it's like a little easier to do in the UI so um maybe like an initial prompt I might start with is like again we're trying to correct grammar of user sentence so let's just ask the model correct the grammar and we're gonna actually have to pass in the user's input and so we'll do that as a prompt template listen yeah um okay yeah um and so let's let's just test this out on a couple of examples to see if this does something reasonable so um let's say you know we went to the store and then um I don't know we go to the store that's I'm getting really good at like writing bad grammar sentences from doing this demo a bunch of times um and yeah it seems like it's doing something reasonable right but how do we know that this is actually performing well beyond just those two examples that I just kind of made up um so what we can do is we can um uh we can take a look at this version that we just built and we can evaluate it so again two pieces of an evaluation where do we get the data from and then how do we actually can like Define metrics on that data so uh our Approach at Gantry for both of those is to use llms to help us um so we're going to generate a data set that we can use to to evaluate this and so yeah grammatically incorrect sentences is kind of what I want here so I'll hit generate data and this is you know hitting openai to generate some example data points that uh you know that we can use to evaluate whether this model is doing the right thing and yeah yesterday they walked to the store a bunch of examples that don't have great grammar so we'll name this our eval data set so now we have some data to evaluate on next thing that we need to do is we need to um we need to actually Define the metrics that we want to evaluate on this data and so in this case um you know we like really what we want to know is whether the outputs are grammatically correct or not so that's our evaluation criteria that's going to be passed to another llm to do the evaluation so this is now running um it'll take a few seconds as we you know wait for open AI latency but um the what's happening in the background is it's generating for each of those inputs it's generating an output from our model and it's uh scoring that according to another llm and maybe not surprisingly this kind of simple prompt for these simple inputs that we developed it's performing pretty well like the model thinks that 100 of these inputs are five out of five right and if we kind of eyeball it it looks like that seems pretty reasonable like we don't like broccoli I like I don't like broccoli that's a pretty reasonable correction um if we want to kind of know why the model thinks that's the answer we can just kind of zoom in and inspect like its chain of thought that it came up with for answering that question so um real world we probably would want to like iterate more on this prompt so make sure that it works reasonably well before rolling it out but this is a demo so we're going to Yellow it into prod um so this is now in prod and what does that mean what that means is that um this is tagged uh this prompt is tagged and you can now pull this prompt um from the Gantry CMS to use in your applications um you can do this in a couple of different ways I'm going to show you kind of the easiest way to do it which is to do it uh just by calling our proxy um so what that looks like is I have like a really simple streamlit app here zoom in um and you know rather than calling like open AI completion endpoint instead we're just going to call a similar completion endpoint for Gantry and instead of passing in the full prompt as well as all the model parameters and all that kind of stuff instead we're just going to pass in that input that we wanted to send to that model and Gantry is going to pull the latest prompt version latest model parameters it's going to complete that prompt template send it to open AI get the response back uh and then log all the results back to Pantry so other ways to integrate this is the simplest one if we run this streamlit app this is really small but it's okay you don't need to see this part foreign and then we open this up then yeah this is like maybe what your end users would see right so we've got a text box so you can write sentences in um this sentence has good grammar um and then they can submit it get a response back um that seems pretty reasonable so they can give like positive feedback on this so rather than like doing hundreds of these I pre-sent a bunch of data to this application and when we get as a result of that is we get this like kind of nice dashboard that tells us like hey here's what's going on as your end users interact with this model so we see some high level statistics here like how much are they spending you know what latency are they experiencing and things like that but we also see uh yeah like what feedback are they giving um kind of interesting statistics like that but then we also see a bunch of derived properties of the inputs and outputs like what was the sentiment of the inputs the users are sending into the model uh what was the sentiment of the responses the model sent back how toxic are they um you know how fluid are the are the outputs and what language are they in right and so one of the really powerful things about Gantry is that we allow you to compute like all these different derived properties of the inputs and outputs to your to your model and our infrastructure so you can just like iterate on them really simply um just by writing python functions but these are the ones that we have right now and so one thing I notice here is that you know there's a bunch of inputs that are not in English so that's kind of weird so if I filter down on those what I see is yeah so some users have been submitting some text in French and you know for the most part like these I don't really speak French very well so I have no idea if it's getting the answer right on these um but there's some where it's like actually translating the French to English so that doesn't really seem like the behavior that we want right so let's uh grab these data points and you know coming back to like our test driven development philosophy um what do we want to do well we want to add these data points back to our data set so that when we evaluate the next version of our model we can test it and see whether it's improving its performance on these data points where it failed in production so all these to our eval data set and if we go take a look at this now it has like kind of the 10 original examples that we had as well as these you know four examples in French um Next Step here is let's see if we can fix this problem so I'm going to try to do some like live prompt engineering here to fix this issue with the model and um so I'll add another version and rather than just like correcting the grammar of the user's input I will tell the model make sure that your response is in the same language as the user's input now you know I'd probably if I had more time I would do a more careful job of prompt engineering than this but for now let's just run this and see if it does something reasonable um so it does not interesting well um yeah let's uh maybe we need to give an example of the uh live prompt engineering it's very uh very stressful uh let's see example like follow the example format this is a good prompt engineering trick for those of you who are learning prompt engineering um user inputs oh yeah bracket ah thank you thank you all right live prompt engineering okay yeah so now let's do something more reasonable right um so okay so now is this working um next thing we're going to do is we're going to take this new version that we just developed and we're going to evaluate it um and so now we have a richer data set to evaluate it on we have versions to compare to and we can add another criteria here that says like results should be this is like an awkward angle to type at there we go um and let's run this and see what happens so sometimes it takes a little bit longer to run what's happening in the background here is models taking these 14 inputs it's running both of these versions against those inputs it's taking the outputs it's scoring both of the outputs according to this evaluator model and then it's going to give us comparison to tell us whether this model actually fixed the problem or not um so while we wait for this to run are there any questions for evaluation we're using uh I believe 3.5 for this because it's you know four it takes a long time for live demos but four is probably what uh what I would use in production yeah at least my versions also behind the scissors like this running behind the scene Zone uh also dbd 3.5 like it's version one version three or this is a um I believe like we can go easily check that um so I believe what we did is yeah I just used text DaVinci uh just kind of as a default and because it also fails more which makes it easier to do this uh in a quick way um okay I'll take more questions in a second but meantime we've got our evaluation report and um interesting so there's one example where V3 was better in one example where V1 was better um so if we kind of look down at those um it uh it looks like actually yeah our The Prompt engineering that we did for version three here uh fixed the problem for one of the examples but it did not fix the problem for one of the other examples so I failed at my uh my prompt engineering tests but this can give you an idea of like how you can do this in a more iterative way where you make these changes to the prompt you run an evaluation and now I have feedback from this evaluation that tells me like hey I need to go back and keep working on that prompt because I ran it against all the examples I care about and it didn't really improve performance um and so that's kind of like the core workflow that we're trying to support and really like the way we think that um that folks should think about building applications on top of LMS um so yeah I'll pause there and um we'll always take questions I will also like flash a QR code here um we are going to give everyone here early access to this product um so we're kind of like doing a sort of quiet Alpha of the product right now um building a lot of exciting stuff on top of this um and yeah if you use this link then you should be able to just like sign up and try the product for free um and yeah our only assets just like give us a lot of feedback in the alpha you can drop in our Discord from the products and would love to hear what you think great um happy to take questions repeat them for the recording okay sounds good uh one and uh professionals with better models who is the human dear great question um it could be anyone whose opinion you care about the model so um and so who that is depends on the where you are in the development cycle if you're if it's day one and you're working on this model by yourself then you're the human you're the human that gets Steve the feedback on the model if it's like day five right and you have your your uh your prototype and you're ready to share with your team you're ready to share with your friends those are the people who you should be gathering feedback from um when you roll this out into production like to all of your end users those end users are usually the people giving you feedback and then in a really steady state like once the product is mature most teams end up building kind of a um a QA team to uh to like be the sort of professional evaluators of these models because it turns out it's very difficult to perform better than gpt4 on um on evaluating example outputs so oftentimes you need professionals yeah over here everything is the regular question for the recording but Mike isn't what's up lady right I can hear and just repeat from yeah look like the unit of evaluation was a prompt in the workplace showed up it's like improving the prompt re-engineering The Prompt what are other changes that Gantry supports either like at the model level or with fine-tuned that like would set up a new version that you'd want to remind all of your different yeah so it could be a different Model A different base model right you could pick a model from a different model provider you know a different uh you could see if if the trade-off of you know reducing cost by switching to a cheaper faster model is worth it um it could be uh changing the chain so we're gonna support chains um in a few weeks um and then for fine tuning um we do have some tools to help with that um but that's like that isn't as baked into this product experience yet um so fine tuning is more finicky and um I my general advice to folks is not really to worry about it until you really need to yeah so if you find auto evaluator excuse me Geographic Auto evaluated with all this I think one of those factors is the accuracy of the evaluation model uh the accuracy of the evaluation model can impact the accuracy of my test so how does country think that new amount because it's essentially when you have a wronged inside that's trying to measure the accuracy of my problem so how does that work yeah so the um the question is if you're going to rely on models for evaluation then those models themselves need to be accurate um how do we deal with that so uh two two points there one is um we we drink our own medicine so um we also are building sort of hooks to gather feedback from our end users on how the evaluations are working um and behind the scenes we're using that feedback to make our evaluation models better um and so that's really that's sort of the magic of the way that we're developing this product um one point that's kind of hidden there is that I don't really think that my um my recommendation is not to replace human evaluation with automated evaluation I think that's um that's misguided because ultimately um you know you the whole reason why we need evaluation is because models are unreliable and so at the end of the day you want someone who is on the hook for saying yes these results are good the way I think automated evaluation is really useful is for two things one is for faster iteration so if you're working on this by yourself and you want to just get quick sense for whether things are moving in the right direction automated evaluation is really key and then the second is to reduce the costs of manual evaluation so in the post you know gpt4 world to do better than gpd4 at evaluation you're not going to be able to just find random people on like mcherk they generally numbers I've heard are something like 85 or 90 percent of uh of triggers are worse than gpt4 at most labeling tasks right so let that sink in for a second but it's pretty crazy and so what that means is that if you really care about like quality assurance for your models you need to train people to do quality assurance for the specific tasks that you care about and you need to make sure that those people are better than the models that they're evaluating um and so I think uh the the implication of that is that quality assurance and human evaluation is becoming more and more expensive which makes sense when you think about it right because as the models get better the the sort of judgment that you need to apply to their outputs uh becomes more and more specialized and so um the role of automated evaluation Beyond just iteration speed is you know I think we should try to use automated evaluation to reduce the number of data points that humans need to look at and focus on the data points where uh where we need to spot check the model or maybe we're most uncertain about how the model is performing on this data point question here yeah so in your examples you've been using one maps to uh evaluate all observed performance I'm curious what's your recommendation when you don't have multiple metrics multiple objects do proposed to collapse to also Define foreign yeah I think it's better to split out multiple metrics um and so you can you know you can define a weighting across those metrics if you want to have one number to look at um but I think it's better you know you you really want to look at how model performs on each of those metrics and really even more granular than that like each of those metrics across each of the different sort of slices of data that you that are important to you and your problem here is the there are cases um where they need it in the same order for us to remove the generation with American machine uh maybe the the generation Park The Prompt has not been willing to fly and are there cases where like the same model can evaluate different terms or themselves uh and say that one is better yeah so um surprising fact but true facts uh gpt4 is pretty good at evaluating gpt4 props um so it's uh you know if you want some intuition about why that is um when you're doing like when you're generating tokens as output from a language model the model can only really think one step ahead um so this is why like Chain of Thought prompting and things like that work so well because you're giving the model a scratch Pad you're giving it like some tokens to use to think before it has forced to spit out the final answer so like another you know easy like free prompt engineering trick is um if you uh like if you're trying to get a structured output like always return this word then you should always tell them all to return that at the end not at the beginning because then it's that time to think before it returns that output um and so the reason why automated evaluation works are the intuition about why it works is kind of similar which is that um the model's thinking on the fly to generate the output but then when it's evaluating the output as the benefit of hindsight so it has the ability to look over the entire sequence incorporate that into its decision about what's the right thing to do or not the right thing to do so there is some evidence that models are like have some bias towards the output of um of their own evaluation uh uh the outputs of themselves so like one thing that might not work that well is if you wanted to use gpt4 to compare you know gpt4 and anthropic um you might get some bias there but there's there's ways around that that were that were working into this yeah you talk to you mentioned that there's no focus on fine tuning but uh I know I don't know if you're familiar with vikuna for example it was fine too based on gb4 and as a result it got much better results than the others with the small set of prompts so just wanted to hear your perspective yeah no I think um I think continuity is making a comeback I think um open source LMS I'm really excited about them uh and we will support them just not supported right now yeah um fine-tuning is kind of a is itself a difficult problem um and so you know we that's just stopping our Focus up to this point any other questions cool yes so well that's uh interesting interesting question so do you think open source or private LMS will win um both yeah I I think this is I mean this is going to be such a big character like all approaches are going to succeed um uh proprietary ones are always going to be you know highest quality and easiest to use um open source has obvious advantages you can deploy it on-prem um you can customize it a lot more easily both approaches are going to succeed yeah it's kind of like you know Mac versus Linux or something right which one won well people make a lot more money on Mac than they do on Linux so maybe Mac one but Linux is installed on a lot more devices as far as I know so maybe Linux one great well yeah thank you all so much for coming [Applause] thank you