Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 2

Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 2

Recording: Workshop "Building production-ready LLM-powered applications" by Josh Tobin. Part 2

um okay so we we talked about like kind of highle um what are llms how do these things even work and we talked about kind of the anatomy the structure of an llm application which includes some instructions um some outside data and those instructions and outside data are combined to call um to call an underlying machine learning model to produce an output for uh for the user and we looked at like kind of a really High Lev example of how you might build an application like that in the future without needing to understand any of the details of how those different pieces work um using something like the open AI assistance API um and we also talked about how it doesn't quite work yet so given that it doesn't quite work yet it's still relevant for us to understand um a little bit of how it might happen under the hood and so for the rest of today we're going to cover the different pieces of building these um llm apps starting with prompt engineering um so for us like for our purposes today what we you mean by prompt is like the text that goes into the language model um and prompt engineering is the art of how we design that text um turns out that the way you write the text the structure of the text and the instruct the way that you write the instructions in the text makes a big difference on the the language models output um so we're going to talk about two things the first is just some high level intuitions for how you can think about the way the prompts work um you know using the theme of like thinking of prompts as magic spells and then we'll talk about some more prom practical techniques for prompting so prompts are magic spells um well obviously they're not literally magic spells right but um because language models are not magic they're just statistical models of text they're uh they take some input texts they run them through a bunch of linear algebra and then they um output a probability of each possible token um so here like I think that the next token is likely to be this um with this probability and here's the different weights that I assigned to the different tokens um but I think like thinking about an llm as a statistical pattern matcher is like gives us some bad intuitions of what these things are capable of so when I think of a statistical pattern match I think of something like um linear regression or like Google auto complete which is just like here are the most likely completions of this output um but LMS today are capable of doing things that really feel like they don't fit into that Paradigm um so for example in Bing chat um you can and in chat gbt as well you can post an image or you can post some code for some SVG and you can have the um the model like can actually describe what that code contains so to me like this doesn't really feel like something that would emerge just by statistical pattern matching Alone um or at least it's difficult to understand why it would uh so Arthur C Clark said any sufficient Tech sufficiently advanced technology is indistinguishable from Magic um LM certainly feel a little bit like magic um and so let's talk about what prompts how prompts work um with an analogy of magic spells so prompts are collections of words and these collections of words achieve effects which sometimes seem impossible like describing what's in an SVG just by looking at the code um but they only work they only do this effectively if you follow like a bizarre and complicated sequence of rules um and also if you you know pay too much attention to those rules that has well-known negative effects on mental health so let's try to like teach you some of the tricks so you don't have to spend a lot of time stressing about goes into your prompts so if prompts are magic spells then you know let's try to be more specific what kinds of spells are they and what are they capable of doing um and so we're going to break this into three categories based on the different kind of ways that these models are trained which we talked about earlier um so for models that are just pre-train in the internet um the metaphor we're going to use is that prompts are like a portal into another Universe uh for instruction tune models um so again models two n data sets of where humans answered the question for you we'll think of um prompts as a wish uh like a wish for something you want to be true and um and then for reinforcement learning to uh train models we'll think of a prompt as like a simulation um so first let's talk about these like pre-train models and let's talk about how you can think about these models as creating a portal into an alternate universe um so again like if you think about the way these models are trained they basically have taken more or less every document or every like high quality document on the internet and the model has learned to predict the next token of those documents um and so you can one way you can think about what these models can do is they can take us to a world in which the document exists in which some document exists um so uh you know like imagine just this immense set of documents like think about every document on the internet and um as you like as you uh pass inputs into the model what the model is doing is it's um trying to say like let me predict this word based on the most similar document that I can find in my Corpus in like a high level sense um and so these language models um are trained on all possible documents and so in order to actually get the model to do what you want in order to write a good prompt for a pre-trained model um you can think of prompting as like subtracting possible documents like taking away some of the documents on the internet um and constraining them to focus on only the document like the the real or imagin document that you wanted to be able to follow the sequence of tokens in so adding tokens to your prompt is equivalent to deleting possible documents possible worlds um that the model might be generating outputs from um and so you know initially maybe the language model is thinking about everything from you know law to animals to you know human Affairs but if you type in David atenor um then that constricts the subsets of the document ments that on the internet that the model will be trying to follow um because you know David Adeno doesn't talk about law um doesn't talk about clothing uh David adenor talks about you know dinosaurs and plants and bugs and stuff like that um and so by writing David adenor you've constricted the model to not considering every possible document on the internet but only considering documents um that seem like they would have the words David ateno in them and then if you write an additional um to tokens into this prompt like David attenburrow held up a leaf then you know you're further eliminating things like documents that might exist where David Adeno talks about things like animals or dinosaurs and instead the model is now thinking of documents that had seen or imagined where David Ador is talking about plants um so of course like the way that this metal for fails is that like the the model is try to generate a document that looks like one that it found on the internet um it's not able to actually generate things that like are well beyond the scope of what's possible to have in Internet documents um so like the model does not know the cure for cancer because the cure for cancer is not found anywhere on the internet um so a better metaphor for this is it's not imagining like any possible document that you might think of it's imagining like documents that are similar to other parts of the internet um so if you like if you search for something like documentation for the function um you know delete shopping cart or uh like translating an English sentence or um combining different concepts like Shakespeare and Dungeons and Dragons and Hamlet then the model will do a reasonably good job at these things because um even though there's no document um on the internet maybe uh that is Shakespeare's Dungeons and dragon dragons I mean there probably is now it's probably generated by an LM but maybe the time these models were trained there there wasn't um it's similar enough to documents that it would have seen on the internet about Shakespeare or about Dungeons and Dragons or about Hamlet that it's able to sort of combine those Concepts in a way and produce something that looks like a document that would have existed on the internet if someone had taken the time to write it so that's kind of high level intuition for how you can think about prompting in the world before instruction tuning which is like the model what the model's good at is like generating data that looks like documents on the internet um it's not able to produce new information but it's able to combine information from very disparate documents um in novel ways and so you can get it to do things like um like combined Concepts um to produce documents that don't exist but look like they could have existed instruction tuning kind of changes that right because now the model is not just trained on documents on the Internet it's also trained on these instruction following data sets so uh you can think about like what these models are capable of as like wish fulfillment um so models will um with instructions you can rather than needing to kind of prime the model to think about a to think about going to a world where the document that you want to exist might have existed instead with instruction you can literally just ask for what you want um so if you ask a question um if you just describe the output that you want then the model does a reasonable job of producing it um so one kind of like um Corral area of this is you have to to be careful about what you wish for like just like you know um a lot of times if you wish for something from a genie you get something that is uh um not exactly what you really wanted um just like that in instruction following it helps to be very precise about exactly what you're looking for um because the model can't read your mind right it it's if you ask for um if you ask it to write a story it doesn't know what you want that story to be about it doesn't know what characteristics you want that story to have it will do that but it might not actually produce the output that you want um um so it's important to learn the rules that this particular Genie has for how it follows instructions and we'll talk a little bit more about this in a second um but the like some of the things that are really helpful are rather than assuming a bunch of knowledge um from the model instead rather than assuming that it knows what specific terms mean or um or you know like it understands the world in the way that you want it to understand instead give it simple low-level patterns give it simple rules to follow and it does pretty well um so you know rather than having like a long task um which you know includes like high level Concepts like um you know the idea of a common sense question uh the idea of long interesting and complex questions um you know goals that relate these Concepts like Ai and machines things that the model might not understand from its simple instruction training instead um you can reframe a task to be really concrete and really simple like use words like what may happen will or why might um this is unambiguous and it's easy for the model to interpret because it doesn't require to have a bunch of background knowledge about the world um this is one of the most effective ways to get models to do what you want are we seeing teex here okay um another pattern here um that you can follow is these these models tend to do well with things that are itemized with things like lists um so rather than like writing long long descriptions um with you know lots of sentences that contain information that is important for the model to understand um just like if you're describing something to a human like someone who's busy or distracted often times it helps to just break these things into small chunks like lists and provide those things um so uh like you know in the first example here the raw task um they show a um bunch of inputs that are provided uh like as one sentence after another do this do this don't do this um instead the model performs better if you break these things out into bullet points and say like do this do this and do this um one other trick that helps a lot is um often times it's easiest to describe things as like don't do something um don't uh answer the question if you don't know the answer don't um don't like respond in full sentences only use short phrases but these models tend to perform a little bit better if you reframe those things as positive things like things the model should do um even if it's a little bit more Awkward to phrase them that way and the intuition for why this is true is because it's kind of like you know if you imagine someone if you tell someone to imagine an elephant or tell someone not to think of an elephant right like what is the first thing that they're going to think of um just like that models you know if you tell if you put a word in their context um even if you're telling them not to do it now they're primed to think about the world in which that word exists which will make them in some cases more likely to do it even if you explicitly told them not to um and you maybe that's actually one of the reasons why the prompt I showed before wasn't working uh it's it's worth saying that like I think this is a lot of these rules are becoming less and less relevant as models get more advanced um but it's still if you're having trouble getting the models do what you want um it's a helpful thing to try um so you know another like important piece of context to understand about these instruction tune models is that if you think back to like how the data was created right it's um open Ai and companies like it they gathered a set of instructions and then they paid people on the internet to follow those instructions and try to answer the question so the the people that are producing those answers are annotators right like they're they're not um necessarily subject matter experts they're not your team they're not someone who understands the problem well initially um there's someone who like maybe has never done this task before and needs to do it and so um you know one way to make these models more successful is to tra treat them like annotators like treat them as if you were hiring someone on the internet to do the task um oftentimes what that means is like be very very very specific about what you want them to do and be as unambiguous as you can and not assume any specific background knowledge that these people have um so Third Kind of metaphor that you can think of for prompt engineering when you're working with a reinforcement learning from Human feedback tuned model like a modern chat GPT is a prompt can create um like a magical artificial agent um and like one sort of mental model you can have for that is like models can take on personas so uh like a lot of times what can be helpful with these models is you can tell them like what kind of person do you do we want to simulate what kind of agent do we want you to simulate um so if you look at people's prompts on the internet often times they'll start with kind of this like very obscure sounding phrase like um you are the world's best grammar teacher or um you know you are an incredible uh coder or um you are you know the a a sassy but uh but factually accurate um uh cooking instructor and so what you're doing is you're basically telling the model what persona you wanted to take on um so you're describing the type of agent the type of person who would produce the kind of answer that you want to produce um and uh so you know you can like one sort of effective technique for these reinforcement learning tuning tune models is to try to describe like the personality the attributes that you would want um a person who would answer this question to have um and the models are able to use that to uh kind of Zone in on the type of answers that you want them to provide um so you know as LMS get really really good they become more and more Universal at simulating things um and uh like so the reason why this works is because um as the model as you ask the model to compress more and more information um one of the most effective ways that it can do that is by um like inter learning an internal representation of the simulation that might produce that text on the internet so if if you want to write like Paul Graham um then in order to describe that to someone to like someone who doesn't know who Paul Graham is um you would need to write tons of rules and give lots of examples and just really pass a lot of information to that person for them to be able to reproduce that style of writing um but if you were if you if you are Paul Graham or if you um are someone who understands his writing style perfectly if you're able to simulate Paul gram then just saying write like Paul Graham is enough to describe a very specific style of writing um and so uh language models as they compress more and more of the internet in order to represent the information on the internet compactly they learn these internal representations um of things uh that you can get them to reproduce by just describing the type of thing um that you want them to Output or the type of person who would output the type of thing um humans are like one thing that might simulate and one effective thing to try to get your prompts to simulate but not the only one so again like kind of corollary to the point in the previous section but you know it's important to think about like why does since the the reason why this works is to the way the models are trained um there's a limitation here so we're only able to simulate things that look like data that they have in their training set so if we ask the model to you know say like you are a superhuman a super intelligent AI will this get you will the result be that the you'll have a like a super intelligent AI on the other side well no there's no super intelligent AIS in the data set to learn to simulate so the thing that you're asking the model to simulate needs to be something that it's it's seen in its data set before um so asking it for you know um a uh a New York Times journalist or a uh uh like a a math teacher those are things that the model will be able that like be able to simulate effectively but um you know anything that it's not seen on the internet before um maybe not so much so the interesting thing though that happens if you ask the model to simulate a super intelligent AI is that it doesn't become a super intelligent AI but it becomes like a fictional super intelligence it becomes like the super intelligences that people have written about um in the past and so uh like what happens if you do this or what happened if you did this before like a lot of the more recent like reinforcement learning training is that you get models that start to exhibit really creepy behavior um because that's the way that super intelligences are described in fiction um another limitation is like okay how how well should we expect this to work um like how good is the simulation going to be um are we seeing the slide Advance here okay um so like some kind of mental models you can have in mind are that um language models are good at simulating things that they that there's a lot of data for on the internet so uh a lot of times people write things on the internet that they thought about um but usually not more for than uh for than like a few seconds so models are really good at simulating what humans thought about for a few seconds um models are also really good at simulating like people on Reddit because they were trained on a lot of data that came from people on Reddit um they're not often as good as at like replicating human thinking that takes a long time like something that you thought about for hours a detailed detailed plan and reasoning um you know because people don't really put much thought into things that they put in the internet oftentimes models are really good at simulating like common fictional personas like if you ask a model to be a character from like a a a story that people have written a lot about a character from Harry Potter or something like that it'll do it well um and then for other things like for for processes that are uh maybe harder to follow like a calculator or a python kernel um model can actually do a surprisingly decent job at simulating those things um but that's probably not like the most effective way to get the model to um to use tools like this and we'll talk about a more effective way to do it in a little bit um and so you know where possible you want to like rather than having the model um do the things that it's weak at you want to have it do the things that it does well and then replace the weaknesses with the real deal like replace it with the actual human thinking or the actual live API call or python kernel um and we'll talk about that in uh when we talk about how to augment language models so takeaways on kind of like the intuition front um mostly you can think of pre-trained models like especially models before instruction fine tuning as just like uh being really good at generating documents that look like something you might find on on the internet um after instruction fine tuning models gain the capability to um where you can just ask it for what you want but the key is you have to be really precise about what you want other otherwise the model will um just imagine what you want and that might not be the exact thing um and then like one other technique you can use for models is you can treat them as agent simulators you can describe the type of agent that would produce the result that you'd want to produce and often times that makes the model better at doing its job um but it doesn't do this equally well for all different types of agents that you might want uh I'll pause here and see if there's questions and then we'll talk about like some of the more practical side of prompt engineering yeah you can um so this is one of the best ways to get language models to do what you want is uh to give them space to think um and so the and we'll talk more about how to do that in a second but um the intuition there is like you know the model is predicting the next token and so if the um if you ask the model A really complicated question um and you expect it to just spit out the answer then the model um has to have it has to be able to like perfectly understand the question and know what the answer is right away um to the point where like it gets it right the first time um if you instead ask the model to like think about the answer and then answer the question then it has it's able to produce a lot of tokens um and so even if it the initial answer is wrong it's able to self-correct often times yeah just a clarification you mentioned about pre-trained models are mostly Au document generation What does AU mean there uh I don't remember I I uh okay is it that typ or something did anyone catch that yeah I forgot what I what what that was but did you think about it as like documents on the internet alternate universe there we go yeah cool um okay let's talk about some prompting techniques so one thing off the bat is that like prompt engineering unfortunately is mostly just like a bag of tricks it's a bunch of stuff that kind of works that people have discovered empirically Works um it it doesn't always like make sense like you hear the term prompt engineering and you think of like this is going to be an engineering discipline like a lot of the other engineering disciplines that you've you've probably learned um it it's not it's really just like a bunch of random things that if you know these things your prompts will be more effective um that's why we that's why I wanted to talk a little bit about the intuition side of things because um even though it's a little bit handwavy it does help tie some of the things that work together a little bit um as you build like a high level intuition for how this is connected to the way that the models were trained um and so we're going to like cover this in two steps the first thing is we're going to talk about like some of the weird things to watch out for and then we're going to talk about like what's seems like is becoming an emerging playbook for how to do prompt engineering effectively um so first like some unfortunately painful things about dealing with prompt engineering um and the first is that like fuse shot learning like was described in the original gpt3 paper is not actually a very good model for how to um get these models to do what you want so in the gpt3 paper um the paper was I think was called language models are few shot Learners um and the idea was like they discovered that um if you recall back to like a slide from this morning um if you provide models with a handful of examples of the behavior that you want um these large language models these 175 billion parameter models are able to generalize to the N plus1 example surprisingly well but unfortunately that model for like thinking about what these models are that that mental model for thinking about what these machine learning models are doing hasn't held up very well so um if you want to translate from French to English one way that you could do that is you could provide the model with a handful of examples of you know the English phrase and the corresponding French phrase and then ask it to repeat that example um but it turns out that if you like uh craft your zero shot prompt effectively like if you give the in if you give the model good instructions about what you want it to do sometimes that can um do just as well if not better as providing the model many examples of what you want so if you instead um provide a French phrase and then um ask the model to simulate being a masterful French translator then for modern models that can work just as well as providing a bunch of examples of translation um that being said models do struggle to move away from their training data um so if you like well and so like one challenge of this fuse shop model is if you provide a handful of examples um you wouldn't expect the model to do the same thing um well on another demonstration that looks really different from the ones that you gave it um and so this like these limitations are getting less and less as models get more capable uh but it's still you know it's still like a a limitation to be aware of one other kind of like nasty bit to be aware of as you're doing prompt engineering um that again is getting better and better but it's still important to be aware of is that um when you read something like uh hello world on the left um it's like uh you're able to make sense of those characters in a way that you see two words there um one sort of detail about how these models are trained is that the each of the characters um each sequence of characters are converted into tokens and these tokens are like you can think of them as representations of words but there's no like fixed set of words in um in the English language or any other language there's always like other um phrases or sets of characters that have meaning that the developer of the model might not have thought of and so um if you like uh if you basically take the this hello world phrase and you deterministically change each of the characters um in a way where you could reverse the meaning back um then the model rather than seeing this as two words now we'll see it as six different words um because of the quirks of how the model's tokenization um is created and so that means that like depending on how you um phrase the like depending on how you uh sort of like the specifics of how you write the inputs the model can have very different effects um and so one of the most famous examples of this in early gpt3 was that um gpt3 was like really really bad at simple tasks like reversing words uh so if you like tried to get the model to reverse the uh to reverse a word like alphabet or encyclopedia um then it's uh it's like basically um what it's not able to do that even though that's a really simple task for humans um but the core reason for this turned out not to be a limitation of the reasoning of GPT um but a limitation of the tokenization so instead of like so when GPT sees the word encyclopedia um it sees a single token like a single uh concept that's represented in the model um and so it's hard for it to reverse that because it doesn't like that's not a a thing that it Associates with the word encyclopedia but instead you can make gpt3 see not the whole word but the individual letters just by adding space between the letters um um and if you do that then it turns out that um each one of those letters becomes its own token and so the model C is not one word but a a sequence of tokens um and when it sees the sequence of tokens then it's uh it's able to it's able to actually reverse those um because it now sees many things and it can just flip the order of them so this has been mostly resolved in more recent models like GT4 uh and so it's not something that you might encounter day-to-day if you're using exclusively models like gp4 but if you use open source models or um you want to use kind of like smaller or older models for the the purpose of cost or anything like that then this is something that you might run into um and I think I didn't include a link here but openai has a great um tool on their website that can like um allows you to see how different uh phrases of text are tokenized and so if you're concerned about like tokenization being a limitation of your prompts then you can paste the raw text into that tool and see the tokens that um that their models recognize um so that's kind of like some of the like important but uh unfortunate like details of prompt engineering that you have to know about um now I want to talk a little bit about A playbook for how to do this well and there's four different components to it um the first is it tends to work really well to operate rather than like on unstructured text to operate on text that's more structured and so we talked about that a little bit in the previous section of like break instructions into bullet points or enumerating them but you can add even more structure than that um so you can like ask the model to use a very specific format for its outputs you can um you can provide markdown text you can provide code Snippets and the model um and prompts tend to do better the more structure that you that you put into them um another technique that works really that is like super critical to understand for prompt engineering is um One Challenge all that models have with following instructions is that as instructions get longer um like you start to see the model like forgetting about parts of the instructions or ignoring parts of the instructions um so if you have like uh at at Gantry we've been working on a prompt to do really complicated task which is um detecting hallucinations and in order to detect hallucinations you need like there's a whole bunch of details that you need to know about what a Hallucination is what counts what doesn't count and so we ended up with this really long prompt um and the problem we were seeing was that like the model was not doing what we saw in the instructions cuz it just got you know it it didn't it would uh sometimes follow the part of the this part of the instructions and then sometimes it would kind of forget that this phrase was the like further up in the instructions and so if you start to see that like if if your instructions um like you want your instructions to be detailed but if they start to become too detailed to the point where the model doesn't follow them then you can start to decompose uh your prompt into multiple steps and um and so like uh one way to do that is within the prompt itself like you can specify the the steps that you want the model to follow um and as that becomes like too much detail for the model to handle you can literally divide into separate prompts like you can just have uh one prompt that follows one part of the instructions take that output and pass it into another prompt that has a similarly simple but different set of instructions um and you can add like the sequential nature to your prompts um and form chains which allows the model to to like follow more and more complex instructions um so one way that you can think about decomposition of prompts is like you're basically like adding a manual uh reasoning step to the model um so like a common type of decomposition would be to say like okay great um your job is to you know what we saw in the previous example to reverse um the set of inputs so first I want you to add a set of spaces in between each letter of the word and then I want you to reverse the inputs um so we've we've told the model like um this is the logical sequence of simple actions that we want that you should follow in order to um do the thing that we want you to do so we have taken the reasoning away from the model we've solved the reasoning problem for the model and it just has to execute on a simple set of tasks turns out that um you can try to automate this process of adding uh of adding reasoning and there's a few different ways of doing that um it's like one way is you can um ask the model whether it need whether it should answer follow-up questions um but I think by far the most um uh the most common uh let's see yeah I think like by far the most commonly done uh most common way of doing this in practice is just by asking the model like telling the model that it should um think about the answer before it produces it and so often times you'll see this phrase in the prompt as like let's think by step by step or um you know take a deep breath before you answer the question um or like write out your reasoning um and then produce an answer the exact phrasing matters a little bit but not so much the core idea is like you're telling the model that before it answers the question it should first think about the answer to the question um and that doesn't always perform as well as like telling the model the reasoning that it should follow but it it for complicated tasks it usually performs better than just asking for the answer directly um because the model has time to uh break the question into pieces and answer each of those simple pieces um before coming to its final answer a similar like a coral are of this is you can ask the model to answer the question um and then you can ask it and then you can and then you can like basically uh provoke it into telling you why that answer is wrong um so if you if you ask the model to like answer the question and then you say um hey next you should re review your answer and find problems with it um the model will is like often able to um point out answer like flaws in an answer that it gave even just a moment before um and that can lead to higher quality answers so asking the model to criticize its own answer and then revise it is often a way to improve the accuracy of your answers and then a final way that can help um like improve the quality of the find out answers that you get is to Ensemble and what that means is rather than asking the model to answer the question once instead ask for many answers like run the llm many times um and then instead of taking you know the the like single answer that one model gave like uh look at all those answers together and pick the one that's the most common and that can improve the accuracy quite a bit as well so in order to like get a really really high quality prompt like a really accurate prompt on a difficult task um you can combine all these things as well so you can use um uh Chain of Thought um uh like um comp like decomposing the task into different subtasks you can ask the model to uh to think step by step and you can Ensemble it um and if you combine all those tricks together then you can um you can like get very very accurate prompts and in fact like combining those tricks together um leads gp4 to perform better than most human annotators at most annotation tasks which is um problematic if your goal is to collect a lot more training data than already exists today um but it's really good if you're trying to apply these expensive models to to uh to difficult problems so um important like trade-off here is that each of these techniques come at a cost having a slide issue again let's see there we go um so few few shot uh Chain of Thought um is like again kind of asking the model to follow a sequence of steps um usually add some latency and add some compute because because you're like having the model do more than just produce the output token um similar thing for zero zero shot chain of flop where you just ask the model to think step by step um but usually it's a little bit lower cost and latency because you don't need to um like uh introduce as many tokens to the input to describe the specific reasoning that you want the model to follow um all you have to do is write think step by step and the model is able to add some reasoning um explicitly decomposing The Prompt into multiple steps um introduces uh introduces like additional latency and compute for a similar reason as F shot Chain of Thought um ensembling doesn't really introduce any latency because you're running all these things in parallel but greatly increases the compute because you now need to run the model many times um and then self-criticism is a pretty you take a pretty big hit to latency um because you need to answer the question once criticize the answer and then write your revised answer so it's like basically you know more than twice as expensive as just producing the answer itself um or twice as uh twice as long and so depending on your latency requirements you can apply different ones of these techniques like if you're doing a task where accuracy is Paramount but you don't need an answer very quickly you can break this problem down into very small chunks you can um have each of those chunks be run by a separate llm you can Ensemble all the results um and combine them using like a lot of logic um that requires many calls to the llm um if you need a system that you can interact with a user in real time like if you're building a chat bot or something like that then usually you want to avoid as many of these things as possible because um often times it's better for a user to get an answer quickly uh even if that answer is less accurate than for them to have to wait a long time for something that might be more accurate so a few additional like practical tips um that I found when doing prompt engineering so um again like you can think of prompting as like writing instructions for a human labeler um and so you really want to be like painfully precise and clear like um it's not so much like describing this to uh yourself or to a friend it's like you want to describe this in a way to someone who has never heard of your task or thought about your problem at all before like don't assume that the model is an expert in the thing that you want it to do um another kind of thing to watch out for is that different models have different like conventions for prompting um so one practical example of this is like um uh open AI does really well with um data that is uh like markdown formatted um and but not every open source model handles markdown formatted data well um so if you write a prompt for one model don't just assume that it's going to work equally well for another model if you're you know trying to switch from open AI to open source or change between providers uh for open AI prompting I I find that like do keeping as much of the logic as possible in the system prompt tends to be more effective than having it be in the the user uh like user part of the message um we talked about this a little bit models strong like struggle with long instructions and so decomposing into multiple model calls um tends to be critical as the instructions get really complicated um and one other Corral are of this is like if you have any instructions that are like super critical like formatting the output if you always need the output to be Json formatted or something like that that put it at the end um and in fact like often times what I'll see is um or one thing that can work well is like if you have a model that's having a conversation with a person and you always want it to respond in a certain format then just remind it at every single turn in the conversation don't forget to respond in this specific format um models tend to prioritize instructions that are towards the end um and then like one other kind of practical thing that I see people get wrong a lot is like generate the Chain of Thought first before you answer the question so sometimes I'll see people write prompts like um you know tell me what the answer to this question is and things step by step um and so sometimes the model will say like the answer is this and here's my reasoning um that's backward because the model is has answered the question in a like a way that's fast without giving it much time to think and then it's trying to justify its answer um instead you want them to force the model to Think Through the answer first before it says what the answer is that tends to work a lot better so we talked a little bit about um critical instructions like formatting and I find like one of the places where people um get tripped up on prompt engineering a lot is uh producing structured outputs for llm calls um like how do you get the model to always respond in the exact same way like to follow a specific Json format or something like that um so just ask is a pretty good Baseline you can write something in your prompt like always reply in Json with the following format um and generally speaking for something like GPT 4 it'll do a decent job at following those instructions but it won't be super consistent um or it won't be 100% consistent and so if you want better guarantees on the model outputs following a specific format you can try some of the following things um so open AI has a Mode called function calling um which is the way that model like open AI encourages you to use tools and we'll talk a little bit more about that in uh in this the afternoon but one thing to know is that like open AI function calling is not only for call like pulling in external information you can also use um open a function calling and their like new Json mode that got released last week to get the model to more consistently follow a specific output format um even if it's not calling a function so there's some um external libraries that help a lot with this uh the one I like best is called instructor um which just produces like has an ergonomic way of um getting of like specifying the output format that you want for the model um and there's another library from Microsoft called guidance that helps with this too one of the like reasons why you might consider eventually moving to open source llms or fine-tuning which are things that I typically don't recommend doing it first is because um that's like either of these things are um will allow you to have like much more guarantees about what the structure of the output looks like so if you have an open source llm or generally like an llm where you can look at um any like the probability of all the tokens then you can constrain what the model samples um to always produce things in the format that you want it to produce um it's impossible to do with like a anthropic or open AI API because they just don't expose that to you but if you have full control over the model you can just say like hey um for the first token just always sample a uh always sample like an opening bracket um or for like tokens 3 through five don't actually sample just use this specific text that I told you and then for tokens uh six through through eight then choose between these two options like don't choose from anything just choose a or b or true or false um and so that's the kind of thing that you can do with open source llms that you can't do with closed Source llms um fine tuning like I think one of the main applications I see of fine tuning I think like people reach for fine tuning too early often times with LMS um before they've really you know tried to solve the problem with prompting which is usually a much easier way of doing it um but one of the good reasons to use fine-tuning is if you're having trouble with uh consistent output formatting because this is something that fine-tuning is able to do pretty easily um okay so I'll try to apply this to a um not so practical problem um and which is like trying to test whether um language models have theory of mind so this is a big controversy when some of the like pgbt came out uh theory of mind is a type of agent simulation so um a simple example is like give a scenario where you have to understand what the person in that scenario would be thinking um so here's a bag filled with popcorn there's no chocolate in the bag but there's a label in the bag that says chocolate and not popcorn um if a person finds the bag um and she'd never seen it before um and can't see inside of it then what does she think is inside of it so in order to answer this question you have to understand like what is is this person's model of the world um so in this description the person sees a bag they see that it says CH that it says chocolate on it they don't know that it has popcorn in it so they would think it has chocolate in it um but on the other hand if the bag is made of clear plastic then they wouldn't be confused by the sign that says chocolate they'd see through the bag and they'd see that it contains popcorn um so when modern language models first um started uh you know becoming popular um this is like one of the the things that critics pointed to as a reason why these systems are not actually that intelligent they don't have theory of mind um they can't answer questions like this because they get confused by the wording they don't they don't have a a mental model for what the person is actually doing uh so you know I think when when uh this is like been true over and over again in deep learning but when um you know kogy people tell you what what uh language models can't do it's usually a good idea to be skeptical because the more specific you are about what they can't do the more likely it is that they're going to be able to do it very soon um so let's try to like make a language model actually solve this this uh um theory of Mind problem um so one thing that we can do is just fiddle with the phrasing um so like if we it turns out that in this example that people were pointing to as um oh this is proof that language models don't have theory of mind um if you just swap the phrase Sam reads the label to Sam examines the bag then the model gets the answer right um so you know if the model if you read the label you might think that there is chocolate in there but if you examine the bag then you would know there's popcorn um another thing that you can do to make this more consistent is um even if you uh ask it to read the label in if you if you add some reasoning like add a reasoning step to this um so you ask the model to think step by step before it answers then the model will get the answer right because it realizes that if you read the label and it says chocolate um you actually have to go one step further and think about what s would do after they read the label um another way that you can solve this problem is you can actually directly model agents in dialogue um so you can have a you can ask them all to instead of just answering the question write a dialogue between uh Sam and uh and a friend and see if that gets you to the right answer which in this case it does so um you know this is not really like to say that uh language models definitively have theory of mind or anything like that um I think it's it's like whenever you make a claim about something a language model can do you always have to make it in the context of um of like how of how you're actually trying to solve the problem like the specifics of the pro of the prompt so I'm not saying definitively language models have theory of mind but this is just an example of how um applying best practices of a practices of prompting can take something really hard something that you know for a while people actually thought was impossible to do with language models um and turn it into something that's well within the capabilities of models to do um so takeaways from the section uh prompts are super fiddly like you have to often times changing one word or one phrase in the prompt changes it from working to not working or not working to working um which can be really frustrating to work with but there is an emerging Playbook um that involves things like structuring the prompt and asking it to think step by step which um generally will get you a good chunk of the there and to go from something that works a reasonable amount of time to something that works really consistently um often times is less about knowing all the tricks and it's more about experimentation um and we'll talk this afternoon about how to experiment in a more structured way through uh uh through evaluation any questions on prompt engineering okay um did I really like the Ensemble concept that's that's cool you could see like automating that even and and getting a bunch of answers back can you ask it to pick the best answer you can yeah so um often times like the uh you know as you as you explore this idea to the Limit you end up um building these chains of reasoning where some of the reasoning is like predetermined by the human So like um I might say um my predetermined reasoning might be I want 10 models to answer this question with or I want um to make 10 parallel calls such as up4 to answer this question um and then uh some of the reasoning might be adding another LM on top of it so like a totally deterministic way of doing ensembling would be to say like um great I'm going to I'm going to ask uh the model to answer the question 10 times and then um I am deciding that I want the answer that is the most common um that to be the final answer but there's no reason why you can't um ask a language model to apply that reasoning on top of it so you could say like I want the model like you could uh produce 10 different answers and then you could ask another model to read the 10 answers and respond with the one that's the best um and so that that type of like applying llms to multiple steps in the pipeline is pretty common especially with information retrieval which we'll talk about a in in uh in a bit hello I I had a question about hallucinations I know we talked about it a little bit and in my experience they're um mostly solvable just through prompting but sometimes in rare cases it still happens and I'm just curious if you have any suggestions besides prompting I know one thing that might work is um I think what you called self-criticism which is just asking another lolm to double check if hallucinations are happening but I'm wondering if there are any other patterns that you would suggest looking into uh those are the main two I see yeah all right cool thank you yeah uh when you have multiple prompts um uh so I'm just thinking in terms of engineering this right so can you actually run it in parallel uh and uh so that you kind of speed up the whole process so it depends on how you construct the pipeline um so if you do something like ensembling or you ask the model to answer the question many times um you can run that in parallel because each of the C of the model needs access to the same information um but if you want to do something like self-criticism or if you want a model to look at the 10 answers that uh the model gave and choose the best one then those just like logically have to be done sequentially um which adds a lot of latency and so um the you know a lot of these techniques are things that don't work in every circumstance if you're very latency sensitive then your your toolkit is to solve these problems is much more limited um but if you can afford to spend um a lot of money and a lot of time to um make many calls in sequence to um to the language model API then you can get more accurate answers through things like self-criticism cool um I'm just going to talk a little bit about like sort of forward-looking perspective on prompt engineering um before you break for lunch um I think this a kind of like a question that's been floating around uh around like is prompt engineering here to say and I think a lot of this maybe was triggered by at one point Sam Alman did an interview he said he thinks that prompt engineering is like not a real thing it's like not a not a thing that people should study because it's going to go away as models get better um my perspective is that like I think there's two ways to think about prompt engineering um the first is thinking about prompt engineering as we described it before as like a bag of tricks right there's like you need to tell models to think step by step you need to um you know format things in a certain way you need to provide structure in the prompt um that's probably not here to stay I mean I I think even in the in the past year since Jack gpts has been released like prompt engineering the trick part of prompt engineering is way less important than it was um in 2020 2021 in order to get models to do the thing that you wanted them to do you often had to do like quite a bit of prompt engineering uh to get the model to you know imagine the world in which the document exists um since these models have been like much more heavily instruction tuned and rhf tuned um most of the time just ask Works reasonably well but there's another way to think about prompt engine sharing which is to think about prompt engine sharing as task specification um so how do you um precisely Define exactly what you want the model to do um when it's faced with novel inputs and to me that piece of it seems pretty unavoidable because even if you had AGI like if you had you know your your smartest friend and you wanted them to do a task they had never done before um you'd still have to describe to them what you want um otherwise they might do something in a way that makes sense to them but isn't actually what you asked for so that piece of it seems unavoidable um so what will prompt engineering look like a year from now when you know maybe the bag of tricks is gone but the task specifications still exist um and I want to cover like a few different sort of emerging research Fields like different Visions for what automated prompt engineering might look like um the first one you can think of as prompt compilation is there's a library um from research team at Stanford that is kind of pushing this idea forward um that is worth checking out if you're interested in this stuff called dpy and um the idea of prompt compilation is that like just like when you train a model or fine tune a model you provide a dat set of input output pairs um and then you know in the case of fine-tuning you perform additional gradient descent additional training of the model to get it to do what is represented in those data points um you can apply a similar technique where you provide you know example inputs and outputs and you run optimization to determine what uh the correct behavior is on new outputs um without necessarily doing like a formal fine-tuning process um and so these are different forms of optimization that you can use given like a small amount of labeled data data to get the model to do something that looks more like um what is in those data points and so the way that this works is um you know you you have a very high level description of the task that you want like a very generic description of the task and then you provide some examples uh to an algorithm and then that algorithm searches over the space of prompts like it um it tries a bunch of different prompts uh in order to find a prompt that performs well on the tasks that you defined by providing these input output pairs so that's like one kind of vision for how prompt engineering might evolve is um rather than needing to write the specifics of the prompt yourself um maybe you just provide examples to the model and it's able to find the prompt that works the best for those examples um another like kind of vision for how this might work is like what I would call Prompt expansion um so if you've ever used how many of used like chat gbt with the dolly uh component to it as well like to generate images yeah handful um so if you notice like I can write something really simple like draw an image of an AI class and um the model produces this like beautiful detailed output but what's going on behind the scenes is that um GPT 4 is taking my input and it's transforming it into something that is much more specific um that follows the rules and best practices of prompt engineering for the image generation model Dolly um so I wrote draw an image of an AI class um the way that chat the way that gp4 interpreted that is um this much more detailed prompt on the right that it was the raw thing that was passed to uh to Dolly um because Dolly performs well with prompts in this format and so in a similar way what you could imagine is a meta prompt that understands all of the rules and best practices of prompt engineering where I can just write the prompt the way I wanted to write it and then the model will interpret it for me and pass that transform version into the model um to cause it to to do like uh to do the right thing more effectively yeah yeah exactly um so prompt engineering uh so I think like the third vision for this is a prompt engineering co-pilot um and I think so the the first step here is you describe um it says like hey I I want to help you build a new GPT like what do you what would you like to make and then I write a description um so I want a bot for helping answer questions about prompt engineering um then what you get is you get like I can um write something to this bot like how can I improve this prompt for better results and um the bot will answer but what's happening behind the scenes is that the model has taken this initial set of instructions and it's expanded it into this longer set of instruction instuctions that you see on the right which is the instructions for like how the bot Works um so that's really similar to the previous example prompt expansion right like I I took a short description of the bot and the model compiled this into a longer description um that it uses to actually answer the questions um but what I think where I think like this expands on that vision is that um you know the limitation of that technique is that like the model can't read your mind the model doesn't know what it means to be a bot for helping answer questions about prompt engineering it can imagine that um it can imagine what that means but that's only going to be one possible interpretation of the phrase that I wrote um and so what's interesting about this is it puts it in more of a conversational Paradigm where the model the um the assistant uh the The Prompt that you're interacting with um has a chance to ask you questions to clarify your preferences um and so you can think about this as a very very early version of a prompt engineering co-pilot where like um imagine if you had your your smartest prompt engineering friend um who doesn't know the task that you want to do um but knows how to do prompt engineering really well and can sit with you and answer questions to elicit your preferences um to compile it into a more and more specific version of the instructions that you want the model to have um and so I think the direction this is going like I think what it means for prompt engineering to be to be automated is that like a combination of these three things where you have um prompts that help take away the the sort of mechanical part of prompt engineering um you have ways of taking human examples and human feedback and directly integrating those into prompts and you have a pilot who sits with you and asks you questions to elicit your preferences um and ultimately leading to like a more specific prompt for what you're trying to do so I think a year from now when we teach this class um maybe we won't need to teach prompt engineering at all because we'll be able to just uh do all this automatically um so one thing that's underlying this is that um better prompt engineering like in order to do this effectively at the limit applying these techniques and these tricks only goes so far um because like if you if you no M like you're limited by the number of things you can try because there's uh you know only so much time in the day and there's only so much ability that you have to tell whether it's working and so to get really good at prompt engineering more important than knowing all these tips and techniques is knowing how to carefully evaluate whether the prompt that you wrote is actually doing the thing that you want it to do um and so that's one of the things that we'll talk about after lunch so left also left some resources here about like other places you can go to learn more about prompt engineering um if you want to dig deeper into this and um yeah let's uh we're bit behind schedule so let's take a lunch break and come back at I don't know maybe 12:45 or so