Omar Khattab, DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
[Music] so as Devin said I'll be talking about the DSP framework the title of the talk and also the corresponding research paper is compiling declarative language model calls into self-improving pipelines this is Joint work with a lot of folks at Stanford NLP um as well as Berkeley Ai and other uh folks and this is all open source of course at ds. so the context of this work is that through our research and lots of um you know the research that the community is doing and lots of the exploration that the uh open source Community has been doing uh it's become very clear that we can actually build good reliable systems with these new artifacts that are language that are language models um but importantly this is conditioned on us adapting them as well as stacking them well and really the second part here is is key so if you've been you know following this space you would see that there has been a tremendous amount of prompting techniques being proposed every week uh things like Chain of Thought um let's think step by step retrieval augmented generation uh agents uh program of thought selfrefined all these keywords and names that are being proposed in the literature and that practitioners are adapting in all kinds of applications at the same time these techniques are not being applied in isolation in order to build high quality systems what we're increasingly seeing is that people are stacking language model calls into bigger pipelines uh in order to build uh sort of endtoend systems so three years ago we built this Bine system which answers complex multihop questions um by fine-tuning a bunch of language models and retrieval models in this complex pipeline that I have here at the bottom and the details of the pipeline don't matter you know as much today but what matters is that um you know increasingly and and in all kinds of tasks we're seeing such pipelines pop up so D SQL is a different cool uh pipeline that works on text to to SQL generation and as you can see it has all of these stages and and and pipeline steps in which it will invoke a language model or an interact with a database and RAR is another one from Google in which they sort of uh adapt a bunch of language models through prompting and intera interactions with retrieval in order to generate text and then ground that text and fact check it in order to make sure that it you know it remains faithful to documents in the collection so the cool thing in this space is that when we're building pipelines like this and Building Systems like this uh with these sorts of techniques there's a lot of potential for modularity in principle it's conceptually very clear to say I'd like to do Chain of Thought with f shot prompting and I would like to use retrieval in the loop so I want to do this with rag um and maybe when I'm building my examples I want to use you know challenging negative ative examples conceptually to someone who works in this space these few words are almost well defined you know we we kind of we can conceptualize what these things mean but in practice in order to convey these sorts of structures to a language model it is pretty tricky you need a lot of prompt engineering and the reason for that is that language models are pretty sensitive to how exactly you instruct them to do things and the message of my talk is that you know this gets a lot trickier when you're building pipelines you're no you're no longer trying to convince the language model to solve your task you're in instead trying to interact with it such that its outputs are predictable enough to be inputs to other stages in your in your pipeline and you really don't want to do this by hand despite all of that if you look at existing research and a lot of Open Source Frameworks are really popular um they currently are based entirely on the notion of string templates so someone sits down writes this long string in which they sort of like say all right you like I I'll have some string inter you know variable interpolation here that inserts some inputs from whatever earlier stages of the pipeline are there and I'll sort of try to coers the language model to give me the right output by being very subtle about how I say things the problem with this as as you all know is that it's super brittle and in addition to that it's not scalable the language model changes over time your prompts Break um you want to plug your module into a different pipeline it doesn't work so well with the new components anymore you want to you want to let it interact with new tools maybe a new retriever comes out or you know a new database um has a slightly different SQL flavor and now your pipeline doesn't work as well anymore and examples of this are everywhere I have this sort of um um long prompt from a popular um uh library for interacting with and chaining language model calls I have these Snippets from research paper appendices and GitHub releases so this is kind of the pervasive way we are doing things now as a community and what I want to say today is um the DSP Paradigm is different it says let's program not prompt language models so let's sort of dig deeper into what that um new way of thinking entails our goal is to shift your focus as a developer or uh you know a practitioner in this space or a researcher away from tweaking the language model itself or the language model prompts and into good old software engineering and good overarching system design and in particular we'll achieve this by treating the language model as your device you know if you're working right now in in in typical sort of deep neural networks usually you assume in your mental model that you have a CPU and a GPU and there's sort of nice abstractions things like pytorch where you could just sort of say I want to have a matrix multiplication or I want to have a linear layer or a convolution layer or an lstm or whatever it is and it can sort sort of start stacking these and conceptually you're working on a much higher level of abstraction than sitting down and like writing cud a code let alone like program manual instructions for your device in exactly the same way would like the language model to be a third device that you can access um in your programming and we want to do this with an abstraction similar to deep neurer networks which is sort of an area that has been highly developed and has a lot of similarities to the pipelines we build with language models and in order to this to do this in dpy we introduced just three simple Concepts these concepts are instead of writing handwritten prompts that are long lengthy and brittle you're going to specify natural language type signatures so these are simply Declarations of the behavior you expect as opposed to uh you know specifications of how it should be achieved so this is going to focus on the what and not the how a very nice common sort of declarative motivation here um to use these signatures and to make them useful we're going to give you and allow you to build your own modules as opposed to prompting techniques which are these conceptual things that need to be mapped into prompts we're going to give you modules that take any signature and automatically produce uh high quality prompts for your task in your pipeline and to make all of this possible we have a notion of optimizers that can take any program any set of modules that you've connected together and end to endend make sure that the prompts in there or the uh um language model weights themselves are updated um so that your pipeline achieves uh sort of whatever metrics you are trying to uh maximize so the idea at the high level is that you will be using an Optimizer to compile your code which makes language model calls so that each module in your pipeline is optimized into a prompt that is automatically generated for you or a new fine-tuned set of weights for your language model um that fits the task uh that you are trying to solve so let's sort of dig deeper here I talked about natural language signatures how do these look like and these are going to replace your prompts so um as I said these are going to tell the as what a transformation is supposed to do not how to prompt the model to do it so I have here three signatures and it's that short um you have on the left hand side a signature that just says my input is a question and my output is the answer for that question uh I have another one that says my input is a long document and my output is a summary and you can have multiple inputs and output fields and as you can see here um what's nice about this is that first of all we're going to handle the structured formatting uh and and poring logic you're not going to start you know um formatting complex strings for language models which gets messy but much more importantly because this signature sort of encapsulates a whole class of potential implementations like I could prompt gp4 in a with a really good zero shot prompt to do this or maybe I could prompt a local llama model but with a with a few short set of examples that gets it to answer questions in the particular format that I want or maybe I could fine-tune um a local model maybe a a fl5 model that's really small in order to to conduct the same signature these decisions these lowlevel decisions should not be committed at the level of your program design um and an important factor here is that we're not giving you a small set of hard-coded names instead because the language model is now our device it's going to interpret uh and infer the role of these fields using their names um and traces that it runs what at compile time through your modules and we're going to sort of see examples of this in a minute the second sort of uh pillar here that matters a lot is modules so once you've defined the signature in order to actually use it in your program um you need to use a module and a module is going to abstract uh prompting techniques that uh you might use so in particular it is acting like a parameterized layer if you think of neuron networks this could this is a layer that could be optimized uh within a larger uh uh program that you're building and it will um express your signature by um inter internally implementing a prompting technique so some of the modules in dpy include Chain of Thought So you could say I have this signature and I'd like to um basically attach it to a Chain of Thought module in order to solve this question in an effective way by thinking step by step in a way that is optimized for my pipeline program of thought says I'd like to solve this signature by actually writing code and executing that program that little program and using the results in order to inform the output of the language model and react is is saying I'd like to build a small agent that solves my sort of or implements my signature by interacting with a bunch of tools that I specify uh once a language a module is declared with a particular signature it gives you a callable function so you can give it the inputs of your signature the values that fit these values and it will sort of internally conduct the language model calls and give you back the outputs that you need and what's really important here and this is sort of the Crux is that in order to express a particular particular signature we as the dspi sort of um developers need to worry about three decisions and this is what we're going to optimize for you um which language model to call what prompt instructions and sort of what what details of the prompt are we going to um map your signature into and maybe the most you know um fundamental or you know powerful uh um lever we have here is what examples of this um sort of module that we have are we going to create and then utilize in order to teach the language model whether through fuse shot prompting or fine-tuning how to implement your signature in an effective way so let's get concrete this was a lot of sort of of the high level picture but let's get very concrete let's solve uh an interesting sort of simple task that allows us to expose a lot of these um um um components so this is the task of multihop question uh answering with a data set called Hotpot QA and you can actually sort of play with this in on the ds. a GitHub repo um U there's a lot of examples with this and with other tasks so the typical structure here is that we're going to get a question something like which a word did Gary zuk's first book receive and these questions are interesting in the sense that there's usually some level of indirection right I first need to know Gary zuk Cav's first book in order to know what award it received and the task has two subcomponents we need to retrieve the right documents so in particular a document about Gary zukav and then a document about his first book which is the Dan answering wly Masters and then we need to use them to correctly answer the question so if you're using another tool maybe Lang chain or llama index or other other things you could build a simple prompt template for this task you could sit down and say all right I know how to prompt GPT 4 or GPT 3.5 and I know how to sort of get it to do um thinking step by step I'll teach it to do uh Chain of Thought and then you know extract an answer from that and this is a prompt that does precisely this and you know I could then write a small wrapper code around my prompt and extract the answers accordingly the challenge uh here is twofold first well you got to write this prompt where you're making a lot of ad hoc decisions about how to sort of uh coers the language model to follow your instructions the other thing is well it doesn't really get you too far in terms of quality because you know uh you don't really have that much insight into the specifics of what gets the model to answer these questions correctly you could also use an off-the-shelf chain that that's already pre-implemented so for examp example you could use a react agent that sort of knows how to interact with retrieval and other tools and indeed there are open source implementations of these things that sort of accept uh arbitrary tools the problem is whether you do this with a zero shot prompt or a handcrafted prompt for other tasks um you know you're only going to score around 30% answering in terms of answering these questions correctly so what I'm going to show you is an alternative to this where we're going to use DSP we're not going to write any prompts by hand I didn't write prompts internally in dsy that that solve any of this you're not going to do it we're going to rely on dpy to do it and to make things sort of um more um um instructive and also more realistic we're going to build up our program in three stages starting from something really simple and growing it into a more sophisticated thing that achieves really high quality so just as a start maybe we can build a Chain of Thought um module in which it just takes the question it thinks step by step and then it outputs an answer you know before we jump into like interacting with tools or retrieval or anything like like that so this is the entire program in DSP it's simply one module right it's a pre-existing module and we give it our signature which is I take a question and I output an answer and then I sort of assign this module um here uh to my uh uh to my variable and you know this is not a good enough module in principle for these kinds of questions because it's not interacting with tools um for the reasons that we're all familiar with at this point so in particular if you ask GPT 3.5 this sort of question it often makes up the castle name it infers it incorrectly from the name of the uh the person David Gregory and it simply makes up a fake number of stories for that castle that said it's still instructive to go through compiling this before we build a more sophisticated uh program so in know compiling in dspi is a little piece of code in which we will uh give the model sort of more details about how to take our modules and um optimize for the metric of our choice so in particular here we're going to give the um uh compiler um one example of our uh task so you know a question and its answer and we're not going to specify anything specific to our pipeline so our pipeline might have one step might have seven steps we're not going to uh worry about what intermediate outputs are needed in the pipeline we're simply going to specify inputs and whatever our metric needs to validate which which in this case is the final answer um and for answer evaluation you know there's a lot of metrics built in one of them is exact match evaluation we can rely on this here for Simplicity so this this this code will sort of uh internally uh look at the modules that we have in this case it was just a Chain of Thought module and optimize the part in Blue uh where basically the model learns how to think step by step in order to solve this task um so the interesting thing here is that your language model itself is what's conducting these compilation process proc so I have here uh on the left sort of a dummy uh example of this but on the right is an actual kind of snippet from The Prompt that is optimized by llama 2 13 billion chat which is kind of a popular open source model um for answering math questions and the interesting thing that you see is that depending on the type of of data that you're working with and depending on the model that you're working with the compiled prompt might look drastically different because what works well for a different model or for a different task could be drastically different um all right so the question here well I guess we could come back to this um so let's build a more sophisticated program uh for this task the program that we saw before simply relied on the language models knowledge uh itself in order to conduct the task a common lesson of the past couple of years is that retrieval really helps for factuality so what we can do here is that we can build this pipeline a pipeline that takes your question um calls a retriever maybe uh Kar uh if you're familiar with that which is something we built um and takes the retrieved information and then gives them to the language model in order to answer the question so let's build that module in dpy if you're familiar with pytorch this is going to look very uh very intuitive um so what we need to do now is we need to specify two methods the first method is just initialization which is where we declare the modules that we're going to use the first module is we need a retriever and that retriever has some parameters in this case how many passages we'd like it to retrieve how many Snippets and we need a module that will actually take this context that we retrieved and generates an answer and this is not built in in dpy but dsy gives us a general purpose Chain of Thought um component and that component accepts a signature so we can say I'd like a Chain of Thought component that takes a context which is what we retrieved takes a question and gives me an answer by thinking step by step because that's the nature of this module and there are other modules that you could use here once you have declared your modules you now Define the forward uh method and in the forward method you write any python code you want you could use Loops exceptions conditionals whatever you want as long as there you make sure you use the modules that you declared at initialization so here this is a very simple pipeline we simply retrieve uh a bunch of passages and then we give this these passages and the question to the generate answer module and we simply return what it gives us um and you know this is uh if you if you test that in practice because of the nature of the questions it's certainly a little more factual however it does not actually uh NE it's not always relevant because retrieval in a single hop is not always able to answer these kinds of questions nonetheless we're sort of making progress here so let's build a a design for our pipeline that will actually live up to this task and in particular what we're going to do is um we're going to take our question and then break it down into smaller queries that our retriever can uh affect L work with so we can take our question which asked how many stories are in the castle that David Gregory inherited and we'd like to go to the language model and say what's a simple search quy here that will allow me to find um relevant paragraphs in the language model my you know ideally say well I'd like to know what Castle did David Gregory inherit that's a question that a good retriever should almost always be able to answer because there is a paragraph that would say that somewhere in the web given that information um uh given that information we can sort of go back to the language model and say well how many stories are uh in this castle that you just told me about which is called kardi Castle another question that the retriever could answer correctly and given this totality of information it's very easy for a language model to sort of compose and synthesize the answer for this now this is the right answer and as complex as this pipeline might seem to be it's entirely sort of implementable in these few lines of the Spy code so as you can see here we have these two uh modules in blue in which sort of we can build a Chain of Thought component that takes a you know that takes potentially empty context at the beginning or context retrieved by the retriever downline uh Downstream takes a question and outputs a search query we also have the Retrievers in Orange and we have the generate answer component um that will take the context we retrieved and generate an answer and a forward function here is slightly more complex but at the end of the day it's simply a loop in which we generate search queries retrieve passages with them append them into the context and um sort of assign that context as input to the generate answer module which allows us to uh finally return the response for our question and indeed if you simply compile that under the hood this single line is mapped into this really high quality prompt for a local language model potentially in this case it's for llama 13 billion that teaches it exactly how to generate search queries for this task uh in a way that maximizes your quality and there are many different compilation strategies built into dsy that sort of take your signatures and modules and spit out high quality optimized prompts for your metric I'll skip over these in the interest of time um so the questions here are does this stuff work and this has sort of two branches can we ask new systematic questions that we couldn't have asked before and this is really interesting sort of from a research standpoint and the answer is like absolutely because we can now not talk about you know my prompt worked or my prompt didn't and we can say like when I compile my module for for language model L with with Optimizer o the quality that I get is the following which is a highly reproducible and just a more uh systematic way of getting at this and the second is can they deliver gains and the answer is also absolutely yes we built these three programs here and if you compile them you can you can see that by simply applying um you know uh the automatic bootstrapping in the compiler we can boost the quality quite a bit over simple few shot learning and you know if you take an agent and you use it zero shot or it with human reasoning this is something that you can actually outperform quite a bit by different language models by allowing thei to compile this and and moreover the multihop program that we built whereas F shop prompting doesn't really take you very far you can get into the 50s um if you um in terms of quality if you rely on automatic compilation I said earlier at the beginning that compilation also subsumes fine tuning and so the same program that we have here when compiled to a fine-tuned flant T5 model that is under 1 billion parameters you could run it on a CPU on your machine um it scores um 39 answer exact match and is actually competitive with GPT 3.5 after compilation in terms of uh passage recall so wrapping up the takeaways are let's move away from messing with the prompts by hand let's define these modules and these optimizers don't expect one language model call to solve complex problems you know divide and conquer break down your problem into components and let the optimizer figure out how to connect them together and you know if you're interested in reproducibility or just running your own programs um local models when compiled are really competitive so this is all open source at dy. and you can also find our research papers there uh thank you thank you very [Applause] much I so you well y so pretty much uh what I get is like you guys are attending with this Library kind of convert the promting engineering into an optimization tax right we convert the prompting engineering into into an optimization task yes so what kind of metric are you guys using to you know what is kind of the L function or whatever you guys using in the optimizer to tell so that the prom is getting better and better until like you get the the better version of it so the components here are you give us a metric that will measure what whatever consideration you're maximizing in simplest cases it's the quality of the answer but there you know we have a lot of richer stuff built in there as well and um each of your modules gives us three things we can optimize but the two most important are what set of instructions um to use at each step so how do we take the signature and tell the model you know exactly what it should be applying and the second thing which is really key is What demonstrations we can build that show that so the internally what the compiler can do is it could sort of run questions through your pipeline or run inputs through your pipeline keep traces see what traces actually succeed with respect to your metric and Achieve high quality and then see when I when it uses these examples as F shot demonstrations throughout your pipeline and what combination of them works well um which sort of converts prompt engineering into a discrete optimization problem that is something we really understand how to do well in AI so that's sort of the short answer but the paper includes a lot of results and discussions of these uh