Automatic Agent Optimization and the Rise of Self-Improving Agents | Vincent Koc, AI By the Bay 2025
I promise I'll try to make this entertaining uh since it's the last talk. But you'll see in the background there's some ramen there and you're going to be like, "Oh my god, Vincent, what the hell has ramen got to do with anything around AI?" Uh or so you think. So I'll make it interesting. We're going to talk about prompt optimization, but I'm going to use Ramen as a vehicle for this uh uh example. And and when we talk a lot about research topics, things like optimization and algorithms and things, it can it can get a bit confusing. So, it's good to ground ourselves with something delicious such as food. I might make you hungry. And these are some of my ramen picks actually
So, uh so I'm Vincent. Had a quick intro. I've worked for a number of companies. I coined myself a futurist. It's a self-proclaimed term. I don't think I've met anyone that's been called a futurist by anyone else. And you can see on the right there's Oculus Rift goggles. This was like one of the alpha versions
I think it was like 15 something years ago. Who would have believed that even VR was a thing. But I was putting this thing on my face. It came with a warning saying don't wear it for more than a minute. I wore it for 2 hours. I vomited profusely thereafter. So technology same as AI and anything else we're seeing is experimental. It's different
It's fun. It doesn't work half the time, but we have to keep trying and we have to keep experimenting. And it's that experimentation process that sometimes leads us to something interesting. I'm going to touch on OPIC and comment just two seconds, but that's not the focus of my talk today. So, uh, OPIC is an observability platform and if you want to find out anything about what we're doing, feel free to come and speak to us downstairs. The team are downstairs, I'm downstairs, or grab me afterwards. But the part that I'm excited about is how we go from evaluation and telemetry to action. So if we're capturing telemetry and we're capturing what's happening inside your AI application, like to me it just didn't make sense that we couldn't do anything with that, right? Like we should take action
We should fix the problem rather than telling you by the way you have a problem. I mean we're at a point now with like these IDs like cursor and other tools like that that we can suddenly get to a point where oh I I see that this is happening. I'm going to do x y and zed. To me, this X, Y, and Zed seemed like a simple thing to do, but as I went down that rabbit hole, it got a little bit more complicated. And to me, this this analogy, I think this analogy, this someone can tell me history behind this. It goes back many many years, but it kind of touches on this like impossible triangle is the best way I can describe it. And I kind of put put these three pillars and and tie that back to LLM. So, you have speed, cost, and accuracy
And when we're when we're going to build anything, right? You're going to build a house, you're going to cook ramen, whatever. You can have it quick, you can have it cheap, but you're going to lose out on accuracy. So, you got to just imagine there's a pinpoint somewhere along that triangle. Something is a trade-off with something else. And our journey into large language models and the way we started somewhat ended up in this in this space as well. You know, we started off with prompt engineering, which was what as soon as like these LLMs came out, it became this this sort of term that we we bucketed ourselves into. We went into the world of context engineering, as one of my colleagues at Red Panda earlier kind of uh pointed out that, you know, rag is not a new concept. This thing of context engineering has been around for quite some time
And then we're kind of going into this world that I like to call intent engineering. And I'll kind of touch on that a little bit because when I look at prompts and the way we've been doing prompts at the start, I kind of use this wording of like doom, I don't know, like doom word smithing. We're essentially taking words and we're bashing it into an LLM. Like you must do this. If you don't do this, I'm going to lose my job. You're going to lose your job, too. Someone's going to die. And it's just like this like words, words, words to like to try and force this model to conform and behave in a certain way
Kind of felt like it made sense a few years ago, but the more I look at it, it just seemed like utter nonsense, but for some weird reason, it somewhat worked and got us somewhere. So, you had these like people that had worked out this magic flavor of tokens, so to speak, like a like a secret sauce, uh, like the KFC recipe almost in a way to LLMs. And that was kind of getting us somewhere there, but not quite there. And then we kind of landed into this world of context engineering. We're like, well, we can't solve this accuracy hallucination problem. What the hell are we going to do about it? Well, it was like, well, we can bring some context. We can bring some memory into these models and give it some some start. But we're still stuck in this paradigm of the input as the starting point and then the hopes that it leads us to the output
And that's why I'm coining this like intent engineering. Someone's got a better wording for this, by all means. like go for it. But I had to put it in a placeholder to kind of explain this concept. And I think optimization kind of sits in this bucket. Fine-tuning to a degree. There's going to be other forms of this, but it's really I want to start with the output. I want to define the output and work backwards
Let's say I'm a PM in an enterprise company. I should be able to like define what my expected outcome is, see some examples, create a golden data set, and then that's the starting point. And if if it doesn't match that expectation, there's a failure and there's like a score attached to that or I know where it's going wrong. Right now, it's like I tell someone to write a prompt and they go off and do this whole horrid process and it's back to front. So, I think we need to flip it on its head to actually get us to a point where the model works for you in your given context and we have a bit more clarity around what that actually means. So, we start with the agent. uh it's just a really simplified view but like even when we look at the agent there's many components to it. [snorts] So when I talk about optimization I'm going to talk about one of the components but as you can see there's multiple facets to this thing or you might even in some cases have multiple agents
So then okay how does this work in combination with with each other. So keeping it simple you have an input with some data set that's going in which could explain what's going on and that's going to create the user and system prompt with that input. Uh in the middle layer you've got some tool function. I mean an agent could just be a prompt but in reality it's going to have some ability to take action. Could be function calling. It could be writing messages. You define whatever it is. And then you've got like the base which is the inference which is the model itself as well as the parameters that come with that model
So one way of optimizing could be finding a faster model at a price that makes sense to you, right? Like which is a logical step you can do like you may not have to change the instruction or you can fine-tune the model or find a better model, right? So you kind of start off in this prompt engineering jury where you go, I'm getting some output. And then you're going, well, why is this model hallucinating? Let me try to fix that. And then you just kind of vibe check it. And you're like, yeah, I'm going to bash some words into this thing and hopefully create a better prompt. And that's flawed. And it's flawed for the same reasons I spoke about before, cuz it's missing that signal, the triangle. Like, we don't know if this thing is kind of heading in that right direction, and it's kind of making sense whatsoever. But if we say, "Let me write this better, but let me test this and kind of build a bit of structure around it." That's when you actually starting to get into some sort of systematic approach to prompt engineering
And funny enough, I still know companies that are doing this that are enterprises and have smart people on their teams when the leap to this is like very small, but the delta and the improvement is massive. I think there's a fear that, you know, they're going to realize that maybe the problems they were writing were not working or not not useful. But same as in machine learning, we define the measure of success, the speed, the accuracy, the latency, whatever, and we test against that. I'm not advocating for you to use OPIC to solve this. You can, it's open source, you can get started for free. But like even an Excel spreadsheet and even a simple way to just validate and have a test case to say, is this working every time is a way to start on that. and con, you know, congratulations, you've kind of reached level one. We've got some noodle on the plate
We've get something out of a model. It looks a bit bland, but at least the machine's working. Like, we're getting an output that's sort of conforming to what we want. But we then sort of went along this journey and we're like, look, we've got the noodle, but it's still not quite right. Like, it it doesn't match my flavor and taste. It's super generic. Like, it's hallucinating. It's not what I expected
And it's really taken us like a couple years of research. And the big sort of shift that happened was essentially this concept of meta prompting and which kind of led to a lot of the some of the stuff you've seen around chain of thought and reasoning models and things like that as well. But the the short answer of the research was yes, we can actually use an LLM to do this task for us. So the part that I just showed you, maybe an LLM could write the prompt. We can test that and we can keep doing this loop over and over again. So our version of a metaprompter does something similar to this. It takes the prompt uh creates a version of the prompt, multiple versions for a better word, tests them, learns from that, and keeps iterating on that loop until we get an outcome. Now, you could do this yourself with just an LLM and a couple loops and just keep doing it, right? Like it's not a complex process
There are some other tricks and stuff you can put under the hood, but you know, in essence, this is like a very simple basic optimization methodology you can use. Um, so in this case, the chef looks at the recipe and says, "I can write a better one, but also is kind of doing all the work itself." So, a better sort of way to look at this is that, you know, you've got this like Michelin star dish, ramen, whatever, with all the ingredients that you've never seen before, and you've been told to cook. And it's kind of true of LLMs. If you're getting an LLM to write the prompt for you, and it hasn't seen this thing before, it's just going to create lots of things. It doesn't mean those things are going to be any good. What we're doing is we're creating more trials, more versions of prompts and we're just hoping that one of them are going to strike gold. And nine times out of 10, if you do it hard enough, long enough with enough context, you will or this thing has some domain knowledge in the pro in the LLM. So maybe it's been exposed to prompts before a data set
So it has an idea of how your thing works. It's a good starting point, right? So this works, but it only gets us so far. You know, we get stuck. Uh the ramen still comes out. Maybe it looks a little bit better, but you just end up with lots of different ramen. So, in this case, you've kind of reached level two. Yes, you've got a better prompt sort of hopefully, but at the same time, it's not exactly what you need. You just got lucky by creating a million different versions of prompts, the one that kind of got you closer to what you're looking for
It wasn't precision. It's kind of like me kind of coming up with lots of drugs and medicine and just giving them to lots of people and saying a lot of people got cured, which is kind of a little bit how medicine kind of works with this trial and error thing, but it's not precision. It's not designed for that specific individual mind. It's designed for a group of people uh in a given scenario. And that's kind of what's happening here with with the prompt engineering in its naent stages uh where we kind of started off. But there's alternatives to this brute forcing methodology. And apologies, I'm going a little bit fast. I'll have time for questions at the end
Like I'm trying to cover a lot of these topics in a short space of time. But yeah, there are alternatives to this methodology as well. So we can get smarter about pinpointing the exact mechanics of what is happening without having to go through that like constant like just getting the machine to carry out words. And there's a bunch of other optimizers that came before these, but some of the ones, two of the ones I'm going to highlight are own. And one of the ones is the Ger one, if you've heard from DSPY. The GE one and the one we wrote in April uses sort of a genetic methodology, which I'll talk to in a little bit. And I'm also going to touch on the the other method, which is kind of using like a retrospective, I would call it. Um I may or may not have time to go into ger but I'll kind of touch on the examples
So this new wave has kind of emerged in the last sort of 6 months or so where we're starting to see that like these things can like genuinely do some like real serious work and create prompts of like like pretty decent value that can sometimes like rival prompt engineering uh in some organizations or in some use cases as well. So a quick touch on like genetic algorithms. Essentially on the left is the example. Let's just say the meta prompting where we're starting with a prompt at the top here, the system prompt. We take that system prompt out and we say, "Hey, we're going to create like 10 variations." And we find this one here is working really well for us. And then we go down. We take that and we create another 10 variations based off that one as the seed. And we go down
So this is what we've been doing with like the metaprompting example that I touched on. But when we talk about genetic algorithms, it's literally what you would think when you think about or evolutionary. Different people use a different term interchangeably. You have this concept where you take the prompt, but in this case, you create that prompt has children, right? So you're going to end up with like multiple people uh and people are prompts in this case and a population. So the population at the top might be a population of five. But what we're then going to do is we're going to mix these children together. Uh children, they're going to grow older. They're going to mutate
their genetics are going to mutate. We're going to change little parts of their prompt so they're not all exactly the same and they're going to have children of their own. So we're going to find that two prompts work really well. What happens if they have a child? Like we join them together and they have children. So it kind of gets down this like really interesting rabbit hole of like learning and modifying and changing which is very similar to some of the concepts you see in like reinforcement learning but applying that concept to prompts. This is a really simplified view of what's happening, but imagine you've got like a population of 50 and you've got like a thousand or two three thousand trials that are happening and you end up with a prompt at the end which is like some weird combination of like all these learnings that have happened uh over years so to speak within a sort of prompt speak. So the evolutionary optimizers sort of follow a sim similar pattern but instead of going in and saying hey we're going to just take the prompt and the population um and just create prompts. What we're doing is we're mutating them and we're merging these little components of prompts together, making different versions, evaluating them, and then going back to the start
But the difference in this case is as we go through each evolution, we kind of do this concept, at least in our in our optimizer, called a hall of fame. And that kind of works a little bit like a a census is a probably a better analogy in this case of like we're we're basically jotting down what worked, what didn't, and where's the population at and what was the winning thing. So in this case, like what was the biggest most important job or what was the most uh used name uh in that time of year, right? So we're kind of keeping a record of like what's happened and that record is what's informing this optimizer at the top to then come up with new prompts. Uh we're also doing other tricks like snipping top and bottom and and things like that which are very similar to like in in ML where you don't want to reach like the same point every single time. We want to inject a bit of randomness into it. So going back to the ramen example, in this case, you're going to have multiple ramen chefs essentially competing against each other and the best dish will evolve and the losers go home. If anyone's watched Master Chef or anything along those lines or any of these like cooking programs, this essentially what we're doing in this case um where we're maintaining that population and we're going around. There's other couple tricks that we've deployed where we've got like a cold start problem or a problem where we might converge into the same prompt all the time
So every so often we might reinject the starting prompt back into the population or we might take the data set without looking at the prompt and ask the LLM to create a prompt based on this data set and inject that back into the population. This is very similar to like events that will happen in the real world where I don't know there's like a pandemic or a disease or different things happen that will change like the course of action. So we're kind of bringing in that level of randomness uh into this case. Uh Ger is very similar to this approach. The only difference with Gera, there's quite a few differences, but the main one here is that instead of doing population in the sense of I'm going to change the entire prompt, the prompt gets chopped up into pieces like ingredients instead of the entire dish, we're changing the ingredients and we're mutating the ingredients and we're stitching those ingredients together and and seeing what works. So, it takes a kind of similarish process with some other kind of differences to that. and it has a scorecard which is called like a Pareto function which looks at the relationship between that triangle that I spoke about the cost the accuracy and stuff like that. So you might end up with a prompt ideally that isn't like bloated with lots and lots of tokens and it's it's trying to balance out the needs
So if accuracy is quite similar, it might preference something that's maybe a shorter prompt instead of a larger one. So that's kind of the the very kind of the the nuance difference between the two, but they kind of behave in a similar way. So in this case, as I mentioned, we break that dish down into its components. So in a ramen, there's like the noodle, there's the broth, there's the oil, there's a spice. So you're working on those pieces and optimizing those pieces and then reassembling those dishes together to come up with like a new variation of a prompt instead of going, I'm going to build entire dishes altogether and mutate parts of that dish um as a difference. Now, where we changed this approach entirely and we decided, look, this is kind of working. What we tend to find, and the research on this is, you know, we're still going to be publishing this pretty soon, is these work great in like an example where you've got like a research data set of hugging face and you load it in and you give it a go and you're like, "Wow, this is great." But in reality, prompts are more complex in organizations. Like you take one company and their prompt is maybe like 20,000 token lines long, right? You've already reached that like maxing yourself out with that like prompt engineering work that you've done
Then where do you go from that? Like you try and put it in somewhere like this, it's going to it's not going to do well because it's always designed to start with some smaller basin and add to it and subtract to it and grow. [snorts] And the more we started thinking about this internally, the more we started thinking like actually if I was a prompt engineer and I was facing this problem, how might I think about it? Like how do you think about a a a problem which you may have no idea about? And it's really kind of simple. It kind of boils down to what we call like root cause analysis. So in this case we our next optimizer this was very recent only like released like maybe a month or month and a half ago um called Hapo which is hierarchical adaptive prompt optimizer essentially starts with a failure. This is kind of going into that intent engineering world that I was touching on where we're actually going back to front and we're going hey we looked at this prompt and we ran it and something went wrong. So in this case, it could be a very simple piece of code maybe that runs, but something about that code means the accuracy doesn't work quite right. And I'm going to show you an example where we've applied this into production. And we look at that and we go, okay, we're going to come up with five reasons why we think this has failed
Imagine you got five top engineers in front of you. Go, you, A, what went wrong? B person, what went wrong? And they might say something like, oh, the syntax was wrong. You used the wrong package. um the way the class was defined could be a little x y and zed. So you come up with these strategies or you group these strategies based on the failures and based oh no you take these failures you group them and you come up with strategies and those strategies become new mutations on the prompts. You apply that to the prompt. You create an evaluation process and you reflect on that evaluation. You go okay we reran this
this was the new score, but actually we found out you fixed this one little thing a little bit better, I think, but these other areas are still a failure point. So, we keep going asking why why why looping through. And we find that this takes you in a completely different pathway from some of the other optimizers and the way they approach it. And kind of going back to that food example, you don't know what good tastes like. You might not know like you know you may not have tested best dish or you may not have tasted you know had listened to the the most been in a sports supercar or something like that. So you have you might not have experienced that like the model may not have experienced the thing that we're looking for but it's within its realm of possibilities in a search space. So taking a dish example it's like I taste a dish I might not be the perfect chef but I can tell you that maybe it feels a little salty maybe the texture is off. I'm just kind of listing out things that I think could be going wrong in this example and then just tweaking it to see what happens
And taking all of these sort of like failure modes and grouping them together we find kind of leads us into this sort of really interesting space. Um just realized I've got the wrong two seconds. I'm going to change. So with this one I was explaining like we go through that reflective process and essentially it's similar to like a doctor diagnosing symptoms prescribing targeted medication. We get an idea, we see the output, we kind of revolve and we do that again. And an example of where we've managed to get this into the wild and this is something we're going to do more of and if you have suggestions let us know is we tried a couple of different methods and we've posted this up. You can check the link in the bottom in our repo which might be hard to see but we'll make sure we send them around is lang chain has a JSON mode. So if you have ever worked with like open- source models, you're trying to create JSON mode
They have like a function you can use. And I met people that like this thing keeps failing all the time. Can you please for the love of God like prove your Apple's worth and just go and apply this optimizer in in this case? So we applied it here. It ran for a number of iterations. We used a number of different optimizers. So we we tested the one that gave us the best result. In this case, it's gone from uh 0.12, so like a 12% baseline of like 100% accuracy all the time. It, you know, two times out of 10, and we got it to like a 97% accuracy, uh, which is like a 700% improvement on this particular prompt
And you can see we're just explaining what to do with the JSON. And we the the data set's all public. We're using examples of JSON data sets we found on HuggingFace. And we're using this reflective process to come up with a better prompt. and then we're applying that back. So, we wanted to find examples in the wild that we can test this in like live use cases. We've got a number of other similar ones that are coming up, but if you see any like multi- aent systems and prompts that are in open source or if your any of your products or services are open source and you have prompts available, I'd be more than happy to have a crack at um optimizing that for you and and applying that optimization back into your uh organization's codebase. So, here, congratulations
We've reached level three. we finally have something that kind of resembles ramen cuz you can see here that we're actually getting the JSON output that we're looking for. Now, we are going to get optimizers that are going to get more and more complex. You know, things are going to get a little bit more interesting. And I think the key takeaway for me here is that like there's going to be other forms like you remember that PI that I showed you, what happens when you have multiple agents, what happens when you have tool calling and things like that. So, we've been looking into that as well. So, we've got like optimization for MCP and function calling. We've been looking at like that kind of different strategies of how do we sort of tackle that
But I think the key thing here is to experiment, right? Like taste things, try it, give it a go and [snorts] see if it's right for you. Like if you're if you're starting with the prompt and you're struggling like what does prompt engineering look like? In my opinion, prompt engineering has shifted. Uh I hate to use the word it's dead. Um but it's dead in the old way of how we know it of like just bashing words into a into a machine and making sure that works and leverage them for your projects. And like another good way of example of thinking about it as I mentioned is like start to think about how you define your prompts internally currently in your organizations. What does your prompts life cycle look like? And is a future change where optimizers become more current. Is that going to change the way you think about it? Like a year and a half ago we had people's jobs that were like prompt optimizers, right? They were hired to do this specific job. So have a think about that
So you've got the access to the docs on the on the right and then my LinkedIn on the middle. uh I'll welcome questions but I will add that a lot of this work sorry all of this work that has been done is open source the entire codebase all the optimizers I've walked you through is publicly available and we have a number of announcements around the benchmarks and some of the methodologies that are going to be coming out over the next few days and weeks as well thank you >> thank you Vincent we are opening up the floor for questions for Vincent anybody has questions No. Oh, I see. Okay. >> Hi. Thank you. Uh it's a great talk. I guess my question is mostly around how do you measure this accuracy and how do you make sure that it's specific to the domain that the prompt is uh curated for
>> Yeah, good question. So if we go back to some of these earlier slides, let me find this like if we go back to this really simple loop, right? You've got you've got the data set which then informs the input. So without a c a good curated data set, your optimization fails. Uh and we go back to this same thing with anything in AI like it starts back at that data set. You need to define a really clear data set that kind of has good examples and bad which I'll touch on super quickly but also the metric like what do you define as success? Is it accuracy? Is it hallucination? Is it cost? Like that triangle that I spoke about is like one part of it. But like being really clear on that and that's where the opic side of our platform which does the evals comes in because we have like close to 30 different sort of evaluations out of the box that you can plug into. So maybe that's your starting point. But when it comes to data sets, if you're like, "Oh my god, like human data set
I'm not going to sit there and write a thousand examples. Maybe you take existing traces of your application, you export that and you use that as a starting point. You get a human to annotate it or an LLM to evaluate good and bad and you start there. Or you just synthetically do half and half or you use traces and get an LM to create more examples of that and then you annotate it. But there's like a bit of a workflow needed to like kind of create a baseline of examples. But we found on another paper I published is like something like 20 to 30 examples can give a signal to an LLM if it's not it it might not be the best signal but it's still a signal. So like don't feel like there's a massive amount of data needed to make this work but obviously more data means better confidence means better outcomes and likely higher accuracy readings cuz you know if you're trying to score something with five examples or 10 examples you're not going to get a good read on if this is going to work well all the time. Thank you
>> Thank you. Question. Question somebody. Thank you. Um would you say that this is sort of like doing hyperparameter tuning but instead of with parameters you're doing it on the prompts? Yeah, essentially the only difference with hyperparameter tuning, you you tend to have an idea of what the search space looks like in in a in a loosely defined way like you kind of know what you're looking for. You just need a you need a smarter way of getting there. In this case, the search space is somewhat near and more infinite than than than a hyperparameter tuning. So it becomes more complex to like find
It's like looking for a needle on Earth rather than like a needle in a in a in a room where you know you can just go through every single dot with your eyes and work your way or maybe you could be smarter about how you scan which is essentially what hyperparameter tuning is. Um does that analogy give a a bit more context to like Yeah. Okay. So this is pretty cool and new. So, um I guess the the question that sort of bugs my mind is do you have um any real world cases that you can share where using your optimization actually ended up resulting [snorts] in improvement? You don't have to share the names and so on, but it would be nice to know. >> Yeah. So, the one I've shared was the Langchain one which we've I've myself personally done and then committed back to Langchain and then within a couple of hours it was merged. Surprise
Um but some of the other use cases that we're looking at and without talking about customers I I'll give one examp two examples. The first one is [clears throat] when we think about the process of when you create an LLM as a judge where an LLM is creating and looking at uh the input data coming from another LLM and and scoring it. That's a prompt as well. So, one use case that we're looking at is like, okay, if you've annotated your data set, could I take that annotation data that you've annotated your traces on in terms of what's coming out of your AI and actually fine-tune, not fine-tune, but like instruction tune your LLM as a judge to like make the judge prompt better and actually have an accuracy attached to that. So now suddenly we go from like is LLM's any good at judging my work to like LLM are good at judging my work with a degree of confidence of X and Y. So that's like a very proven sort of pattern that we can follow. Um we've got some more extreme use cases like we've got one company we're working with which is like a large media company on a PC and I've been working on multimodal evals as well. So we've got like video based LLM as a judge that's like a whole another conversation in itself
And what we're doing is because our evals support it so does our optimizers. So, we're kind of getting to this space where like I've got an agent that the data set input is an image or a video instead of text and I want to tune create a better classification like I might be looking at safety risk or is this person an influencer or not or is there any children in this video things like that right where you just want to use an LLM to do that like a vision based LLM instead of like kind of doing computer vision stuff which is as good but it just takes time to set up. Yeah. Right. Cool. >> Thank you. >> Yep. >> Anybody else have questions? I can't see
>> No. Oh, thank you. >> Okay. Uh uh just a question. How you can compare uh opic efficiency with DSP uh prompt optimization? [clears throat] >> Yes. So when you're talking about the efficiency, uh DSPY is a uh for some of you that do know or don't know is an agentic framework. DSPY first released the ME Pro with like one of the optimizers that was like in the earlier list of like optimizers that kind of really started this race. [snorts] And the issue is probably not so much the issue if you love DSPY, but it relied on DSPY as a framework
When we built this optimization package, we took the MERO algorithm out of DSPY and we used that as our starting base because we needed something that didn't rely on like a framework and was totally agnostic. You can use any LLM you want. You can use any framework agent you want. [snorts] Since then they've done a lot of work and not DSPY but the team behind DSPY who are researchers at like Berkeley and Stanford and a few of the people at data bricks have then subsequently built GAP um or just you know partially it's a it's a sub project of DSPY but it doesn't rely on DSPY. So I think there's a lot of misconception in this space that like DSPY is an optimizer. I think there are optimizers within DSPY which is what's making DSPY really popular but we also give you those same optimizers agnostic of DSPY so you can run those same optimizers in our SDK but we also have our own optimizers as well and they behave differently given different scenarios given data sets I don't want to share just now because we're still going through the benchmarking process and we'll be publishing results but depends on your use case and the data set and domain like Each optimizer has its different quirks and they provide different results. [snorts] >> Great. More question
More question. No. Okay. I don't think I'm missing anybody else, but Vincent still here if you want to ask him question outside of this room as well. So, thank you so much. >> Thank you. Thank you so much, Vincent. So [applause] wonderful.