Workshop: Building Durable Deep Research Agents

Building Durable Deep Research Agents | Workshop | Samuel Colvin | Pydantic, p2.

Recording: Building Durable Deep Research Agents | Workshop | Samuel Colvin | Pydantic, p2.

and it can it can you know it can do the like simple find a bug but it can also do the okay I don't want that um like this kind of like open like free form go investigate You see it got an error, something to do with the schema. >> Um, it'll go and try and fix it and it should come back to me in a minute with a table of the five slowest end points. >> Um, it's failed completely to find anything from that, but like Yeah. So you see here it prints out a table of um endpoints by performance. Um and then you can go and say okay find those functions and fix them um in a way that like without ever having interacted with logfire at all. You've basically just given your coding agent access to your like execution data that it normally wouldn't have and that's enough for it to go and do a pretty good job of fixing stuff. >> I integrated with log. >> Yeah

So we have a log MCP server. So it's not actually it's just a um Python package. It's a local MCP server. So it's it communicates over standard IO and it just takes one of our read tokens to give have permission to to query our API. So the the like the the the most powerful bit of logfire the thing that makes it different apart from like you know shiny UI and things is um uh that you can write arbitrary SQL against your data to go and basically investigate what's going on. Um as you see like here these are probably not that interesting SQL but here you see I'm like uh grouping by um collocation and region and like looking at performance whatever else right it's basically an analytical database that has access to your application data as well as a like nice UI to view it um but we have an API for running SQL as well and that's what the MCP server is uh using >> and this is all stored in Mark Yeah. Yep. >> Or in your park if you want to pay us lots of money

But yeah, >> so this select is unique to >> so it's built on data fusion. Um so it's fundamentally data fusion. If you if you look at I think this should return yeah data fusion 50 um if you do like I don't remember that if you want to like yeah the SQL flavor is it's it's pretty much postgress except a few places where you need to remember that you're in fundamentally it's arrow data but it's it's basically Postgress uh syntax but data fusion is what's amazing about data fusion is it's not a database it's basically basically a library for building a database and so we basically we have we have a Postgress is the metadata store and then object store for the actual storage of the park files. So we have Postgress tells us like the names of the files and the paths to the files and what's like some data about what's in each file like minmax values and stuff like that. So when we run a query we can generally uh not have to scan through all of the park files but like go down to just the particular bits you need. Sorry. So all I've done here in um uh if you look at uh log via MCP, if you look here, there's instructions on how to add an MCP server to Claude Claude code here. So you basically follow these instructions to add another MCP server to Claude code

And then the point is Claude basically knows to go and go and query it. And I've I've cheated in one regard which is that um in the claude MD file I have said told it somewhere always look at logfire so that my demos I don't have to say use logfire 2. >> Uh important always use the logfire MCP server when you're asked to investigate behavior eg find bugs or uh errors or find performance issues. So that's why like I didn't have to prompt it and tell it use log fire MCP. >> I have a question about about the MCP server. Uh if I recall correctly last time I looked it doesn't yet have access to EVO, right? >> Well, it has access to all data because all data is exposed via SQL. Um it probably needs to be the prompts need to be improved to tell it how to find evals and what they are. >> Okay

And the table there's a table I think that you can query for evals that I've forgotten the name of. >> So maybe that's the part that I'm not connecting the dots because I was able to just use it and get like uh you know like the stuff that shows up in live etc like the evals. >> I'm very much u um trying to get like the actual score you know like or whatever output for my my judge etc. >> Yeah makes sense. I mean all of those are in spans and so it should be possible. Um but like I will get the team to find if we can't get that to work today when we talk about evals I will get the team to work out how to do it. >> Okay. >> Let me let me actually ask um >> I'm pretty sure like you say it's just knowing the table and honestly that would be enough

Yeah, >> but I would I would hope for maybe some syntactic sugar on the server that has like an end point that'll be like, oh, you just get get them here or something like that. >> Yeah, >> hopefully someone responds to my message in a minute on our Slack. Um, >> sorry. >> Yeah, if you just search logfire MCP, but yeah, I will I'll put it in here. Oh, and you pointed out that my Python is wrong inside this thing here. So, Okay, that's working. Um, I hadn't really realized that asking that nice guy to get me coffee now involves you all waiting 10 minutes. Um, but we're think we're in a rush

That's what I was going to try in just a minute. Um, So, one of the things that um is nice about the gateway that I'll just quickly demonstrate is that you can um this obviously should be better integrated, but you can put uh a log by write token into the gateway and it should um uh basically start um should send data to log fire. So you can basically see LLM requests as they're running through the gateway. So >> yeah, and you can see the see the exchange here. Uh so you can see where I'm running it here. So this is without me instrumenting the code. Um, and this is like this is the kind of corporate value is that like if you're running a gateway as a company, you can be like we're going to instrument and we're going to record every request and we can we were talking about the like guardrail security stuff but also just like understanding usage patterns who's doing what? >> Yeah. Um so light lm open router try very hard back to a single API format

>> Same thing. >> Yeah. So they do that. Um so when so one of the reasons we built gateway was that we kept speaking to pe I went and asked a bunch of people in enterprises about what they do and I had two answers only two answers from maybe speaking to about 10 companies. either we built our own gateway and it works pretty well, but like it's kind of annoying to maintain, very strong signal, or we use lightm and they're a really nice team, but it's the most buggy piece of software we've ever used. Um, and so that seemed like a quite a good indicator that we could and I to be fair, I don't think they've done anything wrong. I just think they have a like cisophian task in trying to maintain compatibility with every single API out there whilst not you don't control the client, you don't control the end gateway. you're trying desperately to make two things, two different bits of Lego fit together somehow

And that's quite hard. And then you're doing that in open source software. Most of people aren't paying for it. They're just running lightm somewhere. Um, so the the main our main feature is the lack of a feature, which is the lack of the like conversion stuff. Um, and then I think we will do stuff around self-hosting, caching, security features, blah blah blah blah blah. Also finer grain control on pricing. So you can create an API key with a very fixed limit and then go and use that for stuff or share that with people more easily I think than if you're than with the others

Although I haven't I'm not going to say I've looked at every feature of every single gateway out there. And then from our point of view part of it is about finding a way to to get hold of everyone who's using Pance AI. And so most of our examples now of Pance AI use the gateway and the signup rate has been pretty good for us. I think if you're like if you're an individual, it's probably quite easy to sign up with a gateway and put your card into one system rather than five. Um, with apologies to the nice chap who's gone to get me a coffee, should we start talking about something interesting? [clears throat] Um, I can talk a bit about MCP, uh, and how MCP integrates with, um, pinant AI if that's interesting. Um and then my example crashed. Um and then the other thing that's sort of related. Yeah

Let let me show you show you something that we built quite recently which is um uses MCP but also our chat UI. um >> yeah I'll talk about evals but I feel like once we start talking about evals um all subsequent conversation will be about evals so I'm going to talk about anything else first >> h >> don't forget about >> okay graph should I do graph first and then talk about that >> let's do graph so we don't forget it >> okay I'm going to do graph first okay so >> um We have two we effectively have two two ways of doing graphs. So the best example here this was our our original implementation of graphs and this this still exists and since pyantic AI is v1 we can't just go and get rid of it. The idea is you define a node like this. So you use this data class. It can be a base class. Doesn't actually have to be, but it inherits from base node and base node is generic and a bunch of a bunch of stuff like the state of the graph. And the the point is that the value that we return is an instance of the next node that we run

And so we have this like type- safe way of uh transitioning between nodes. So you can see here that the insert coin node is type annotated as returning uh coin inserted. Um and then obviously for typing time it has to actually return return an instance of this and it has to be because it has to be an instance of this data class it has to pass the right values and so we get this type safe way of connecting together nodes um that you wouldn't otherwise have that that doesn't exist in in for example langraph. So this is the way you creating the edge between the nodes. >> Yes. Effect the edge is defined >> when by the return value of the note and then if you look at the somewhere we have end uh which will um this this can return end or insert coin or select product. And so this ends our ends our graph. And so when you define your graph like this, all of the for in typing terms, all of the nodes of the graph need to be generic in the same types, need to be generic in the right state so that uh this all works

And sure enough, we get this like type safe graph implementation and we can go and inspect the type annotations and build a mermaid diagram of our of our graph. And and this works and and it and it works fairly well. than the support within this for snapshotting as at the end of uh each node being executed. Um we have generic implementation I think that saves to disk and you could go and implement your own. There are two problems really with this node this graph implementation. One is that the the the logic of the of the graph and of which node you would run next is baked into the node definition itself. So in theory I could I suppose I could take the function body out of this node and just define the node somewhere and define the function somewhere else. But like we but we've still fundamentally got like we we have to we cannot compose together a graph just by like passing a bunch of nodes like this because the the nodes need to know something about the graph

The other problem with this is that the way that we do snapshotting is um oh my god amazing. I haven't said anything remotely interesting yet. >> Oh my god. Um, >> I'm going to come and grab a coffee. Sorry. One minute. >> Oh, only one of these. >> Can I steal the >> Yeah, the milk and the or the cream

>> Oh my god. >> I think sit and sit and enjoy it. If people want to get cut, they can. Okay. Yeah. >> Because I think that you've missed a missed a bit already. >> Okay. >> This is very kind

I don't know how many people want coffee, but I do. >> Well, we'll find out. If not, we can just have a lot of coffee. >> Seems like a sol. >> That's very, very kind. Thank you. But that'll be a way better way of doing it. I got way too much milk

>> That's fine. I can do that. No, that's not >> anyone. Is that right if I >> go for sure? >> Anyone else want some coffee? Go for it. Now I really really need to not spill coffee on my laptop. >> I love that for you. >> Thanks. Um I'm actually gonna Can I steal this chair? >> Sure

>> Put my coffee on it and not risk No. So the so there are two problems with this this structure. One is as I said that you the nodes and the graph structure are fundamentally interlin and the second limitation is that you cannot do multiple node like like parallel node and the reason that without going into all the detail basically you cannot do snapshotting you cannot have you want you know the cap theorem of like you want three things in a database and you like it can be proved you can't have all three well there's a kind of like not so scientific thing In graphs, you want them to be type safe, have snapshotting, and do parallel node execution. You just cannot have those three things. Just isn't we went thought about it for a very long time. You end up building an entire new type system on top of Python that's like incrementable value or like append list stuff like that that you can then and then you have to end up returning like append to list or increment value as your outputs from nodes and it becomes complete hell. And so Langraph have dealt with that by basically not being type safe as in you have this state object. You can go whatever you like to it in different nodes and you know good luck

>> Yeah. But you you that's not type safe right as in um we chose to solve that in a different way which was basically not to try and do the snapshotting thing at all but instead to rely on durable execution temporal and deboss stuff you've all seen today. And now we don't have to deal with this like snapshot snapshot or like a kind of like uh atomic way of incrementing our state. We can just use normal Python constructs and then we can deal with um uh and then we can we can make it both type safe and allow parallel node execution. And so after a lot of thought, literally six months of thinking about this from from David R, our CTO, we have a new way of doing graphs which I think the plan is to eventually to replace um the the like class-based graph implementation that I showed you. Um um and I mean I'm no expert on this, but we will work through an example. And because it is it as a as code it should be it should work fine with temporal because it's just Python code ultimately that's that's running and the the the the hard work is in making making it type safe and then you can call agents pantic AI agents or indeed whatever other code you like within nodes to go and do the activity bit to do the LLM calls or whatever else you like. Um and so if we look at a uh very quick um counter example counter example is not the not what I mean an example of counting

>> Y >> um here um No. So you can do state here as we have counter state in this example. The point is that because we're not trying to do any cleverness in terms of snapshotting and we're not trying to do any form of so you you can imagine if you're running these two nodes in parallel at the end of it we get back two different states. How do I merge those two things together? I can go and build an entire basically atomic structure for incrementing state that like allows everything to be merged that is safe that is basically like implementing one of those systems like we need to be able to merge these things. If I want that to be type safe, we end up with an entire basically type system dedicated to those those constructs, right? We can't just do append to our list. We have to have our new like custom list type which will get appended and now that has to be can't know we we we have to be resilient to which order those two actually appended to because we're running these two nodes at the same time. We end up with an entire you know more and more our own programming language and we really don't want that. We want standard Python type work and so the our proposed solution to that and I I I'll be open this is this is in beta

is relatively new is to completely not try and solve that problem. We do not have orchestrators. We are not trying to do running these two things across some big distributed system. we are saying that you can you can go and run code. Um this this effectively is just a type- safe way of defining our graph and then of um uh yeah so we we basically define these functions that are apart from the state that they take pretty much independent of the the rest of the structure of the graph and then we compose them together um using this graph.add add or graph builduer add method. And this is where the the like type safety uh comes in because we're doing clever clever stuff in the edge from and to method. Um I admit I didn't write this code and I'm learning it as much as you are. So let's copy it and put it into a file and try it

Um >> question on this as you're doing that. Does it support the idea of two of a node going to two different edges? And how would that be? >> Yes, I think it does. >> Like a condition. Yeah, basically a conditional edge. >> Yes, I'm sure it does. >> Okay. >> Yes, because I talked about it a lot with them. >> Um, great

>> What is the AI use? I mean, you're talking about graph independent of AI. So in theory it's completely independent of AI. It doesn't need to be used with AI. >> I think >> time will tell whether or not graphs are really needed for AI. But there are a lot of people who are on Langraph who who are like I want to use a graph but I want it to be type safe. What do I use? And pantic graph is an answer to that question. Um I think my guess is that we will as has happened in numerous previous evolutions of tech realize that graphs are unnecessary probably in the long term. Um but if you do want the main knowledge graph or this >> this is not knowledge graph

This is this is if I come back over to unfortunately when David wrote these docs he did not add nice mermaid diagrams but like um if you look at like uh this like it's this kind of workflow graph right of like the sequence of of what happens. And some people think that this is this is extremely useful. Some people think this is absolutely key to building like state machines. Some people think it's completely unnecessary. I consider I mean someone said to me the best argument for graphs is to make people who want graphs happy. And if you think about graphs as an answer to that question then then it makes complete sense and you don't have to think too hard about what the point of them is. I think if you asked Langchain uh why you need graphs, they would say, "Oh, well, you really need it for for this like concept of snapshotting. Now I have some like durable system that can run and if one if if something fails in the node, I can just restart from where that node last failed." >> Okay, >> I think my take and the majority of people's take would be let's let proper durable execution frameworks do that stuff and then we can also get the snapshotting deep within a node

If a node takes an hour and does a thousand things, I can re recover from thing 682 rather than having to start from zero. >> And so Langraph to do it but you won't get >> well if you use lang graph you would lose the type safety >> yeah that's what I'm saying >> if you use pantic graph you get the type safety but you need to rely on temporal deboss etc to give you the durable like component of it um does that does that make some sense um so I mean the first thing to say is I've imported this example it seems to be completely type safe which is nice. Uh I'm going to run it. Um so you can see here that I have set my counter state. Um I'm going to run the graph and I'm going to find out what happens. Basically that's not very helpful. I'm going to add logfire in to see what happens. Uh and so you can already start to see the nodes that got executed

This is all basically instant because none of them are doing anything particularly interesting. But you you see that like we can see the nodes that were executed. I'm wondering if there's a more interesting example in this page that's actually going to give us some value. Um so this is a this is a parallel processing example. Um and I'm just like um so the G dot join is the critical bit and we're um how does reduce list depend? Okay. And if we look up reduce list append um a reducer that appends to a list. So we we can sorry um if we look at um sorry uh this one or the previous one uh which should print out that I mean if if you just copy paste it and run it it doesn't have anything to run. So if I look at this example yeah you can see we we've run the run the nodes

Um yeah um and we've we've got back the like state object and also got back the result value from running the graph. We see that we had our inputs and I'm just trying to understand how how we work through the the nodes. Yeah. So you can see that we're we're basically squaring um we're saying so the first if you look at the arguments to to add Okay, that's just edges. Um, we start with G dot um, uh, start node um, which is going to call square. Then we have the square node um, which then calls this collect results and then ultimately then we call collect results um, and we have going to end node. So those are our like three steps. Um, I guess you always have at least those three steps

Um, I mean, I suppose that the making this into a temporal workflow should be as simple as wrapping this whole thing in a temporal workflow. And if we have an agent within it, we need to do the agent plugin stuff. Then an example of something that I can try and get to work here rather than than just like running these examples. >> Questioning readation. >> Yep. >> That's it. Yep. >> Yeah

So, so in this case it's going to be extremely boring because all we're doing is taking I'm going to copy paste. Um so we have to define our workflow. I'm going to come in here. Um graph workflow and that's just going to call main. I mean here it's just as simple as calling main. And where did we import workflow from? I guess I need those three. Uh, and then so I define my graph and then I define my method that's going to call it from within temporal. Um, go on then

Do me some helpful imports. Um, I don't need any of that stuff cuz I haven't got any of those. I don't need the resume route. I'm just going to say workflow run. I'm not going to worry about the ID. Why is that complaining? I think this might literally just be that temporal has not got the right type annotation. So, let me try running that and see how we get on. I've got temporal running

Okay. And I ran the wrong function. So, it's not surprising it worked. Uh where is ID? Okay. Oh, so you have to pass an ID. Okay. I bet you it's going to say it wants to be a string. So I'll do that

It's complaining. import. I'm going to restart temporal in case it's got some state involved in what's going on. Um, I would say that it's not working because we need to somehow mark UUID as something that's safe somewhere. And so we thought graphs should just work, but they don't right now. I think we need to go and make some fixes. I don't know, unless I'm Yeah, you can literally see we're calling this UUID method uh deep inside the graph. and you need to somehow mark your UID as a safe method to be called within a workflow, which is I think is what roughly what this is saying

And so I thought they were working. I feel like we need to go in and and uh fix that unfortunately. Um unless unless David's already done it and I'm just not seeing um unless there's some kind of graph plugin. Presumably what we need is a coming back to the question earlier about what plugins do. I feel like what we should have here is a plugin that tells us how to that tells um uh temporal what to ignore, but I don't think we do have be really great if there's a graph plugin I didn't know about, but uh it doesn't seem to be. Okay, so I think that everyone in the company's assumption that this just worked uh is wrong and we need to go fix some stuff. Uh which is weird because uh I'm sure we've had people trying it, but apparently I haven't. So I will I'll ask someone later and I'll get back to you on that

But like the principle is that because the graph is deterministic, you can put what activities in nodes and um I suppose that the actual point is that UU ID is not deterministic. it is by definition generates a random ID and so they're right to have like pointed to an error because technically you've done something nondeterministic by calling the UU ID method um but I'm sure we can work around that uh sorry that was not as successful as I had hoped um but we will we will get it to work um uh what was the were we going to go on to we were going on to MCP or to eval We were going on to talk about MCP one. >> Okay. Um so you will see if you look in the actually I'm going to run a different example here. Um, >> sorry. >> Examples are also in the same GitHub location or >> the so the the actual the graph examples that are complete are in in the docs as I was showing you in the docs. The one I just made up with temporal that didn't work is currently where I just wrote it. I will get it working and get an example added to that rep soon

Thank you. Um so let me um so this this um example here I should be able to on and we should be able to come over here and look at Okay. And if I ask it a question, I'll just check it's all working. And then um it's definitely not working. Uh no I I think I am here we are. Is it that the MCP server is not running? Um, let me turn off the MCP server for a minute and show you the the outer part of what we're doing and then I will um go into some more details in a moment. So when people are building um with uh podanti or other agent frameworks, what you don't want to have to go and do is like all of the work to implement the um front end yourself and the protocol for communicating between the end and the back end. And so there are two protocols now uh which allow you to to basically uh effectively uh use them instead of to implement that yourself

They are AGUI which is supposed to be this like open protocol for for UIs. The other one is Vel's AI protocol. Um Vel's AI protocol I think is probably more widely used and comes with a nice bunch of react components for building UIs um from from the Vel guys. And so that is what I am using here. So the idea of this application which will soon we'll have a a pre-built well we already have a pre-built bundle. We'll soon have this like shipped with pyantici for basically building well one for for like running a chat UI very quickly but also as a as a kickoff point to go and build full applications using their components is uh a chat GPT style interface for um interacting with an LLM. So, with a bit of luck, this really should be responding to me, but it is not, and I'm not sure why. Um, and then I'm going to add an MCP server to it in a minute, but I need to work out what the hell it's doing wrong

Um, if this does if this example doesn't work, I can show you a much simpler MCP example, but it would be very nice to show you this as well. Um, I think I might be just I haven't set the Let me set this. Let's see if that gets things working. So once I get this actually working, I'll show you what what we can do. But I don't want to waste your time if I'm going to have endless issues getting it to work. We one of the things we definitely need to do here is like return the error to the user if things don't work. Okay. And so this gives us the I can zoom in a bit the relatively nice interface for for interacting with LM

This should have said um it in fact didn't say it was clawed but if it wanted to it could have done and if I try chat GPT or GPT5 it should say we can we can switch models and we should be able to GPT5 is incredibly slow once it pulls its finger out and actually >> there's no orchestrator in this case so we're not we're not doing anything durable at this point Um yeah, >> I don't have any metadata about my exact model. Um um but what is what is neat about this example is we can come over here to the agent. And so this this um is a is a very simple agent that that in fact has nothing registered with it. It just has a system prompt about about how it should answer. Um, oh, and it has one tool registered for fetch, which I'll show you why that's relevant in a minute, but we're able to augment it and make it much more powerful by registering an MCP server with it. So, this is an MCP server running on fast MCP. Um, but you can see the code for that over here, and it's um uh it's basically this is the this is the actual MCP server. So, it has a bunch of FAQs about our Pi AI

So I first gave this last week at our first Pi AI meetup and the idea is this has a bunch of FAQs about the meetup and it exposes a search tool which lets you go and search that and get back the most relevant uh FAQs about the so um pyantic AI going on in here at all. It's just fast MCP. You could also use uh pyanski within a server if you wanted to but we don't have to on this occasion. And we've used this BM25 package for doing the like ranking of our FAQs. This is obviously not a scalable solution for for a big database, but it means you have a self-contained example that should just work. Um, and we, as I say, we come over here and the only thing we need to do, this one line of code here, we can get rid of when I was running it locally. One line of code to a tool set the MCP server over HTTP. And now our agent gets that that um FAQs tool exposed to it

And now it can go answer questions about um the events in the series effectively. So if I come back over to the UI, make sure I've restarted new vehicorn. Um start a new conversation. When is the event in New York? Oh, I'm using GPT5, so it's going to take half an hour. But um once that gets around Oh, got there rather quickly. You see it's now called the the search FAQs tool. Um you can see the input to the tool. So that the the parameters it passed to that particular tool, the output value

Um and it's now it's called it again. Um it's now decided to go and call fetch on the luma link that it got from the response. Now it's going to think for a bit and hopefully it will at some point spit out um some explanation. I hope if we look at um in Logfire what it's doing that is the wrong Logfire project. Um, okay. And it has indeed now come back and responded and told us when the next event is and given us the link to open. Um, so for the most part from the Pyantic AI side, if you're running uh Pyantic AI as an MCP client, it's incredibly simple. You just add this one line of code to your um agent definition and now you have an MCP server and you can add multiple MCP servers and we have some functionality for adding MCP servers and changing the prefix of tools if you have two different MCP servers and you don't want them to to conflict

There are some more fancy things we can do with MCP like sampling which I often use as a demo example mostly because it's the kind of most egregious example of what you might try to do in terms of tracing. So uh going to look in this this particular I mean is this interesting or I mean I can move on to something else if if if you see it. I can also show you how to run the example that I just just showed you because I know I skipped past it quite quickly. Do you want to >> you want to to see how to run that example? Okay, so it's a different repo unfortunately. We should definitely build it into um Pantic but into the demo that you've seen. But if you go to our uh uh GitHub or you search for chat, you will see this repo called AI chat UI. And then I actually think that the interesting code is in sorry wrong branch in this branch here. So I'll put this link here into uh the gist if anyone wants it

And obviously I will at some point probably delete the anthropic keys or just delete the key itself but I will leave this around. Um but that link is is in there. But the principle should be clone this repo switch into the agent directory which is the so most of this is dedicated to basically building that is a react app for building that react UI you saw. Um, but in the agent code, we have the code for running uh running that UI. And so the actual logic for turning an agent into an endpoint that you can use within a within a React component like that is just just this code here. So we just this is a fast API server. We just add one endpoint API/ chat. Um, we actually added a few more for like configuring the the drop down of choice of different models

Um and um we're um yeah but effectively this for sale AI adapter is passed the agent the incoming fast API request the dependencies relevant here the model we're going to choose the built-in tools if we want it and and that's enough and that will basically stream out from the request the uh SSE events that that the Vel library on the front end is expected to to receive. Um, make sense. Happy to get that running. And then the other thing to say is the the nice thing here is you don't have to deal at all with um building the React app at all. We built that for you and served it up over JS Deliver. So if you want to get the the front end running, you just uh basically get that one one response and return that as as uh from the root of your app. And now you're you basically get the React app running for you. And so this will allow us to basically distribute that chat application as a Python package without having to make you load and do a React build or any of any of that stuff

Does that make make sense? So the actual MCP bit of this is relatively simple here where we're adding it where we actually have the agent but it's the it's a bit of there's a there's a bit of >> sorry >> if you want to run this server you should just clone that repo that I showed you switch to the the branch that I that I put in the link and then you should just be able to go into the agent directory and run follow these instructions just run this command here basically um go into agent u sync is technically redundant and then run it and again this this will be set up with multiple different models so I think that is actually not true as of this branch because I've used I think gateway in all of the examples um >> yeah yeah the same padans gateway key I think should work because if you look in the server I'm already I've already set it up to use uh gateway in all three of the model choices. Um so should I would it be helpful to show the sampling example of MCP? Um so the term sampling is perhaps the worst choice of technical name for for for any anything in MCP possibly anything ever. It does not refer to sampling as anyone would normally refer to sampling. It mean it is the idea of effectively proxying your LLM requests back through the client. U and the idea is that if I have an MCP server which needs to use an LLM, I don't want to have to provision that server with my own account on gateway or on on a foundational model. um the client who's connecting they should be the person who is who's going to pay and you can imagine that in the case if I run some FCP server in the cloud somewhere to do some useful task it needs to do AI but I don't want to give my inference the end user connecting can do it um uh or in a large corporation we may have many MCP servers and we would but we would like to attribute cost to the person using them not to the not to the team who built them. Um the problem really with sampling at the moment is that very few clients support it and it is seen as a kind of dangerous thing that you have to switch on. Well I think I don't think it is seen as dangerous and so most servers don't implement it and so it's a vicious cycle in this case we have control and the server we can go and go and implement it

So this is a very simple example of a pinant AI agent. Um, it runs uh GPT4.1 mini. I use mini because it's fast and the thing it's going to do is going to otherwise be relatively slow. And I'm going to register one tool with it again via the MCP server, but this MCP server runs on standard IO. So effectively run a runs locally as a subprocess. Question. >> Is part of the reason you're running 4.10 mini also to do with cost? No, it's literally because when I give people demos of this, I don't want to wait for a minute and a half for it to generate a picture of a pink robot. And um um and so I'm just going to ask the agent uh create an image of a robot um in a punk style

It should be pink. But the server that it is connected to this uh generate SVG server. This this is a pyantic uh sorry this is an MCP server but it has inside it inside the tool call to generate image we call another pyantic AI agent in this case. But the clever bit uh is that the model we pass when we run the agent is not the name of a model or an instance of one of our normal models. It is an instance of MCP sampling model which takes as its argument the MCP session context and now we can effectively use that as a model implementation within Pyance AI to to do inference but actually it's doing all the work of proxying that back through the back through the client um and then it will return it'll print out in fact in this case the path to the to the robot image and return that path and then it should turn. Um, make sense. We're doing something slightly complicated. Should I run it or should I answer some questions? Questions

>> Make sure I understand. So that I can use that basically the user can put in their own credentials for whatever it >> Yes. But they don't put in credentials and the credentials don't get sent to the server. instead the >> yeah I mean like the session their session essentially is the thing that gets used >> their their MCP their connection right and and in particular padantic AI is taking care here you see we've set we've called set sampling model here and that is what has a enabled sampling and b said uh use the same model in this case to do the to so what what we could do if we wanted was we I think this takes yeah we could say use 4.1 here but use whatever I'm going to use just uh sorry 4.1 mini for the sampling but but 41 for the main agent. >> Oh, okay. >> Um and so all all that it we're really doing is we're using the birectional communication that MCP provides to to route our LLM request back through the client. >> Okay. >> Um >> so what what is the reason you chose them to like separately set >> be able to have a choice? Yeah, I mean mostly you don't need to

Maybe I'll undo that. Mostly we just just calling we we don't turn on sampling by default because people see it as a security risk. So you just have to call set sampling model basically enable sampling. >> Turning it on >> turning it on in this case. Yeah. >> But you could turn it on with a different model if you wanted to. >> Okay. Um uh but but I think that the yeah I mean the the the cool bit in my opinion is is this bit here where you're able to use pantic AI but use basically sampling as your as your way of um doing inference

Um obviously in this case it's pantic AI connecting to pantic AI but because this is an open protocol that doesn't have to be one of those things can be paid as long as the other one supports sampling it will work. For the most part today though, the downside of this is that right now if I go and set up sampling on my server and then as someone connects with cursor or with claw code or virtually anything else, it'll you'll just get back an error being like sampling is not supported. So you for you need to it's probably fine in a corporate environment where you're like guys we all need to inst enable sampling because that's going to be our method for for doing things. But if you're going to go and run a server in the you know the logfire MCP server if we ran it hosted we couldn't uh we couldn't say you know it doesn't do any inference inside the model inside the server but if it did we couldn't try and use sampling because almost everyone's client will refuse it but I know that that fast MCP how many people are aware of the confusion around what fast MCP is and the two different names >> I mean I'm aware there is one I don't I couldn't describe it to you >> okay fast MCP was built by my friend Jeremiah who confusingly runs prefect who are an orchestration platform. He got very excited about MCP the week it came out built fast MCP which is a open source library for building MCP servers. Uh a month later David uh Pere who is the MCP creator thought this better abstraction for doing for defining MCP servers in Python we'll pull that fast MCP into the main MCP package then didn't really maintain it that much. So then two months later Jeremiah got annoyed by constantly having these issues on his abandoned repo being like please can you like restart this project. So he then restarted fast MCP

So now there is fast MCP1 that is inside the MCP SDK and there is uh fast MCP2 which is a separate package and I am using a mixture of the two without telling you which one is which because I haven't got around to it. Um, and there's now fast MCP cloud, which is a hosting provider built by Prefect. Um, who are about to change their name because they do more MCP than Prefect stuff now. >> Does that make sense? Everyone clear? >> It's an exercise left to the reader. >> Yeah. Yeah. I'll leave that as an exercise to the reader and I will try and run this sampling example and and see how we get on. Um, what was the name of the library directory? Uh so I'm going to run the client

So the point is that because the client because of this standard IO standard IO is another confusing name from within MCP, it will communicate over standard IO. But what it actually more importantly means is it will run that server as a subprocess within when making the request. um and it will then you see so we're making the second request to uh GPT4.1 and if we come over and look and on this one I will promise you that it's easier to understand with logfire involved um although it is relatively complex what's going on you see they're relatively similar similar colors but we we're we have distributed tracing going back and forth between the client and the server but we started in the client we got a tools list from the server. Then we called GPD4.1 here saying you've got this one tool available to you. Um uh image generator. It chose to call image generator. Then we did uh on the client we did um we made the tool call. The client called the MCP server within that tool

Um, which then called Pantic AI again, which called back to the client, went back to the server and then back to the client and then eventually spat out somewhere uh at least the link the name of a file that we should now be able to open. We may or may not get a pretty image of a robot. Oh, it's a really boring image of a robot. >> Um, >> but if we we'll we'll do this. I did this the other day and it went actually did much much worse. But, uh, I'll try and do I can't spell anthropic. Um, >> it doesn't want Claude Claude instant one. It's suggesting for me sue if you create a company called Anthropic, right? Why? Um, create a >> I spelled that right

>> Start a company close. >> That's just a Did you see that? Did you see >> it open source? >> That's right. [laughter] >> Samuck. >> Did you see um Gary Marcus said yesterday when the duster settles we'll all refer to him as Scam Alman? I thought that was like >> it's pretty good. >> To be clear, before anyone sues me, I'm not claiming that. I'm just reporting the news. >> Uh not financial advice. >> I'm just a messenger

>> What's the I can't remember. Oh, I've done the >> Uncle Sam. >> Let me um Okay, so it will now run should do the same thing, but with a bit of luck, we'll generate a more interesting image of a of a pink robot. Um but more to the point I hope demonstrate that sampling is if not a good idea then at least an idea. >> Thank you for this. I never seen something working ever in my life and >> I don't know is there anything supporting it. >> I I think I mean I think it's a one of truthfully one of the reasons I demonstrate it is it's a it's a cool demonstration of distributed tracing working back and forth between the client and the server not because it's actually something people are going to like go off and use that much. Um, >> I mean it's it's a it's a but again it's one of those things if it's not supported no one will use it and then no one will support it

>> Moment of truth. >> Bit better picture of a robot. >> That's the evil robot. >> It's It's pink and it's kind of cyber. >> The other one was >> except for this one is a line. The other one's kind of off center. >> Yeah. I mean >> um actually >> but the head is not attached to the body

>> Well, there's a bow tie. You just can't see them. >> It's got a like it's got a like some oval bow tie. >> The robot. >> Suddenly I've discovered something you're all interested in and have an opinion on how this is what it took. >> That's the real reason you >> um one of the other places I I have another example uh coming back to the SQL generation question that we had earlier. Um, one of the other, it doesn't actually require MCP at all and it definitely doesn't require sampling, but like one of the things I've seen work quite well is that if you have a tool that basically uh effectively goes off and makes a SQL query, you can put all of the context about the SQL schema and how to write the SQL and the error handling and how to deal with that into your main agent which your main agent particularly if it has lots of other tools or you can have one tool which basically says takes as its argument human description of the data that I want and then it goes and generates a SQL. It does the the loop of like fixing the error and then it returns the results and so you basically hand off the work of generating the SQL to some um sub agent that only that's the only thing it does

I think I have an example of that somewhere but it's a it's a it's a neater way of I don't actually know if that's in this repo. Um I think it's not but uh I can show that one as well but like it's mostly neat not so much for the FCP stuff although I I used it with sampling but because it shows this like basically delegation of respons to an agent without with needs less that's doing one specific thing and has lots of context on that. Um should I show that or should we do some emails? >> Show that. >> Okay I'm showing that. Um, so I am going to we have this like mess of different repos where I have at some point dumped code in a hurry when I'm about to give a talk. Um, oh, that's really not what I want at all. Okay. And I'm looking for AI engineer

Um, >> do you use code? >> Yes, I use claude code a lot. Um, an awful lot of the UI for for gateway was built with with Claude. And in fact, quite a lot of the back end was built with Claude as well as in I built a little micro web framework that allowed it to basically forced it to generate type safe endpoints and then it would just go and spit out those endpoints and write the SQL very efficiently. But I think I'm liking to spend spend some time thinking and then generate your prompt like think hard about it. Not the like other end of the spectrum which is just like prompting cursor continually to try and change something as in spend time on your claude code implementation kind of take >> 90% of your time. >> Yeah, exactly. I like the idea of calling it that's that makes it sound like a much more sophisticated. I mean, especially if you have a British accent, like if you know it

>> Um, >> how would you define agentic? Because a lot of the stuff that the lab doing already like >> everyone else here has to listen to my definition of an agent joke again, but since you asked um >> um so there are three definitions of what agent means. Uh there's the AI definition which is um an LLM calling tools in a loop. That is the like Barry Zang definition. I have some slides here that will probably show that if I uh let me do this. Uh that's not what I want. Um anyway the the the high level idea is um that's the definition the like um code calling tools in a loop. The second definition is the kind of infra definition which is an agent is basically a micros service. >> And then the business definition is something that can replace a human and take their salary

Um and when I talk about an agent here I talk about the AI definition of code calling tools in the loop. But I'm aware that there are definitely multiple other definitions. >> So it's like the the model you're calling will automatically search the web if it doesn't know. >> Yes, >> it's like an agent calling an agent. But >> yeah, let me just try really quickly and show you. It's too much. Yeah. So this this code is taken almost exactly from um so this is Barry Zang's talk from the beginning of the year

Um if I just open one of these. So this was this was his definition of an of an agent from his talk at PI um at AI engineer in February which doesn't make any sense to me at all. This was the code he had on the next slide which didn't make sense to me. This is his like pseudo code for what an agent is. And what you see and what turns out to be half the problem is that there's a bug which is that the loop never ends. And of course the real hard bit of this is when do you end the loop? That's the that's often the tricky bit of like when do I get to like if the model returns text is that the end or is that just it commenting on what it's doing along the way and that's one one of the like that's the probably the most opinionated and therefore bit that don't people don't always agree with in pantic. So we have like it's easy if you have structured outputs because whatever structured output mechanism you're using if they reply with structured output you're like okay it's finished but other than that if it's a pure text uh agent it's not always clear what the definition is. But anyway, let me go back to Did I? So, yes

So, this is the um this is a um pantic AI agent for basically researching um Python packages. Um it's a nice subject for me because we have a lot of downloads. Um and so what it has one you would imagine in the real world it would have lots of tools registered for like looking up repos or looking at number of stars. In this case, it has one tool registered which allows it to go and look at the number of downloads um or indeed query data about downloads from Pippi. And so what you will see when you look at the the tool which is defined in this case as an MCP tool is Jesus. When we come down here, this is what I mean. The the No, that's not what I mean at all. When we Where am I looking? Okay, so this is our MCP server and you see that it takes as its input simply a question

So it does not ask the parent uh agent to do hard work to work out the SQL that it needs to generate to to find the data. It just says what is the question you are trying to answer and it expo and it yeah so so the the parent client has a very easy time because it just asks a question how many times was paid downloaded last year? Um, and then this uh MCP server and in particular the padantic AI model then does the hard yards of turning that query into that that question into a SQL query. And so you see here that it has a whole bunch of context from on the schema of the database um and on some example queries and on how to run those queries. And uh then you'll see we have an output validator in pyanci which um is running and is basically connecting to um bigquery running the query. If we get back an error you'll see we log that with MCP but more to the point we use model retry which is the the prompt point for basically triggering a model to try again giving it the SQL error. And so it can use this to try and work out what to what to do. And I haven't run this code for some time and today most things I run seem to fail. So you get some excitement of me trying to run it

I'll run it once and if it doesn't work we'll we'll move on. But um uh u run and it's called libs client and presumably libs client has in it somewhere uh how many times has pyanced this year? You can see I'm an egotist. Um, and uh, you can see it doing stuff. And if I go and open this repo, you see that it's got to calling. You see how old it is? It's claude 4 called Sonic 4. It's like two months old. Jesus. Um, and you can see the uh SQL that it generated wrapped with a bunch of text

I hope we've managed to successfully chop out. Then it ran the BigQuery query. We have BigQuery instrumented. So you can see um so you can see exactly um well you can see some stuff. The BigQuery instrumentation isn't very good because it's Google. Um, and then you can see it comes back with a total number of downloads, uh, 4.1 billion. >> That's an incredibly impressive number of downloads. >> See, I chose this example

>> Um, >> um, >> you wrote a tool that gives you Congratulations. [laughter] >> That's just called an AI these days. Um, >> sorry. >> 4.14. Yeah. Yeah. >> Uh well, what's what's lovely about this is we can go and look at the SQL that it generated >> and the SQL that it generated was select countst star as number of downloads from the correct table where file is the file.p project is pantic and where date is the current date that appears to be correct. So this is part of BigQuery

BigQuery have a table where you can go and run any query including in one of their public data sets that they like pay to query it them but um they and we should be able to see >> lot of libraries >> uh yeah what's crazy is that actually about half the downloads seem to be attributed to being a dependency. the other half of the downloads it's very hard to know what I mean what they don't have is like if you transitive dependency what are you a transitive dependency of your transitive dependency but um yeah we should be able to see if we look at the top level run of the agent um yes so the the raw output from the tool again we use format as XML when we returned the like raw rows because obviously the query could have been something more complicated not just a single number if we count the zeros. That looks like 4.14. It looks like it got it right. >> Yeah. >> The thing to remember about these downloads, it's like every time anyone does a pip install, it's incrementing one, right? That includes every like GitHub CI run. Yeah. Right

Uh so like that number of users. No, no, no, no. Every time somebody has hit the end point. >> You mean if you look at the downloads, right? I mean, this is like >> it's still it's still amazing to be clear. Like it's still huge. >> If you look at Pyantics, Yeah. I mean, if you look like a normal like agent and just >> Yeah. I mean, more I mean, that's right because you can see this is like moved number 25 to number 20 in terms of most downloaded package this year

So, it's like it's ballooned it relative popularity. That is like Python >> continues on the current trajectory. something is gonna happen on earth. >> One trillion downloads um fastest to one trillion. >> Yeah, basically >> because uh because this is a pant. How would you take this output and massage or force it to just return as an int that number? >> If I I mean, should we try? Um >> that's what I was asking. I was like this is extremely cool and then could you make it a deeper process by saying all I want out of here is the end. >> Um, yep

Um, if I'm going to say on the agent output type int. >> Interesting. >> Uh, oh, you're going up. >> Uh, [laughter] your children depend on it. I have no idea whether case matters but like it's clearer to me >> like at least it used to mark down also. >> There we are. >> Hey, that's very okay. And all you had to do was add the That's >> Yeah

So if you look if you look in in what we actually did here if you look at the so there were two calls to uh the model and you'll see here that it had one tool register with it uh final result um and it was told it had to call um well there was another tool from MCP but like the only the local tool is this one and that so this schema it was told it had to return call call one of the tools and this was one of them and so it's basically forced to return an integer and then we'll run pyantic validation over the result. So even if the schema JSON schema does not strictly define everything you might want as in you might say this number needs to be greater than zero or you might say this output needs to be a set or something like that and that's where pyantic comes in. Also they they can they don't always get they don't always even match the JSON schema unless strict um which is makes things slower. >> Cool. It has to go and validate against the schema. No, because it uses the way that models do um strict structured output is custom grammarss where you basically adjust the um the log the weights of each token at each position to enforce JSON schema. And so the model has to has to return an integer has to return a decimal value in and you know the the the schema for this part the the grammar for this one would be very simple because it's just like every every value has to be a decimal but you can imagine that the schema for like the the grammar setup for some very complex like uh JSON object gets pretty pretty hellish. >> It almost sounds like you can write a regular expression

So that so so open AI GPT5's custom grammarss allow you to literally pass a JSON a reax of what it's allowed >> and you can see the weight or you can't see the wave. >> So if you run it locally you can use something like outlines which is a txt project to basically do this on local models >> if it's um if it's openi then you can't see it it's like deep inside their stuff. >> Yeah. >> Anthropic announced support for custom grammarss but they only support JSON on yesterday day before yesterday. So it's like >> so how it seems like it's no longer just >> uh I mean and of course sorry >> translation between what is >> it can still hallucinate though right >> yeah you can still I mean still need podantic validation one because you might put you might put a field function like a a field validator like the one I showed earlier of like must be in the 19th century you can't enforce that as a grammar or if you could it's going to be tooated to do um uh and also there are numerous types you might want out like a set or a tupil that you literally can't get from or a path that you can't get from JSON so you have to use padantic to do the upcasting bit >> why can't you get it >> well there's there's No. Um, >> yeah, there's no JSON type for for a date, right? So, I want a date back. What do I do? Like, I could have technically even if I had a perfect enforcement of the schema, I'll get back a ISO A601 string of the date >> in JSON. [snorts] >> I see that XML

So my the in this particular case it's it's not necessary but if you have some like very complex uh object that you're want to pass to an LLM my theory is and I've you know other people have said this I'm not going to say I've run some compl and found this but like how do I know as a model that I've got to the end of this this chunk of text it's a heck of a lot easier if I see a closing XML bracket than if I just see a like I have to count the number of like double quotes I'm in or count the number of closed brackets to work out which which part of some object I'm in. So my impression is that XML works pretty also. You don't have to with XML for the R like format as XML is like we should probably call it XMLish as in it does a little bit of work to like not allow you to escape uh XML within it but it doesn't go and convert every um every value inside the the text to correct XML types which makes it easier for the model to understand. Whereas if you have JSON and you have lots of double quotes and it's now got all these backslashes and it's got like it's got you know does does uh JSON support uh depends what JSON encoder you right but now suddenly my emojis are now becoming like whatever whatever the syntax is in uh JSON for you know back slashu four digits stuff like that confuses the model so I find those things are presumably easier with XMLish stuff like this but I don't pretend to have done any scientific research that is better. >> No, because it's people started talking about it two days ago. I'm really hoping it goes away. Um I spent a year and a half building a very very fast JSON paser inside pyantic which is does SIMD and does everything else and it's probably the fastest JSON paraser in in Rust and in Python. I really don't want to have to go and do that for some other format that's going to last a week

>> So we'll see. >> Makes a lot of sense, right? Because if you like returning a table in this format, it's ballooning your context and all the >> but what people miss is that that that tables are the worst thing possible for for models. >> So so like I have this example where this is a complete distraction but I think relevant point to make. Take a uh markdown table and then imagine how a model sees it. not as a nice grid on the table on the on the page that you can view as an image, but instead as everything spread out as one long line and try and work out what value over there relates to what column. It's completely impossible. So, what's tune? What's JSON doing? Well, it's it's basically giving you a like list of dict. And now I know what the key is and what the value is

So, all of those extra tokens that tune is getting rid of are incredibly valuable for making the model good. >> And that plus the fact the models are trained on JSON, I I I cannot believe that it's actually better in most cases. Sure, it's fewer tokens, but like who cares about the number of tokens for the most part? >> Yeah. Actually, like when you're when you work for like scale or these other human in the loop or reinforce human learning that like the requirement is that use markdown or XML because it's the way that the the model then ingest the positive the positive example. And so like I I fundamentally agree that like yeah you might save a few chunks but then you might keep multiple chunks, right? And so like when someone says like I work at a fortune 500 company and my team of seven like when our stuff went down like no one could buy a few comments and I was the only engineer for two years I was like using AI and no one on my team knew and I would get comments like great code and then they all the seven other engineers all told me and had an extreme argument that like I can't write this code but because you don't know how to interact with the model like the data is important when you're creating the process is important but then ultimately it's like how you interact with >> um should we talk about evals or anything else on MCP that anyone wants to see? >> Okay, let's do a break. That sounds like a great idea. Um check if my evals example actually works first. Great

[laughter] >> Eval your emails. Yeah, [snorts] >> I got to run this code but this is not responding back. I see. >> Yeah, I saw this as well and we need to fix why it's not behaving. Have you Where are you excuse me running that server? Okay. Um and have you set the um pantic the pilantic gateway API key? >> Yes, it took it from there. So it asked me, right? >> And it's set. >> It asked, right? >> No, that's for log fire

So have you set the pyanti gateway one? >> Oh, that one. >> Yeah. So you will need to set that one as well because I'm afraid this this comment is out of out of date. So if you if I come over here and I >> just export the same. >> Yeah. This one maybe will work. >> Yeah, I'm afraid it's really unclear that it's not erroring. Oh, sorry

>> If you reload, if you just try >> Yeah, >> that should now be working. >> You won't have seen that. Yeah. Sorry about that. >> Oh, no problem. >> Thank you so [snorts] much. >> All right. >> [snorts] >> So when I ask him does not have the access to data and so I mean how we can extend it get that real time data

I should be able to turn on web search here. >> Okay. And it's >> but also yeah that should that should be the simplest way of giving it or you could register tools as we did before with tailia as I showed you like as in any any form of way adding tools to give it some context. >> Okay. And if I if I want to integrate it with some of the some of the MCP server through P that's related to work and I want to get that data to get to the >> so with we have an example in the code of of registering one MCP server was that PI AI one um you should also be able to I think we have examples of how to do MCP servers with or so if you go into the agent directory yeah and then into the chatbot directory >> basically >> and then if you Look at agent.py. You'll see this has one MCP server registered with it here which is this this one here which was I built just to give information about our those meetups. So if you ask it about like when's the next PI meetup in New York it should be able to answer. >> Oh okay

So similarly I can just >> register more servers. >> Yeah. Or or you can register tools as we've shown earlier today like it has one tool here to fetch. So if you asked it like what's the contents of example.com it should be able to fetch that. But you would obviously add more tools to give it more context of the real world. >> Which one is already here? >> This one here. You see, we've said agents. And we've registered one tool called fetch which will get a URL

>> Okay. >> And return. >> Okay. >> The text of that. >> Okay. >> Yeah. >> Thank you. [snorts] >> 3.1 I think

No, seven. Whatever the latest >> seven, >> something like seven from Pansky. I mean, turns out that like some people they're not that different. like none of the it's weird to say but they basically it doesn't matter that much because like much smaller nuances than the like differences of what models what the different LLM can support or what the different model providers >> um >> do you do you use any conditional logic >> how do you mean >> in your data What do you mean by conditional logic? >> Uh the um so like in draft 7 they introduce like you can have conditional logic like if a do >> I think not >> I think we we we try our best to support everything that um that uh that is needed to define pantic models. >> Yeah. as in but the the main places originally the main place it was used was it by fast API to generate the open API schema and now the main place it's used is in like stuff to LLMs >> okay >> and I don't hear about the draft logic much so Sorry, one minute. >> Sorry. uh in if you're in that are you on the branch I yes you are so if you look here go down a bit I think that is >> oh I tell you if you go to my GitHub >> just start >> no if you go to go to GitHub and go to Samuel Culvin if you go to just go to a browser and open github.com no github.com/ / Samuel Covin

So that's that's gist. Yeah. And you go to repos repos. >> Yeah, it's uh this one here. P AI MCP. >> All right. [clears throat] Sorry, >> this is the eval running. So, I've got something to show you because they take quite a long time to run

>> H >> which GitHub repo is >> uh this is in the same repo there literally next to the 20 questions example. I don't know why that API key has got disabled. Maybe we blew through our credit. I'll try running it without for a minute. on on which one >> is. >> Yeah, it's just the main branch. Shall I carry on on evals? Is it >> working? >> It's running. There's some stuff to see

I'm not saying it's all perfect. >> Um, so this is um uh roughly well this is actually not this is importing from the 20 questions uh code that we started running at the very beginning of the day. Uh and then this is building an eval to see how different models perform at playing 20 questions. There's a bug which we will come to but uh that's kind of secondary to the point here. This is using the um pyantic evals library which as you will have heard one of the things we care about as a company is type safety and it tries to be a type safe library for building evals. So at a high level this is trying to be piest for stochastic functions. So we have some integration with logfire in the form of we can emit open telemetry data but we are not you can view the output locally. Um this this is running now and you can see earlier where it didn't go wrong

um you can see the performance of the evals locally in your terminal and obviously if you don't have logfire set up you can you can view the the output or subversion of the output locally at the same time all of the data we're emitting is open telemetry so in theory you can send it into any platform but it won't look nice but the the main point here is that um we care about this idea of building building a type- safe way of defining our cases and our data sets um so uh a data set is if if you take the piest paradigm, a data set is like all of my tests or a file of tests. A case is like an individual test and then an experiment is the equivalent of like running my tests. So a run um so we define our data set here with um some cases. We give them a name which is obviously always a string. we give them an input which is a generic which needs to match the the the input generic on the data set. So we end up with this type safe definition and then we can also add evaluators to uh our data set. We can also add evaluators if we want uh to an individual case if we so wish. I haven't done that in this case

And we can add multiple different evaluators. And those evaluators can be either return booleans which we we display as assertions or they can display uh uh floats which we display as metrics or they can return a dictionary of keys which we use as the name and then values which become which can either be booleans or floats. Um in this case we have two um uh evaluators. uh these are called question count which takes the uh play result which is what what's effectively I don't know what we call this this generic but is the um the output sorry the result type um so they they get basically the result type they can look at context output which is the output of running our the function we're evaluating and they return based on that some kind of like assertion or metric we have built-in evaluators for things like LLM as a judge. Um, but you can also register your own ones, which is what I've done in this case. And you can build, of course, your own LLM as a judge. That's just an implementation of the evaluator type. We're just inheriting from from here

In this case, we've used a data class. Um, but we don't we don't necessarily need to. But like yeah, you can see there's no we could technically define our evaluators which have their own parameters which I guess like become the rubric for how we um would become for example the rubric in an LLM as a judge. We don't have anything in this case. Um and yeah we have play result which is the result what we get from our stoastic function. Um, and so fundamentally what we end up doing is we need a function that is async that we're going to call and that's all we're really evaluating. That function can be to do with LLMs, it can be not to do with LLMs, it can be used AI, it can be using lang or any other AI library of your choice. Um, fundamentally it's just something back from the the only kind of regard in which we're so mostly we just treat that as a black box

we it takes inputs it gives us back outputs we have some type safe way of defining evaluators there there are a few exceptions to that like blackbox model which is that if you come over to the eval documentation um if you look at evaluators um you look at built-in evaluators we have some that basically um you can you can define evaluators based on on the spans from the open telemetry generated during calling that function. So you can look and go and say was this particular tool called which you might want to do. Basically a way of like inspecting within a tool what it did and obviously stuff like performance and stuff like that which we'll we'll always measure time and tokens those get measured by automatically as metrics but we can also add some more that make make some sense. Um you'll see this has been running now for some time and it is running in a loop endlessly for some reason. Um uh oh well it's running over all the models. So that makes sense. >> Okay. So it does make sense what it's doing now

Um so yeah so we have our in this case we we've have one wrapper function around our actual play function which is in particular it's you know it's it's slightly ugly if it raises an exception. It doesn't actually break the eval library but it's kind of cleaner if we can return in the case of reaching the limit uh that we had 25 steps rather than just exception. Um and then um oh yeah we return we return some basically data about the messages exchanged which are useful. So this is our a little wrapper function around the like simple play method that is the kind of original core to everything. Um, and then yeah, we're going to run that full 20 questions game over, in this case, five different models, which is why it's uh continuing to run. And so you see each time it finishes. Oh, and it has now finished. And it so it's run effectively five experiments

Um, and we can see the outputs here. So we can see the question count. So, so the other thing to say is our cases here were a bunch of different objects that we were going to go and the input here is effectively the like secret that that the model is trying to find. Um, and the name is just our human readable name which in this case make are the same in general. They probably wouldn't be the same thing. Um, and so we're going to run each of our cases for each of our models in this case and we'll generate five experiments and within each experiment we should have however many six different cases that we ran. uh pyantic evals will automatically take care of running these in pal in parallel. So when we call evaluate it will go and run all of the cases in parallel

They're all we assume that this is async. Um the only other thing to say here is that like we have this concept of override which is just which is a pyantic AI concept which allows us to basically override the model that is used within an agent without having to pass that all the way through. So it's effectively it's it's a context manager that lets us customize the model for particularly for cases like unit testing or evals where we don't want to have to add hook points throughout our code to pass the model around. We just want to be able to overwrite it for for running evals. And so these have now finished. We get some description of what's happened here. You can see that it got uh it took 16 questions to get to potato, 12 questions to get to man, seven questions to get to woman. I'm not going to say whether that means anything

Um uh but the the other bit here is to say that obviously in if I can find logfire I've got too many pyantic tabs open. Um we we can come now into the evals tab of logfire and we can see our evals cases. And I apologize, it's going to be incredibly ugly because the, you know, there's no I haven't put a data set name in this case, but um there's no way of getting this on the screen without it being it would look much nicer on my big screen uh or even on a MacBook, let alone on a like prop screen compressed onto a monitor. It's going to look fairly fuzzy and hard to hard to review. But you can see Oh, it's just itself up again. um because of the small screen. Um the first thing to see which is weird so this is this is effectively each experiment that we have run and you first thing you can see which is weird is that Gemini 2.5 flash uh has passed 100% of the time that is a bug in my implementation where if Gemini flash runs out of ideas it just replies and says I don't know uh we've taken that to be success whereas all of the other models faithfully continue until they either succeed or or hit limit. Um, so we probably need some way of like uh and I'm sorry, there's a nasty bug where it jumps to be narrower periodically

>> Oh yeah. >> Um, I guess I think we reload the screen every x seconds and that every time it reloads it snaps. So I apologize for that that bit. Um, and then beyond that you can see perhaps not surprisingly Sonic 45 did best. But the the interesting thing is we can take let's say um sorry about that. um two models and now we can well let's go into one in particular. Well, let's let's now I've got those selected, we can go and look at the the comparative performance of the two different models. So again, this looks incredibly ugly on this tiny screen because there's an awful lot of data to describe, but hopefully you get the idea of the the kind of thing that we're displaying and if you look at it on a big screen, it will be interesting

But you see um the basically graph descri describing the metrics both the metrics we have measured in this case average case duration sorry assertions here but uh case duration is obviously a metric that we're automatically recording you don't have to go and add the evaluator for us to measure measure time and so you can see that sonet 45 was significantly slower than GP4.1 which we might expect um you can see obviously input tokens is much bigger than output tokens, but noticeably fewer input tokens. I guess that's because it's on average taken fewer cases and so hasn't taken as many tokens for. Does that make sense? Doesn't make sense at all. Should be the other way around because uh I think oh well let's look through and see if we can work out what's going on. You can see number of requests which we also record and cost. But the the real point here is to um I'll hide the summary. We'll get into the individual results. And you can see here um the success case they both succeeded or both failed

But here you see a case where they behave differently. Um so one which was set succeeded and two which is 4 GPT 4.1 um failed and we should be able to see so that's assertions we should be able to see over here question count you can see um that GPT 4.1 got to 25 and then failed um and we can see input tokens so yeah and obviously the the Neat thing ultimately is that this is just OTEL data and so we can dive right into the particular span that of spans that represent a particular case and investigate exactly what happened and why it behaved the way it did. And so we can we can see the particular exchange it had with the LLM and where it where it went wrong and dig into exactly what's happened. And this allows us to um yeah both on a on a basically compare two you can compare more than two runs but as you can imagine that works well on a big screen will completely fall over on on this tiny screen but you should get the idea of of how how from basic summary Is it do an evaluation based on? >> Yes. Yeah. So you just you need to effectively you just need to as in this case we change the model as the input. You would need to change the prompt and run multiple different cases with different prompts and see how they perform as in you can either either pass the prompt as a context var or you can pass it as an input to the function. In this case, the input was just um so so the input comes from the uh >> can it be part of the data set? >> Uh the the prompts

>> So uh it could be. Yeah, you could you could define it in the cases as the prompts. No, you wouldn't want to do that because you want the point is you want to change the prompt between different experiments, not between different cases. So there is a way to pass either you can use context vs or you can pass uh there's another argument here um metadata which you can pass through where you can pass the prompt or you can use an override type stuff thing as we've done here um or you could I mean the other simplest way of all would be to um use partial to wrap your function that takes the prompt um and thereby have have a have a simple function here with the right signature. The proper answer to that question is we're about to add managed variables into um into logfire and they will both allow you to manage variables from within the logfire platform but also they'll allow you to basically control managed variables from a context v context file just as we're doing override here. And so you would basically o override when you run a particular experiment with like change my prompt to this. >> That make sense? >> Could you do that programatically? >> Yes. And and of so the the one of the use cases of this uh system will be uh RL style prompt optimization using Jeeper which work the JAR guy to to basically make that library bring prompt optimization into pyanci

Um that's a good that's a good good reaction. Thank you. >> Outside system call evaluator like from outside can say here's the data. >> So I I don't know the exact internal signature of an evaluator. I mean you should be able to to to call them from outside of our evaluate function. For the most part, you you probably don't want to do that. You probably want to call it via evaluate. If you want to put a lot of logic into a function and be able to call it from somewhere else, I would define that function and then call it within an evaluator

>> No, I understand. Even if I call this evaluate from a system that's outside of, you know, >> do you mean do you mean when you say call evaluate, do you mean call like this evaluate the data set method? >> Uh yeah, you could run that code wherever you like. The point is that there are other systems uh where you know so I don't want to run the code in that system. I want to run code here in Py but I want like a it's like a web hook or something you have some data you want hey how this data can evaluate >> you mean as in the the service is running somewhere and you want to evaluate it. So, so obviously I mean your your eval function here here it happens to be a function which runs some local local logic and calls an LM but that that function could be making an making a request some API to go and query the performance of your >> API that can be >> uh there should be no it's Python code there's no reason why you couldn't call evaluate inside some longunning task whether that is a I mean I like there's still work to do on like what is that I think we have somewhere in our docs we say like evals are an emerging art and science and anyone who thinks that they know how it should be done can safely be ignored and I think we're still in the process of working out the best way of doing eval so this is this is an answer to the offline evals improve the performance of my system or get confidence in the performance of my system before I release it the other case of evals is the online evals where you're basically running some code in production now that could of course be staging that could be a C and then running evals against that system that's running and basically building these data sets from um from well in our case from data in logfire that will that is also the the managed variables and the annotation Q stuff required for basically online evals is are two of our big priorities this year as a company. So I think very likely we'll have managed variables by the end of the year. Less certain whether we will queue by the end of the year. >> Use case I was talking about the latter

>> Yeah. Yeah. And I think I know there's been lots of hot air exchanged on Twitter about whether or not evals are a good idea. Swix likes saying how terrible they are. I'm told that is him specifically nagging some person who he doesn't like. But uh rather than actually a particularly profound opinion. My take is everyone who says evals I mean what what the intelligent take seems to be everyone who thinks evals are a bad idea doesn't understand what evals are. Uh and I I mean I kind of get that

At the same time I know lots of like coding platforms have lots of the um coding agents have very few evals. I know speaking to um lovable earlier this year they were like it's all vibes. We have no we have no evals. Um, uh, >> bugs and feedback. >> So, I had amazing amazing. So, I don't know if anyone saw Is this Oh, this is recorded. Um, I'll tell you the story later. [laughter] Um, I'll tell you tell you a related story at some later

Um, where were [laughter] we? >> I do have one. I do have one. So, um, I'm thinking about this stuff a lot. >> So, one of the one of the >> Let everyone know there's coffee upstairs. >> There's coffee upstairs if anyone wants it. >> Nice. Yeah, but we got the coffee hour >> when we needed it before they take it down again. >> Sorry, you were saying >> No, no, yeah, no worries

Um, so one of the one of the problems I've been like I'm going to continue on with this example like here you're basically doing an evaluation on six cases versus five five models. >> Let's just assume that your scenario is something like that. It's figuring out which one of those six model five models is the best. Yep. >> Um the problem with that is like if you were to be like data science rigorously rigorous, you would all of a sudden need to you know run this multiple times >> in order to you know measure statistical significant blah blah blah all of this crap, >> right? >> Um so I guess the first thing is like I struggle doing that because this is where it comes the comments of like okay can I just go into the MTP server and grab all of these evaluations so I can write something that can uh do that. Um >> yeah so I asked the team and they said that there's a logfire.evals table and I tried to query it and I got table does not exist. So uh uh did uh >> live >> uh the next round of funding. >> We may not have the AP endpoint right now but we will do >> but like we can chat about this

We will I mean we we definitely mean to expose it as SQL. We have >> we have some clever stuff that means that in fusion via our database we can query a Postgress table which is where >> evals live. Obviously you can already technically query everything because like I showed here like it ultimately evals all come through as as traces. It's not the nicest view of it but you could get it. >> Yeah, they're there some >> but but the also the EVA evaluations experiments table whatever it's called should be queryable. >> Sorry you say obviously it's a SQL database rather than nonSQL. So logfire's thing is that everything is exposed to SQL and we find that super useful as for solving a whole bunch of problems. So you missed unfortunately I I showed when >> I showed that you use logfire MCP which basically speaks SQL and can go and investigate bugs and and fix them just by like querying our data with SQL and that's where this question came up

>> Yeah. And the the followup to this would be like are you considering doing some sort of experiment that wraps like this logic of around a navy test? will basically have >> are you guessing? >> So, so the the principle of managed variables which I don't want to overhype as an idea I do but I do think that they're really exciting. Mhm. >> Um so they're built on an open protocol called uh open feature which is basically like AB testing open protocol which is not particularly exciting which is not particularly well known because AB testing is not very cool right now because all of the oxygen of everything is taken out by AI but actually the it's a very neat protocol for doing exactly this thing and one of the things it obviously has as a primitive is AB testing so giving some percentage of traffic one one variable versus another or one set of variables versus another. >> Um yeah. >> Yeah. >> I think this comes back to your question of like running it once is not enough. You actually need to go and run it tens of times, maybe even hundreds of times

>> Yeah, that's that's question. >> I was going to say that's a stats question. should be removed. It's just like I'm literally thinking about the things and it depends on your standard deviation. You have a score, you define that score. The score in this particular example, it'll be like six, right? Because you have cases. [snorts] So if you add 100, then you have like a more granular thing, >> but then you want to measure difference between the average, standard deviations and this is a hot mess >> in order for you to say like this is actually better. >> Yeah

If you get to a point, it >> I mean it's precisely that one of the reasons we use this 20 questions example is that it's exactly this kind of like treasure hunt example where it can make itself go wrong very quickly. I one of the cases I had was like was car was the was the result and the third question it asked was is it bigger or smaller than a car and the other model responded either yes or no because that's the only options it had from then on it couldn't never said car because it was like I know it's smaller than a car and so there are situations like that which mess it up which is exactly the kind of thing that's like hard about real world problems and this 20 questions encapsulates exactly this this so slight side note but I think very relev relevant. I went to the hackathon on Saturday arranged by theory ventures and Brian Bischoff who is their head data scientist who is basically old school data scientists and somewhat about everyone saying these things can be solved with AI and so he has spent a year building this like world simulator which generates entire data sets as if you are a large company and so he the beginning of the of the hackathon is basically he gives you 97,000 park files and says go and answer these questions and You then have like 30 questions to go and answer and everyone had to go and build an agent and I failed dismally because I tried to build it with pantic AI. Everyone else those who succeeded basically just pointed claude code at it and said try and use these try and use duct db to answer questions and um apparently the real reason he came up with this was he had seen whilst running theory ventures numerous different oh maybe I'm not supposed to say this. Um we can we clip any bits out of the recording. Uh anyway, no, I think I think this is I think this is presumably reasonably public that he he basically his view was that all of these like BI agent that can investigate your data actually can't solve real stats and or that was a far more complex example. I spent $15 by mistake because I just asked like my AI to run one particular task and in about a minute and a half it spent $20. But like so the the the case was much bigger but it was fundamentally the same problem, right? It's this like treasure hunt of like go and find a file, run that SQL query, then go and look in this other file and try and do a join and etc etc and you may get to an answer and the lots of real world examples come out to be similar to this like treasure hunt style case

>> Isn't that like multi-shot prompting is you like give it a template of like good examples, right? So if you were to abstract that and you were to find like patterns that work, you can give it both the context and then the pattern and you can essentially do it like >> different levels of abstract. >> Yeah. Without speaking for Brian, I think his take is that all the people who claim to have done that haven't actually solved it for real big real world problems. Like the number of examples that they need is is vast, right? >> Yeah. >> Um >> but API is going to fix everything, >> right? But but like that seems like a legitimately very hard problem. It's not like if we te te tether things together correctly like as in the 20 questions should be solvable like everyone in this room could play any of these 20 questions and and get it right I suspect in 20 questions and so um I think look in answer to I think there's a there's a legitimate feature request for pantic evals which is how many times to rerun a case and instead of us showing like house 40 times down this page because I want to run each case 40 times I just say house and it says 40 next to it and now it says a percentage of successes is maybe we can maybe that's already supported and I just haven't found the feature but you sounds like you've tried it harder than I have and it doesn't. >> I mean I haven't looked but I am just basically manually doing that. Another another [clears throat] feature would just be like being able to at least grab this from the state and put it into like an Excel spreadsheet or something

>> Yep. >> Just begin to do the math but right now I cannot even I have to manually grab the numbers. >> Oh Yeah, we can sort that for you soon. >> Okay. But I do believe it makes sense to show multiple cases as in and then and then each of these values basically becomes a like average or a you know >> Yeah. >> like a weighted like affected value. >> Yeah. Yeah

Yeah. >> A funny comment on like the stoasticity uh comment you were like I on the potato one like mine went all the way to question 45. I configured it to go to 50 and then you know like the last 15 questions. Is it a baked potato? Is it a fried potato? [laughter] potato. >> It never actually just hit potato. It was just like it's like mashed potato. No. >> And the reason and sure enough, the reason that I changed it from just bullion was because it wasn't succeeding enough and I made it into this like kind of completely wrong because it got a bit more reliable

But it is a this example is kind of unfair is a good example of how models get things wrong because they get either they start like >> they get an idea in their head and they carry on like a bit like inexperienced people often do or people um anything else I can talk about on on evals or anything else that anyone wants to hear about? I mean I think we're having a really interesting conversation but I mean I know we're not really doing a workshop anymore. We're just having a conversation but maybe that's good. If you had 48 hours to learn about a tool related to emails, where would you go? What would your process be to learn about that? >> My impression is that very few people have done anything genuinely interesting >> uh in evals. Uh so I would probably sleep for quite a lot of that time. That's what I needed. Um I I um I think that online what to call it but like evals against real prod are going to turn out to be the most part of the answer. I think open AI buying stats was basically a signal that they think evals are going to be about put something live and learn about it. I um I have this I have a a very neat technique doing evals that I identified and I was speaking to I won't say the name of the company but a large a unicorn uh legal tech AI startup uh about this exact problem and I described the answer and they all went quiet and took their phone out and noted it down and so basically what you do is if you have a chatbased exchange one of the things you want to know is how good is the is the model's response and you almost already have the email to that which is in the form of the users's follow-up

So you don't eval the result, you look at the user's response and you basically categorize that into like if they say no that's try again, you know, it wasn't very good. If they say thanks so much and then stop the conversation or or just stop the conversation, you're probably right. And so you can use the in that case use the um uh users's response as your eval. uh and there are techniques like that which work very well in the chat interface the chat case but um >> I thought about this too like basically just sentiment of the user response and then the number of >> turns yeah you can probably get a pretty good answer of how it's done and if you're a legal AI startup you can imagine that like that has a chat interface for people that probably works quite well in some cases [laughter] >> I work with a lot of lawyers but something like hey might be best flight to go to the UK. >> Yeah. >> But you know whether you know return something to you >> but but as but the reason the where I got to this idea from and I've seen it work well in a few scenarios is this is how Google works, right? Google's fundamentally ranking of pages is based on they show a page never comes back searches again uh it's seen as a very good if they're back 30 seconds later seconds later you assume it's a bad result and they then derank the page and that's why if you go and look up any recipe online they make you troll all the way through the page all through loads of endless description of the ingredients to actually get the recipe is because they want to keep the time on site nice and high to make their site look like it's more useful and serve more ads >> and serve more ads. Although, interestingly, now they all have a like TLDDR section at the top, which is which has the recipe in it, and that's because AIS want the answer straight away. Um, so I'm told, >> but also because then people are more likely to fowing someone to get the >> you get the get just enough of the recipe, then they're actually going to spend more time on site

>> Yeah. Um but yeah like using the the user's reaction as your eval is probably your best technique. Obviously that is best done online rather than paying for subject matter experts to to go through cases although that is also something people people want. So I have another related question more like generic I'm not sure if you have seen that paper like year ago told about how faithful are red so whenever you're providing the context the model provides a successful result but is it came from the base >> or actually you provided >> I think it's a really hard problem I I talk uh by BlackRock again at one of the AI engineers about how they have these agents and that all that they're trying to do is to get the model to never use its idea of what Apple share price is but only to use like Black Rockck's data on what Apple's share price is when it gives an answer and there is no I spoke to a startup the other day who have these tiny models that they say are like sticking and you can literally debug where from can lots of specific cases like that who knows how well they do because part of the reason that LLM's work is that like all of their other like random knowledge that you know if it was obvious what you were going to take out from a model to make it just good at certain questions people would have done that but the fact is no one knows how to what to remove from its training data to make it just good without ever uh hallucinating. I I don't have an answer uh specifically other than I mean running another LLM guard rail style does this look right probably gets you like gets you an awful long way but doesn't by any means solve the whole problem. Any other questions or anything else that anyone would like me to go through? We are 20 to four. I think we've done done pretty well. Yep

>> What are some of the most >> uh um one of the big model companies will release a new model tomorrow. I think it's more or less an open secret at this point. Uh that will come out. So, it'll be interesting to see. There's lots of like more breathless and annoying people on Twitter saying that will completely change the game. I suspect that a week later they will all be saying the opposite. So I I don't know how true that will really be. >> Uh it's not um uh >> absolutely increase some metric by 20%

>> It will absolutely increase some metric by 20%. Um [laughter] >> I will say though I won't say who them who the company are or what the model is but we got EAP access to it as for pantic AI. We got a message in their Slack being like, "You're such a valued partner. You you're one of the very few people who have had access to this model." And I forget how they put it. They were like, they basically they later on in the message they were like, "So, we're really pleased that you lang chain having access to this model." [laughter] And um they sent us two more messages. They obviously had a a Slack bot set up that had us confused with with Langchain. And then when they gave us access to the model, the the project ID they gave us didn't exist in our so they presumably gave us Langchain's project ID inside their platform. it

>> Uh, I don't think it would have done that much use, but like we obviously weren't entirely the only ones who had access. >> Um, >> stilloring. [laughter] Uh, I think there's there's a there's a like look, MCP was over as as a fan of MCP and definitely like someone who likes David who created it a lot. MCP was overhyped at one point this year. As someone said, basically MCP came out at the same time that LLMs really got good at tool calling. And so it got an awful lot of the like hype around tool calling works and then it got a whole bunch of the push back when people realized that tool calling was not the panacea that someone breathless on Twitter had told them it was. Um, MCP is a like legitimately valuable thing in enterprise. I don't think it's so useful in coding agents, which is the like first place it was used

And so today there's like more people building MCP servers than there are using them. But I do think that long-term it probably is a like boring technology that allows enterprise to connect multiple things together. Um I've seen a few enterprises who are like to use A2A and I think A2A seems to be now that this what's it uh SEP 1692 I think it is came out which is the task for uh MCP uh if I've got that number right. I'm very pleased with myself because I um oh that's why I know it because someone had someone had used our deep that was it um Adam from Prefect had used our deep research example to look up SE 1682 1692 and it had instead come back with some battle that happened in September uh 1692 it's why I remember the number of task basically longunning tasks so basically you can go and kick off a job on an MTP server and then you get back some ID you can go with uh I think stuff like that will end up being the like it won't be the like internet of AI that people sort of thought it might be at one point. I think it will just be a like another boring enterprise piece of technology that's super valuable in how they use AI. >> It's essentially like a machine like agent or LM version of an API standard. >> Yeah. Yeah

>> Yeah. Exactly. And I think that's that's a useful thing, but it's mostly like API useful in like big boring enterprise, not in like it's not going to all agents surf the internet, I don't think. >> Um, >> if you're in this world, I would say I don't know thousands of >> So I think that one of the interesting things Composio are doing a bit of this. I don't think they're doing it quite how I would go about it, but basically you almost want to like if you think about the first time you run something as a bit like uh um so web assembly has three different compilers within it. It has first first interpreter that it starts running the code with when it first gets the web assembly. Then it has second interpreter that it uses to finish the first compiler that it uses to finish off that first run once it's got it running. And then it has a like optimized compiler that gives you fast web assembly later because they care enormously about like the first time you run the code being faster and then the second time being as fast as possible

And I feel like the what we want to get to a point where the first time I run a particular flow of like call this tool then this tool then this tool the LLM goes and does all of the work of working out what tools to call in what what order. But after a bit automatically gets kind of optimized or compiled into like now I have some meta tool that calls these three tools in a row and so that we don't have to go and get the model to figure out every single time how to call each each sequence of tools as in take an example of I connect to stripes MCP server and I say I need to export customer details on all of these customers and work out uh rank them by howable they are and then send them all a Christmas card let's say and it's going to go and run this particular sequence of four different tools over 5,000 customers. The first time it's worth it calling each of those tools in turn. Today, it is probably worth just leaving it like chunking away for a night because I it's still less work than me actually going and doing the hard work to write the code. The moment there were basically two extremes. There's like let the LLM literally figure it out all over like each time all like from scratch or there is like write the code or get the get your coding agent to write the code to do the whole thing. And I think what we will end up with is some middle ground where you have a kind of optimizer in the middle which goes actually this workflow becomes completely deterministic and by calling these three tools >> and I have some ideas about how to do that but I'm not going to talk about them now. >> Do you have a a working example of uh what's been called history processor? Um yes

So on Saturday I was manically trying to build my agent to to process this like world model that failed and I had uh I had a very basic one the basic because one of the problems I kept running into was it kept just like blowing the context and then the whole thing would crash and I would start try it again and so >> um so in here I think I had an example example. I I actually may have deleted it in a hurry in the last 10 minutes of trying to get to something. Um Now I'm failing to find the thing I'm looking for. Uh somewhere I had an example of of basically um uh I can't find it right now. Here we are. Compact method. Um this was the like very brutal version of compaction. Um that would allow me to literally I I mean this this was me writing it in an enormous hurry

So I just generated JSON from the entire set of messages and if they were longer than 200,000 characters. I gave all of that JSON to another model and said make this smaller. Uh that's a dumb way of compaction but it it didn't work. Obviously I didn't win the hackathon but like it kind of started to work. Um if I is that going to work? I I wrote and then did I got uh chatb to write and then didn't use a more complex compaction agent that's here. I haven't tried it, but the idea of here was we put all the messages in as a like big markdown string and then ask it to compact that. Um, but I haven't tried it, so I don't know how well it works. Um, but I mean that's what is is that what the sort of thing you mean? Like basically, >> yeah, this is exactly what I mean

But I was just wondering if the only way to make it work with tool calling is knowing the nitty-gritty details of each one of the LLM providers to make sure that you're matching the tool and the response blah blah blah. >> Yeah, that's problematic. the only way. >> Yeah, because basically if you call a tool, you need if you have a tool call in your history, you need to have a response to that tool call in the history. I think if you do this, which is basically you smash up the whole of your context and you just take everything that's happened in the past and you bang it into one massive this is what happened in the past message, you get around that problem. I don't know the the performance impact of that, but I think that's what like Claude's code compact is fundamentally doing, right? It's just like taking everything and turning it like getting the model to write a summary of it, >> right? It's just for the for the type of for for the type of agents that I've been trying to build like I bump constantly into context limits which is ridiculous because they're pretty big now. Y >> so right now we're kind of like solving it with so agents and just delegation. I think sub aents is a is a great thing anyway because because like if you can if you can isolate a task and just give the context needed for that to that agent it'll do a better job anyway and you can go and debug it better and work out what it's doing wrong and you have a isolated thing you can go and eval or reason about that is like good makes sense anyway but if you still need to do some kind of compaction this this might or might not work or something along these lines I don't think it is I suspect it's not much more sophisticated than take all of the messages and combine them into a big object and pass them to a model and say compact this you might want to be like oh here's an enormous blob of JSON maybe I don't need all of it and that can thereby reduce the amount of stuff that I'm compacting a bit but maybe the middle of that JSON is what matters so it's hard >> and that this is on this is on get if you want it um if you go to and I I would be definitely interested in the idea that we have a um We may add basically um some support for some like automatic compaction method to um to pad AI at some point

So my compact method is in this repo. I have literally I did not look at it because I was too busy. So this is written by claude code once and it's type safe because I make claude code write type safe code but other than that I have no idea what's in it. Um, >> and there's a even there's the brutally aggressive compact version is is literally like pass all and see what happens. >> Cool. Um, anything else I can help with? Otherwise, I think we might draw this to a close. Thank you all for for staying with me. [applause]