Building Durable Deep Research Agents | Workshop | Samuel Colvin | Pydantic, p1.
Recording: Building Durable Deep Research Agents | Workshop | Samuel Colvin | Pydantic, p1.
hi everybody. Um, I don't know if we're waiting for anyone else, but I think we may as well get started. Um, >> oh, that is a very good question. I'm about >> So, I think I think that the like theme for today is um durable agents. So I'm going to walk through what that means and why you might care about that. But I think as I understand it, we have an awful lot of time today. So I'm also very happy to talk about any other part of the um Pantic stack, any other bit of pantic AI as as and when people are interested in it and attempt to get something to work. Um so our thesis as a company for the most part is that AI is still just engineering
that there's a bunch of people who for fun and profit, mostly for profit, spent a few years trying to persuade us that AI was so fundamentally different that we could dispense with standard engineering practices and that somehow AI lived in this vacuum vacuum and didn't need good engineering. And I think what everyone is realizing is that that's not true and that we although AI is profoundly powerful, it also still needs all the same good engineering practices. And in fact, it it exposes some new or at least somewhat novel uh challenges to to running things. One of those is obviously the most the most um obvious is that models are stochastic. They're unreliable or nondeterministic. And so we need to measuring whether they're performing right is not like measuring performance of a standard function which is which is completely deterministic. That is eval. I can talk about evals if if you would like me to
But the other big uh difference with LLMs is uh that they are slow. They are really quite slow. Anything from three seconds where already the user is like why is this not instant through to if you ask GPT5 to write a long story about a short horse, it very reliably takes 173 seconds to do so. And that is not something that we would be used to in the database. Sure, there are database queries that take 173 seconds or or longer, but like for the most part in most consumer applications, we can expect database queries which ultimately are the source of most slowdowns, whether they're behind another network or whether they're you're connecting directly to the database to return in sub 300 milliseconds, let's say. And so that's so we have these very slow requests to LLMs. If we start stringing them together, for example, in a deep research case, we get to these very longunning tasks. And all of us have probably tried deep research on any of the the models and we will have discovered that those those can run for sort of 10 minutes quite happily
So how do you think about executing that that that task? Um it turns out that what all I know openai and anthropic and I suspect others use is temporal who which we're going to talk about in part today which is a durable execution framework. They're not the only one as you see here in pyanski's documentation. We also have support for deboss and prefect which are two other I mean deboss prefect has been around for a long time and didn't describe themselves as durable execution um but I suppose are more and more moving into that space um deboss is relatively new um its advantage is that it um runs entirely on postgress but is open source they have a closed source commercial bit of what they do um but they're basically they're more built on open standards than temporal where the temporal Orchestrator is closed source. Now there is a version of that that you can install locally which is what we will use today but they make their money and they seem to be making an awful lot of money out of you using their central orchestrator for managing managing durable execution. So this is not a sales pitch for temporal or deboss or any of these ex these frameworks. I am I know the the people but I have no financial investment in any of them. So and I don't pretend to be an expert in how they work. So we may have to do a certain amount of searching if you have complex questions on how they work
But I I'll do a I'll try to explain a little bit about how temporal works which I know best but it's probably going to be easiest to describe what it does by looking at a real world example. Um so before we before we get started though if if people are gonna I mean I suppose a question here two ways I can kind of give a talk about this and then we can try it or we can kind of work all together at the same time along. I haven't given this kind of before so I'm I'm open to I don't know what the best answer is and I'm open to any take >> work together >> work together. >> Okay. >> Okay. So let's we will try to work together all the way through. Um the first thing you need to do is in if we're get we'll go with the temporal example first and the first thing you need to do is install temporal which has decided to go and give me a nice big error. Um, if you're on macOS, the simplest way to do it is, uh, assuming you have Brew installed, is Brew installed temporal? Um, and I will then explain to you roughly what it's doing
But I don't want to I I guess if we're going to work together, I'll give everyone a few minutes to get Temporal installed. The other thing to do what if you've got temporal installed is to um uh get the I'll work mostly from our demos repo which contains lots of examples of of pyantic AI and our code and so I would go and clone this URL. >> Is that does that help? >> I built this Chrome extension. I've never used it because all of the QR code Chrome extensions were terrible. Sorry. my first time someone actually goes and uses it. >> Is that working? Or just go to github.com/pidantic and search demo and you will find it. Come on in
So, if anyone has joined or is not sure, we're we're starting off by cloning this repo and getting um brew uh sorry, getting temporal installed with from brew or I'm sorry, I don't know how you do it on Windows or uh I presume it's easy enough and then as I mean is anyone struggling. If you are, please let me know and I will try and help. Are we all MacBooks? It's kind of a sad state of sort of state of affairs that >> I did um it but it's just um that's the wrong URL. Um it's there, but it's also just pilantic. github.com/pantic Well, then search demo. I don't know if I can make that any bigger. There's a way to do a QR code in the terminal, isn't there? Which I will >> sorry. Yeah
>> Uh maybe if if you say so then yes. >> Yeah. >> Okay. It's on the description of the workshop apparently. Um yeah. So once you have um brew installed uh you do temporal server start dev in one terminal and that will start the temporal orchestrator. So the idea is here this is a server which is basically keeping track of the status of the application that you're going to run um and will and will manage its state such that if that process is stopped or crashes you can restart from the same point. That is the that's the fundamental meaning of durable in durable execution
So that allows you get a network error, it can retry perfectly fine. It also allow things like if you want to sleep for 10 years um and then continue um running a process. The temporal orchestrator takes care of stopping and restarting that that um that process or less likely 10 years. If you wanted to just have a process that was like send welcome email that just waited for a week before sending the second email, supposedly temporal allows you to do that without the overhead of like having to have that process running for the entire week. Um uh and as I say I I have the same example in deboss. I tried to run it last night. it was broken. I messaged the deboss founder and she has fixed it supposedly and so bit of luck we will we will be able to run the the deboss example as well
Um so I guess the other thing to do if you've cloned the repo is cd into it and from the root of there run well run u sync and you should have assuming you have UV installed. If you don't have UV installed I guess we need to work out how to install it. Um but once you have that run uh and that will install the rest of the dependencies we'll need while you are doing that I will talk a bit about the example of what we're going to what we're going to run. So this is the the nondurable version of this. So the principle here is that two AIs play 20 questions against each other. So we have a relatively smart model in the form of uh questioner which here is using GPT 4.1. doesn't really matter what model and we have an answerer which is which can be demo in this case it's haiku um and the answerer knows the answer to the question so you can see here this is the code for the answerer agent it's pretty simple we've um set the model to be called hiq you can see I was using grock but decided against it um and it takes a dependency type which is of type string one of the the big value propositions reason of pyantic AI is is that we we care an awful lot about type safety. So you will see the answer agent is generic in both the output type well first in the depths type which is the first generic and in the type which is the the second generic
Um and that means that when we use um a decorator function like this to add more stuff to the system prompt to instructions when we pass the the context argument run context it needs to be generic in the same type as as the type of the dependency which here was strings. So if you can see if I change this to the wrong type I would get a type error immediately. Um but this means I can access the depths attribute and in a type- safe way within the instrument function. Mostly this this um pattern is used for function calling in pinant AI agents which I'll get to in a minute. But here we're using it to add something to the system prompt. But all this is doing is saying that we can pass in the answer to our to our 20 questions um when when we run the agent and the agent will add that into the system prompt. Um um and then this is our um questioner agent which is going to go and try to find the answer to the question. Um, one thing to note is that to get this to complete more often, instead of it being yes, no, as you would normally have in 20 questions, I made it uh hopefully you can all read that code
It's an enum of different answers. Um, which allows you to get there a little bit more quickly because otherwise it was amazingly dumb at working out the answer. Um, and so you can see here, um, we have this game state that we're going to update, which is which is the the the proposed answer. And then this is how we register tools into our agent. So slight digression on what agents are. Anyone who's been keeping up with AI will have heard the word agent used enormous amounts. At the beginning of this year, there was an awful lot of debate about what agent meant. Um my joke is that there are three definitions of agent
There is the like AI definition of agent, which is an AI calling tools in a loop. There is the infro definition of an agent which is basically a synonym for uh micros service and then there is the business definition of agent which is someone who can replace a human. Um when I we talk about agent in this case we talk about the AI definition of agent something that allows an LLM to call tools in a loop. Um, and in this case, uh, we add to the questioner agent one tool, which is ask questions, which is ask question, which is how it's going to effectively try to try to find out the answer. It's going to continually call that tool until it it works out what the answer is. And this this might seem like a trivial Q&A 20 questions joke, but this very accurately reflects in a in a very concise way the kind of what we call the treasure hunt case of agents where they basically need to go and find some answer but they can't get to the eventual answer without finding some intermediate steps along the way. So you can imagine that in some sense claude code is doing that. It's doing a treasure hunt to like find the file which has the exception in it, then find the other file that passed the value that caused the exception and then go and fix the exception
Um, and and obviously many more complex use cases. Um, but this is a kind of like humanly interpretable like viewable when you run it example of of that treasure kind of aantic flow. Um and so you see we print out the question and the and um the answer we got back from the answer agent. So the answer at each step um and then this is our function to play the game. So again no dur durable um execution involved in this case at all. I'm just um getting something that actually demonstrates the the behavior. And we can come over here and UV run uh durable execution 20 questions. And again, is that is that big enough that you can see what I'm doing? Please, if if what I'm doing is not clear, please shout out because >> is that getting to the point where you can see it? I'll do that
So, I'm going to run 20 questions here. And this is not necessarily that exciting to to watch u but we will run it through for a bit to see whether it actually gets to the answer. Did you have a question? >> Yes. >> Do we need log fire? >> So you will you you may use logfire. So um let me explain what logfire is for those who don't know then I will answer your question. So you can see this is running and asking questions and getting yes and no answers. If we come over to you will see in our code that we instrumented um this with logfire. So logfire is our observability platform and this is the SDK for it and there's only those three lines of code
So the short answer is if you don't want to use logfire you can either comment them out or you can say send to logfire if token present and then it will only try and send to logfire if the token is present. So if you don't have a token you don't need to. This is not me trying to like well I suppose everything is me trying to sell logfire but this is not me I'm not primarily trying to sell you logfire right now um but um if we look at so it's gone wrong that that agent run but you can see here we have details on the exact uh exchanges that were were being made with the LLM um so you can see here I'll try to zoom in you can see the the in this case a particular conversation with the answerer agent and we can see perhaps more interesting ly the full conversation with the questionaire agent here where we were going through with it asking questions via the oh that's why I really don't need that right now um >> so this is logfire this is our >> oh it's got into that awful state of zooming in where I don't know how to get it to zoom out again there so so this is this is um our observability platform and this is a trace of in this case a particularly long trace we've already got to which is the trace of the entire run and you can see each individual step of what's happened and at the top you will see that we actually got an exception um uh >> yeah this is this is this is the tracing platform um Log has a generous enough free tier that you can sign up on the free tier and you definitely won't go anywhere near the limit but you don't have to if you if you're getting errors you can either comment it out or as I say just uh set if token present to log fire if token present um and and turn it off. Um but it is helpful obvious well we think it's very helpful in explaining in understanding what has happened and things like measuring cost. So you see here I sent spent 4 cents um on this exchange between the two models running um and on this particular case it completely failed to get to the answer um uh which is unusual. Normally this way it's it's pretty good at getting to the answer. But um I'll try I'll run it one more time and see if it see if it gets there. But obviously the thing to note here about this is sorry go ahead
Question. >> Yeah. So if if you wanted if you want logfire you'll need to sign up for a logfire account and then run logfire or off. I can help you do that or you can just comment it out if you would prefer. And you also need like >> Yes, you will also need a key for an LLM. I can I mean I'm afraid it's a it's a it's a it's a requirement of uh AI agents that you're going to need access to a commercial LLM for for these. There are two options. You can either use your own anthropic key or I can generate a key with Pantic AI gateway which is our new um AI gateway
I am not here to try and sell you that in particular, but that might be a quick way if people don't have an anthropic key of of having a like access to a single key that will allow you to make those LLM requests. So, let me know which you would prefer. >> Okay. Did I say that? Okay. Well, then I will definitely do that. Um, so slight distraction while I um this is this is um our gateway which should allow you access to all of the models we're going to use today. I do have autocharge turned on. Um, and I'm going to switch account actually for this
And um Okay. So, I've got a URL. Um, >> yeah, I will show you in a moment. Um, let me just see if I'm just trying to get the QR code for this quickly. Okay. So, that should be the QR code for getting access to Panski gateway. Um, or I can email it to anyone if if that is not working for you because I know that gets it to your phone but not to your laptop. >> I don't know if there's any commonal way for me to send you all the message
>> Yeah, that then is another QR code to share. But yes, um, but that might not be a bad idea. Um, >> do you know how to do that quickly? >> That is true. Okay. Gist is a very good idea. Okay. Um, >> yeah, you do. And then once you're in once you have an account, you then need to go into like basically create an API key and then you can use that API key
And I'll show you how. >> Um okay. So, if you want to get to links, and I'll put all further links into here, go to gist.github.com/samuelculvin, and then it's the most recent one. Um, uh, probably easier just to type that in. At the moment, it just contains that gateway link I've I've sent you, but I will share anything else there. Um, if if there are anything else to to share. Um, if you want to disable logfire in this example, you literally just want to comment out the three lines that have logfire on them. Um, which are I mean you want to do this basically and that shouldn't allow it to run without logfire
>> Unfortunately not. We built the gateway in a massive hurry and we have a different account system much to the annoyance of numerous users and my team. That was my my mistake. So yeah, there are different accounts today. Um um but yeah, so so if if you have that gateway link and you want to use it to generate a key, come over here. You'll be in user mode. So you'll see a screen like this. Create new API key
Uh don't need to give it anything else. Create that key. And then if you reveal the key, you'll get this command to copy which will set the Pantic AI gateway environment variable. Well, give you the command to set it. Come over here in your terminal. Do that. That's going to set my API key. And then wherever your we where in the code we have um calls to an LLM, you just want to prefix it with gateway slash
And that should be enough if everything holds together for it to start working. Um, so you will see now if I run that 20 questions example again and the gods of demos are with me. Oh, for sake. >> Um, it is called Open AI. I thought so. Okay. So what you um I may have the an older version of Pyantic AI installed. >> Your types should not be wrong but okay
And with the latest version of Pyantic AI that is that is working. Um, I can I what I will do right now is uh I'll stop that and I will push to the the repo those two fixes. >> Okay. >> Sorry. >> Okay. So the the simplest way to get it is to go to get gist.get github.comsamuel. >> I I mean I can't zoom in on the URL, I'm afraid. Um >> uh >> there we are
Is that that URL there? Sorry. >> Uh, have I pushed the changes? >> No, I have not pushed the changes. That's very annoying of me. I'm going to for this example, I'm going to say Okay, I've pushed that to main now. So, if you pull the repo, you should have both the newer version of Panskai and also um I've switched the the two models in this example to use gateway. I will switch them more in a in a minute. Um, is everyone roughly clear on what's going on? Shall I move on to some durable execution, which is Cool. I'm pleased to hear someone's having some success
it it succeeds a bit more than half the time. Um, it's surprisingly dumb. In fact, I tell you what it will be. >> Uh, let I'll come and look in one moment. I think part of it is that I'm using >> GPT4.1. If I used um uh anthropic uh >> Claude >> Sonnet 45 I think it's like that. So if you if you use a a slightly newer model than 4.1, it generally does better. But when you say it's running over, >> what I'm saying is getting the answer >> but it's reaching the usage limit
>> Oh, weird. So it's not exiting when it's when it's found it. >> Yeah, it's so weird. >> Maybe. I mean, it's possible it's reaching it literally on the last one, >> but that seems unable. >> So when I set the usage limit to 25, then it's reached at 25. So it's know the answer before, but does not stop. >> That's fine
I'm not sure. Does it need to run? >> It doesn't it doesn't matter for for demonstrating the point of durable execution. >> Yeah. Anyone else who needs some help before I keep going? >> I I took I I generated a key over here. >> And will this work or No. >> Yep, that should that should work. Yeah. Oh, but you're in your own org
So, did you put did you put some money in? >> No. >> Okay. So if you want to use um >> so if I want to use yours where do I what do I do? >> So go to gist. So gist.github.com >> just put gist at the start of the >> and then this is the link to join. Whether that's going to work with you already logged in, I don't know. If you go to that, if you copy that link, continue with GitHub, let's say. Oh, you're not logged in with GitHub. Maybe with with Gmail if you're logged in
Whichever whichever is easiest. Oh, you're not logged in at all. So, you probably do better in a window where you're logged in with either GitHub or Gmail. >> Okay. So, now you can go and create a key. >> Just accept all of that. That's fine. And then you use this key if you do reveal
This key should work because it's funded by my account. >> I want to know where can >> um >> so you can you can go to settings and get it or in your terminal you can do >> so if I were to go settings. >> Yeah. And then you go to right um Oh no no no you're in the wrong settings. Go back to the project you run. >> Okay. >> If you if you go to the your your project. >> So here >> Yeah
Go to one. Yep. >> And then go to settings here. >> Uhhuh. >> And then you can either uh set up >> or you can just generate a right token manually. >> Sweet. Thank you. Sorry, I got your API key
So, basically, I just put it in a local >> something like that. Yeah. Or export the environment variable. Yeah. But >> but it doesn't seem to work. >> You don't need the anthropic one if you're going to use uh this one. But that's looks like >> Oh, sorry. >> Yeah, you're in a Python code
You can't do export in a Python in Python code. Yeah, >> you need to take that and you need to >> Well, if you want to do it in the Python code, you can do >> I'll just do it in the MV file. >> Yes, but youv files are not automatically read by Python, right? So, if you wanted to So, the simplest way to do it here is to do Can I just type for one second? If you do just uh o environ this way allows you to set an environment variable called Oh, that's the wrong keyboard combination. Uh, and then you probably need to import >> O and then you should that should now effectively set the environment variable in Python code. >> Sorry. I just copied the key and should go to run. >> So you copied the key um from from uh from here. Okay
So if if you scroll down a bit, >> if you just copy click this copy to set the environment variable. Okay. And then go back over to your terminal and just just copy paste into here. >> Yep. And run that. Okay. So that should now have run it. And then if you do UV run >> space no uv space run
Sorry. Can I should I type for a minute? Uh, UV run durable execution 20 questions. That should then run that example. Um, you haven't got UV installed. Okay. Um, I don't know the quickest way to install UV. >> Brew install UV on the machine. >> Okay
So, I'm just installing UV if that's all right. And then you should run that example. And then you you might want uh if you haven't pulled the changes, you might want to just comment out these two lines to not have the issue with logire as well. So then you should and it's going to complain once you've got here because you haven't you probably need to pull is what >> um >> need to get the latest. Yeah, if you're it's gonna it's going to error. And then if we do pull Oh, sorry. What's changed in this file? Okay. So now in theory if you run the example >> and >> the gods are with us
We may Yeah, there we are. It's running. >> Thank you. >> Okay, I'll let you let you run it. >> Everything doing everyone doing okay over here? >> We've got two laptops going well. We're really >> cool. weird. >> I don't know
Maybe it's because I'm using Tai instead of like I don't know what the other LSBs are for Python. >> It may well be. I haven't tried it. If you're using base pyite or pyite, it would be right. >> Okay. >> TY is I haven't got around to trying it in in earnest yet. So, >> yeah. No, so it's we the the the uh language server in in Zed is um base pyite or pyite and that's what we use for all of our type checking
So, we get the same thing. So, no guarantees on TY. >> Not at all. >> Cool. Any questions over here? >> That's fine. >> Yeah. >> H So, so you need you've got first step is get to the right repo. >> Um so, if you go back to should I just show you quickly? Um >> yes, perfect
Um, and then you you pull clone that. >> Um, and then the only thing you explicitly need. Um, if you've cloned that, then in theory, the only thing you need is a >> um >> is an account with either you can use your own AI account if you want, anthropic or whatever, >> or you can use this link to sign up for Pilantic AI gateway and that will give you $100 of credit for basically any of the models. >> Oh, nice. >> But if you if you're still having trouble, let me know and I'll come around again. >> Cool. Uh, so yes, sorry. So you need to go to um, sorry
Oh my goodness, different keyboard. >> If you go to this link here, if you if you open this this workshop uh, gist file because I'd add add any other links here. If you follow this link, sign up for an account with our gateway and then generate a key. Uh, hopefully Jeremy can help you with that. If he's not if you if he's got there, that should be that should get you get you a key and that should let you run the run the code. >> Thank you. >> Doing all right. >> Hello
Nice to see you. How you doing? >> This is Dante. >> I thought you were going to show me the cafe bar again. >> Nice to see you. >> Yeah. Yeah. I'm alone with four kids and my wife is away and >> Hello. >> He's today our mascot
>> Very nice. >> Cool. Okay. Anyone else need any help or shall I should we do some actual durable execution? >> Okay. >> Question. >> Go for it. >> Okay. Yep
What what what drives my choice of model? Uh a lot of fiddling around basically. I mean I have I suppose a bunch of experience gives me some intuition about which models work well for which things for the most part I use donet 45 if I don't have another choice uh if if unless I have no otherwise then for a light model some mixture of haik coup the gro oss model is very very fast but dumber uh sometimes I use openai 4.1 I don't have a good answer I find gbt5 just too slow most of the time So I but it is it is clever. I don't have a particularly clever rubric for making that decision. Yes. >> Yeah. So maybe I should I should start with a bit of a like kind of background on uh how pyantic AI works before I kind of head on into durable execution. So if you look at um this code here, right, this is basically a self-contained agent. Um we can ignore everything else going on and we can just look at this this bit of code here
Okay, we have um an output type of answer. Um and we have this agent which I said was uh like LLM calling tools in the loop. But this particular agent doesn't have any tools. What it does have though is a structured output type registered. And so we take this this thing, we turn it into a pyantic model. We use the pyantic model to generate a JSON schema and we pass the JSON schema to the model and we say you must return this. And we can do that in one of three ways. We can either use tool outputs
So we register a tool called final result. Um and we ask it to call that tool. And because all it can do is call that tool. It will always call that tool. Um and um the second way is there is in openai and gemini and anthrop anthropic as of two days ago they have native structured output systems. So it's like tool calling but it's basically a way where you say you must return data matching this JSON schema. And then the third way of doing it is what we call prompted output where we basically update the system prompt to say here's a JSON schema you must return data matching this and it's not very reliable but it's there as a kind of fallback for everything else. The default is tool calling
So if I come back over to logfire here and I look at this which is the the run of the the like one of the like micro runs as it were of the answerer agent you can see um in fact if we look at the actual call here it's actually you can see even more detail. You can see um sure token stuff but we can also see one tool was registered with it and that tool was called final result and that tool had this schema. Um this is the the the JSON schema representation of an enum. And so we said to it you must return this. Uh and then if you look at what happened when it its output was a call tool call to final result with the with the like required attributes. Um we then run finantic validation over that thing. If it passes then that's what we return to the user. So you can be guaranteed that when we'veated that thing is this must return an answer both at typing time and at runtime we will always get an answer
If validation fails then we basically return the validation error to the model and we say you got it wrong. Please try again. Here's the validation error. It's amazing how often that is enough to succeed. So let me take having just said we're going to get to durable execution. Let me just show you one more example because I think it would be helpful for people to understand. So this is a very sort of hello world example of pyantic AI. I've got logfire in here but again you don't need that if you don't need it if you don't want to use it and we're just doing structured outputs
So here I'm using 4.1. One of the reasons I'm using it is this code was written six months ago and that was the kind of obvious choice. Uh and again it's basically what I just showed you. So we've just got a return um an output type and so agent is generic and the output type you will see that result.output is is of type person at typing time if I try and out access result.output output dot first name uh uh it's a typing error if I access name it's nice right so we get that typing guarantee and we also get the runtime guarantee of of running the validation and so if I go in I'll put in gateway just to make the point here and I'll run this case and I haven't got the gateway key set here which is annoying so um we run this very simple run of an agent, we can see that it it successfully output the the data we were looking for. And if we come over here and we look at the agent run again, you can see the same thing. Um final result, uh it called the final result tool to get the structured output. Now we can be slightly mean to the model and we can basically cause it to fail. So in this case we've basically got the same code except we've added uh a field validator that the date of birth must be before 1900 and obviously where the the unstructured data that it's reading has 87 as the year it's going to assume 1987 it's going to get a validation error from that which is going to be sent back to the model and you hope it's going to use that to then work out the the answer
I can put gateway in again just to show you how it works. Um, and run that. And you see there's two calls to uh 4.1. And if we look at the exchange now, we see what's going on. So we have the system prompt, the users input, it it returns um, as you would expect, 1987 as the date. Then we return to the model. Our tool response is uh validation error please try again and and details on the on the error from from the pidantic validation error. And then it uses that information to try again and successfully get 1887
So it's basically using the information in the validation error to work out what to do. And this is amazingly not it's not a universal panacea but it's very it's surprising how often this does solve problems and stop your workflow from crashing that would otherwise have done so. Um so there are some specific things like um homogeneous dicts are very hard to represent in JSON schema. So like dict of string to int is very hard to represent in JSON schema unlike a type dict or pyantic model or data class which where we have like the kind of typescript interface stuff. And so very often if you ask it for a dict of string to int, it will get it wrong the first time and try and generate some other object. And if you can just prompt it and explain what's wrong, it will very often get it right second time round. The best way if you do have that actually is to have a list of key value pairs and that matches how models think much better. Does that sort of answer the question? Where was your question? Sorry
>> Yes. >> Cool. So any other questions or should I go on to some durable execution stuff? I'll take that as a yes. Yes, I should move on. Not yes, you have more questions. Um, so over here I have in one terminal temporal server start dev which is going to run the temporal orchestration server. Um, which you can see there. Nothing very interesting happening
And I'm going to come over here into 20 questions uh temporal as an example. I'm going to have to again apologies put gateway in because I gateway only came out on Thursday. though I haven't updated my examples yet. I'll put that into both of the cases. So, we're going to make some Oh, Jesus. We haven't lost it. Cool. Um, we're going to make some changes to the code
So, you can see this this is an older version. So, the answer agent just returns bool of yes or no rather than the the enum might make it less likely to work, but we can fix that. But we wrap our agent in this temporal agent um type which gives us back something which basically looks like an agent but supports temporal. Now the complexity of how temporal works is it it splits your code into what it calls workflows and activities and workflows should be entirely deterministic. They cannot do any IO any networking any file system access. They can't call rand what they do has to be entirely deterministic. And then all of your non-deterministic IO code like making a network request or calling an LLM has to go on in an activity and temporal does some heenous things to the Python event loop to split those two things up. And what that means is that the it can basically do some arithmetic to work out what network request or what activity you are going to call every time you call an activity
It's literally counting them through. Uh so when you run your um workflow again to to basically replay it, it can it can basically say, "Oh, I know the answers to all of these activities until I get to one I don't know the answer to and continue." So it's basically counting them off. So it's basically saying, "Give you the answer from activity one. Give you the answer from activity two. Give you the answer from activity three. Oh, you got to activity four. I don't have an answer for that. I'm going to call the activity." So this durable execution when it reruns it what it's basically it's got cached answers for all of the activities that you've already run
Does that make some sense? I'm not an expert on temporal. That is my intuition about what it's doing which you will see in a minute. >> Well when you have a call an activity that is stochastic but where you are rerunning it when you're in replay it's deterministic because it's got cached answers to every activity. um it'll make more I did not I spent six months talking to temporal and not admitting to them I didn't understand at all what their thing did then I wrote this example and it started to make sense to me so hopefully running this example it might make sense to you more than it did to me for the first six months but the the point is that under the hood you need to go and do a bunch of work to decorate all of your activities and workflows as activity or workflow we've done that for you with paidantic AI so you don't have to do the hard work of working out what you need to decorate as an activity and what is a workflow We know that when you run an agent, each LLM call is an activity, each tool call is an activity, and we take care of that stuff for you. So you won't see half as much activity and workflow decorator as you would in most temporal code. Um uh and so so yes, we've wrapped our agent in um this this wrapper. Done the same with our questioner agent. And so then we get to our like somewhat ugly dare I say it hope well this is being recorded but um this so this we basically have to have this class with this workflow definition um decorator and then we have to have a workflow.run run decorator
And this is how we basically define our workflow. Um, and then when we come to call it, we have to set up a temporal worker, which is basically what's going to connect to the orchestrator. We register the workflows. We do a bunch of basically. Um, we register the our plugins, which in this case is um the two agents. Um, and then we get to actually kicking it off. This is the in some sense this is the interesting bit. So um by default if there's no resume ID we're just going to get the workflow handler and and run it and get the result and we're going to print out this workflow this um uh sorry ignore me
If we haven't got a resume ID then we're just going to kick off the workflow execution and we're going to give it uh the workflow an ID. But what what I've added here when we call the function is the capacity to basically pass a resume ID which is basically saying here get me a workflow and and calling results will continue to run it. And so what I can do is show you what it's like to resume a workflow based on an ID. Um the other thing to note in here is if you look at our answer question tool it will fail deliberately one time in 10 10% of the time and that will show how the other thing that temporal can do which is automatically recover from exceptions basically unhandled exceptions it will um automatically handle and retry is that >> yeah high level what is the plugin Good idea. Good question. Um, it is doing the work of I I honestly don't know. It's to do with working out what's an activity and what's a workflow. I didn't write the code
I'm not going to pretend. I'm not going to lie to you and tell you I know what it does. Um, >> uh, let's look at the dock string and see if we can find out. Plug in a collection of plugins to the worker. Any I bet this is written by an LLM and doesn't tell us anything. Um, >> the answer should be read the code and that's fine. Read the docs. I I think the answer is read the docs
I'm sorry. I mean do a bunch of work under the hood to make things work. And I I can I can look into exactly what they are. I don't pretend to be a temporal expert. Um we have done lots of work to make temporal work both with um tools and with agents within PyCAI. I know that temporal also has an integration with OpenAI agents but it doesn't wrap it doesn't wrap tools. So actually once you have an agent with any tools on it, it it stops working. So like I think we have one of the more sophisticated implementations of like support for um support for temporal within the agent framework
But let let me run it and hopefully that will give people a better idea of what's going on. So let me clear my terminal and I'm going to run not that at all. I'm going to run 20 questions temporal and it's going to immediately fail. Why did you fail? It literally ran for me last night and it's given me the most heinous exception trace. I honestly I'm going to get rid of this for a minute and see what's happened. >> Okay. And it worked for me second time round. I mean, it's got a broken because sometimes it break it fails one time in 10, but I honestly don't know why it failed that first time
Um, but you see it you see it now running. Um, and the the the interesting bit is if we come over here and we look at so we have the like tracing on the the this thing running. But in particular, we can go and get that workflow ID that we just kicked it off with. I'm going to copy paste that. And now if I kill this, so this is simulating let's say Kubernetes node shuts down. The point is we can go over and we can restart. I'm going to clear here to make it clearer what's going on. Um, if I go and restart that workflow based on the ID, you see it's gone straight through to question nine
We didn't get the previous answers printed out because they were within the activities that weren't run because we had cached results. If we go in and look at what happened, you'll see these first set of steps were all taking like 1 millisecond, which is obviously it looking up the cache rather than it going and like actually finding the answer. Um, and they're slightly incorrectly placed within the trace. They're not within the uh agent run, I think, which is why that one is still going. But oh no, that's no I'm I I lie. Uh it's working correctly as in you see the trace itself continuing to run and then you see the first whatever it was eight questions and answers were instant and then we got into continuing to run and may or may not discover. Like I say this is just doing the true or false version so it's less likely to be successful but it's got itself confused again about bamboo. So I'm not holding my breath for success
checkpoint. >> No. So, so, so the point is you shouldn't need to do anything. For the most part, it should restart. Um, I don't I think that the the point of the the point is in production, you don't need to do anything. It just takes care of running that thing to completion itself. Um, and you're you're just basically making sure that you have enough nodes running to to take your demand. I'm using this like kill it and rerun with an ID as a like very artificial way to simulate effectively it shut down and and then me rerunning a particular node
I think if I I so so the the this is going to fail, isn't it? It's got into is it stainless steel? I think we can accept it's failed. Um no. So, so what um what would happen is if if I just basically said uh execute workflow here and ran it again, it would work correctly. But because I've asked it to start another workflow, it will now run two in parallel, it will run the previous one I killed and the new one I've just started. And that's very hard to like basically view in a demo. And that is why in my case, I'm I'm not doing that. I'm literally saying take a particular ID and resume it to make it easier for you to understand. But the principles I mean maybe I can just do that
Um >> I mean this was like running. >> Yeah. >> Um >> yeah. Well, I mean they they're running in different async tasks within the same process by default. You can obviously run many processes to scale up, but you're you're using your code level isolation of this is the this is the piece of code that's being executed here to is from the other the other tasks going on around it in the same event loop. Same as if you're running an HTTP server, you might be running thousands of concurrent requests, but like you rely on the like abstractions of the event loop and the uh programming language to to isolate each request gets its own answer. >> Is there any similar checkpoint multiple times? you get on there or >> I don't know if that's possible. It seems that it should be possible given that it has a checkpoint
That is not how it's designed primarily to be run. It's designed to be a durable way of running the same task um running a task and and effectively recovering from failure. >> Yeah. Does any kind of mitigation of that? because it's not meant to be. So >> we don't do anything in that direction at all. >> Okay. >> Um we have in pyantic graph which is a whole different library. We have a system of snapshotting
We're actually moving away from that because snapshotting makes running parallel nodes impossible and we're just moving to relyable execution framework to take care of that stuff because that's what they do. Um, so I would have a have a Google or ask your LLM of choice how whether it's possible to basically branch from a single snapshot. I I don't know the answer, but I I don't think that's their like that's not what they're trying to do. What they're trying to do is give you a way to write normal procedural code that could carry on for weeks or months or whatever it likes. You know, you know, you can imagine the like naivest version of like my first day of coding, I want to send someone an email two days after they signed up. I'm going to sleep for two days and then send an email. We all know that wouldn't work in any real scenario. The idea of temporal is it gives us back the idea that we can write that kind of code
and that they're basically their big the reason that they are so I think one of the reasons for their success is they basically got the Uber contract to deal with all of Uber's state updates and that has given people lots of conviction that that temporal is like very very scalable. I have another example I can run with deboss. I don't know I know almost less about deboss than I do about temporal but I can try I tried it last night and I can I can try that now. But, um, I'm get just going to see how people are getting on with with getting that temporal example to run first. Um, >> 25. >> Okay. I mean, I think if if you changed it over to use the same answer type I had here, you might have more luck. But I I'm actually going to go on to to a like more complete example of um of uh a deep research with temporal next
Anyway, I mean yes. >> Are we going? >> Okay, let me have a look. >> I commented out >> stuff. Yeah, I you you've got the temporal server running somewhere. No. >> Okay. So I I was maybe other people are confused by that. Let me show you
So you should have installed the temporal server and be running temporal server start dev in a separate terminal. >> Sorry if I hadn't made that clear. >> I'll let you try that and I'll come back. Just as someone has pointed out to me that if you're interested in that temporal gives you this UI URL printed out when you run the server where you can look at IDs and see a kind of their version of a trace of like what's gone on within running it. >> Um >> I should have. We spend loads on anthropic. My worry is it's been deleted by um >> that they literally have a like you know how they have clever things for like deactivating a key >> yeah it's gone immediately from my list of keys. So I think it's been deleted because I bet I've got some email some hysterical email being like your key got leaked
Um, >> anthropic credentials detected. Very good example of it working. So, >> I've got I've got a let me try something. Let me try something else. Okay. So, I'm going to take this key. >> I'm gonna say um What is incredible? Let me see if it's clever enough to >> Is it Is it doing I don't know. I mean write a program
Um, Try try running that and see whether or Uh, see how long until that one gets killed. >> Sorry, did you have a question? >> Yeah. So, what I realiz is is basically detecting that is the final answer. So, we completed the workflow. >> So, how can you make it fail? So, this next execution I failed. I reached the limit. I didn't get the answer, but it's still in the running state. So, is there any >> that must be the error that you were running into where it was like raising an exception at the end? >> Yeah, >> when when I've run it and it succeeds, it doesn't raise that exception
So, I don't know why your code is raising an exception at the end. >> Okay. If >> in theory once it finishes and it hasn't raised an exception, it's like, oh, >> if it doesn't conclusion, >> yeah, >> then what kind of state? So is there any way to because maybe not every workflow? >> Yeah, I see what you mean. >> Successfully. >> Yeah. So how do >> there must be a way to be like end I don't know what you raise. So so I think if you raise an exception within a workflow it will end the the the like durable bit of like recovering from errors is when exceptions are raised within activities not within work. So if in the workflow you raise an error like can't can't succeed that should kill the workflow
>> Try to implement an error. >> Yeah. because that's I think that's a use case. >> Yeah. Let me let me try and do that for you and see how we get on I've only got one email about it. >> It says that key is is good. So I haven't I haven't >> Okay. Um you wanted to try um killing the um >> if you have an agent which came to a conclusion it fails but that's your result that you failed
>> Yeah. >> For whatever reason. So if I here if I do raise runtime error exception in workflow like this um I run the server and I come over here and I run that exception in workflow. It continues to wait to run. But I think if you looked at here, >> still says running, doesn't it? >> Yeah, I don't know the answer to that. I I don't know how they det how you mark a workflow as completely dead. I'm sure they have an answer. I'm not an expert on temporal and how they do it
>> I agree with you. It seems weird because if you're if it's a deterministic error, it should happen every time and I don't want to rerun it. I want to like >> Yeah. >> Well, I suppose it's now raising it again because I've I've re I've started a new one. >> I accidentally accident. >> So this is now running two workflows. >> Yeah. Yeah
So it's it's running two because I I literally started off two and and how I've done the print it's it's like >> it's not liking it. Um um can I kill them here? I can terminate them from here. >> Um any other questions or shall I go on to like bigger example of temporal and then I can I can go on to deboss as well and try it. Did you have a question or you okay I'll I'll keep going. Um, so I'm going to I'm going to walk through this example here, which is a a kind of more I guess uh realistic uh use case of durable execution. This is a very simple um deep research example. I was at a talk the other day somewhere and I saw someone using this as an example of how to do uh deep research, which I am not for a second claiming it is. I just put something together in half an hour to a talk to to talk about uh durable execution
But this seems to now be becoming a kind of open source example of of deep research because I think partly because it's very simple. So we the the way that we do this is that we have an agent here which re returns this deep research plan and this plan is made up of the the steps that we need to perform to do our deep research. Um, and it's it's one shot, generate the plan and execute the plan rather than than the like treasure hunt case where the next thing to do is dependent on how far we've got so far. So this is a kind of opposite opposite case, right? The plan is still generated by AI but it is generated once rather than kind of evolving over time. Um, and yeah, so so we have some executive summary of the plan. So quite often you will see in a like deep research it says this is what I'm going to go and do. quite often it'll say are you sure I don't have that bit. Um we have then a bunch of web searches to run
We've used these max length attributes to to constrain how many searches it can run. Um and then um instructions on how to analyze the results. So we start here with the the plan agent. Then we have a search agent which uses I again I'm using different models here because it seems it's gen I find generally the Gemini models are good at search. Google search is the one thing Google really are best at. Um and then we have the analyze um agent if you want to change all of these to be anthropic. I believe that should all work. Um we register built-in tool of web search tool
So this is um not a literally a tool that we're describing defining in Python code. Instead, this is effectively setting the attributes when we make the request to the LLM that says go and do a web search. Um we have another number of other built-in tools and there's some settings for web search, but this is the kind of simplest possible case. Um this is this is just to be this is a non-durable execution case. So you should be able to change these examples to gateway. Um I think we make some tweaks in how we do it when we go over. Um um and then we the one the one extra thing we do is we give um the analyze agent um access to run more searches. So effectively this is this is the kind of catch all for the fact that like the the the the search we originally did was not sufficient
Um uh and we we again we register the previous agent as the as the dependency here for it to um do that. Uh and then here's our actual deep research function. So we use the plan agent to build our plan. Plan is the output here. So the plan is both at typing time and at runtime an instance of deep research plan. Then we go and we use uh async.io task group to run all of our searches in in parallel. Once we get our searches um we take their search results um and we pass them into the analysis agent to to to as its input to basically go and do the analysis. We use this format as XML function which is part of Pyantic AI which basically takes all or most P python objects and basically smashes them together into something that looks somewhat like XML
It's our experience that XML is easier for an agent to understand than than JSON because you have nice closing tags to understand. So you don't need to count the number of quotes. If you see um uh less than slash um fubar, you know that's the end of fubar. That's much much easier to interpret than trying to as I say count the number of quotes or count how deep you are within some JSON object. Whether that's really true or whether that's me uh anthropomorphizing models and it's completely unnecessary. Format XML seems to work well as just a like quick way of taking some structured data and smashing it into an object that into a string we can give to another. Um and we asked it on this occasion and something I think I was actually looking for which was a list of hedge funds um to go and contact about logfire. So but that's not the point here
If I go and run this example deep research and I'll run the the plain example. Um we should be able to see that running. Any any questions on this the non the non-durable version while I while it's running? the code like it seems like you require this deboss thing. I'm not familiar. >> No, that you're looking at the deboss example. If you look at the non deboss example, it doesn't require deboss. I can >> Yeah. So the the non-de research example I don't think requires any extra keys um apart from in this case I use different LLM so you need each of them set up but it should just require an LLM because it uses the LLM's own uh own search in the deep research case
Oh, and I forgot to print out the result. That was very stupid of me. I'm going to run it again and actually print the report it generated. Um, and we can see that. Um, basically there like tracing. get into the details. Is there any difference in >> I mean I think I mean that is definitely speaking to Maxim the CEO of Temporal. He's like uh I won't do an impression of his his like very strong accent but he is like you definitely don't need anything else
We do everything. He's like very like thinks he has the answer to everything. I think temporal is great but obviously you only get that view if you're using temporal. And I think it doesn't give you the same uh like nice view of your exchanges or any of the other stuff in Logfire like the the the metrics and the search and stuff like that. So I mean I don't think you know not that we're the only people in the world who do tracing. I don't think that anyone thinks they're having temporal is is a complete observability solution. It definitely is is a is a useful view. Um, >> so it's you still suggest proper tracing
>> I would say you probably still need some other tracing because not everything is going to be temporal, right? If you have HTTP requests coming in, they are not immediately wrapped in temporal every single time. So you still need something to see your HTTP traffic. >> You still need something to we haven't talked about evals. You still need something to like view your evals etc. So um >> sorry >> to view your metrics and understand the the like your evals as in particular I was talking I think it was evals was the last thing I said >> yeah so I I used Google vertex in this case if if you don't have Google vertx setup. I would replace the models here with whichever. So I would say use anthropic change this to anthropic uh claude >> haiku. >> Oh wow
>> Okay. That's unfortunate. I will let me just say that. >> Yeah. >> Okay. Um there's a fix for it that that DA has done for that thing. I don't know whether um I think this also is an answer to what do those plugins do? Um, we have to you have to register a bunch of maybe it's not, but you have to register a bunch of pass through modules that don't basically break the temporal workflow activity divide. Um, and we used one more module in the gateway code and that was what was breaking it
Um, let me um let so so so that was deep research like the the simple example. Um, let me go on to I'm I'm I know that I'm got a bunch of other keys being used here and that that's going to run into some issues. So, I'm going to see if we can if we can fix them and get it working for you without having to sign up for other things. Um so when I first ran this we had an issue where um built-in search was breaking temporal and so I used tavilli which is uh a separate search service to to get around that when I built this example in a hurry. I believe that that is now fixed and so I'm going to use the built-in tool um for temporal instead of tavi. So that should remove the need for a tav key. Um so this is this is the same code. We will see if it runs roughly
You see same structure of our deep research agent. Um same plan agent. Um same search agent now we've changed how we're doing this. In fact this is such similar code we could probably do a like single import. Yep. >> How much data would you recommend for like how much can it store? How much upper end what's the context of that? Sorry. >> Well, I'm just wondering like would you use it almost like a database where you can store >> when you say it? I'm not I'm not trying to be difficult. I'm trying to understand what you what the question >> right so you're able to resume things >> I think they have a 50 megabyte limit I don't know if that's on a per workflow or a per object basis >> um yeah they have a relatively something like that limit again yeah um um but again Um >> yeah
>> Um and so here we've got the the actual deep research deep research workflow. I think here I haven't done that done that work to do the like resume ID stuff. Um so this will this will just run once. Um again client connect set the three plugins. Um if that's now fixed there's a type error there. Apparently that's now fixed. Um, and so I will run this and see how we get on. Is it any other? >> Yeah
So, you can >> with long chain >> lang chain there's no chance it's going to work. It's just such a ball of mud the code. I mean I know that temporal are like big and not fast at all. I don't there some of the pure lang chain stuff. So look if you're just doing an LLM call you can put that in an activity and use that within your application and that's fine. What is not going to work is basically durable execution within an agent run because the way that their that their code is structured both I know that langraph they've looked at a lot one of the reasons that temporal came to us and we worked together to do this was because lang chain was then the kind of preeminent way of building with AI. I think that that experienced developers have moved off it anyway, but like temporal were worried because there was no way they could ever run Langraph. I think all Langchain actual workflows with temporal and so they work with us and with OpenAI to to support
So OpenAI agents is heavily inspired let's say by by Pantic AI eg they copied our code and changed the changed its variable which was nice of them. I suppose flattery is the you know imitation is the greatest form of flattery. Um but uh so so open AI agents work somewhat with temporal and we work better. I don't think lang chain actual fl work workflows do but again if you have a single isolated task that calls an LLM using lang chain you can put that in an activity and that will work fine because that's not actually interacting with temporal. Yeah. And so, uh, let's just check what it gave it as its answer. Um, oh my god. So, for the first time I ran, the question here that I asked it was, what's the best agent framework to use if I care about durable execution and type safety? And in the past, it's always given pantic AI
And now when I when I give the talk, it said the number one recommendation is Langraph. uh finani second. So I suppose uh I get a bit of um mud on my face for that. Um uh I'm definitely changing model to ask it. Although we got 10 out of 10. Why? This is this is unspeakable. I'm furious. Um but anyway, >> I get the same >> type safety is Langraph is two out of 10 for type safety
Anyway, we we won't worry about that. Whenever I've asked it before, it's >> um >> Oh, it might be it might be because I changed to to from tav tavi to to >> if I change this back to to >> Yeah, I mean they have their own library. We have a very simple wrapper for it that implements tool. Um I can't find the API key. So um I'm not so bothered about going and getting a te key. Um I mean I do think it's a separate talk and and a separate subject of like I don't interest but like more and more of this like GEO like basically optimizing your site for agents. I think everyone is realizing how important it is and there's this like incredibly fast feedback loop where you basically ask the LLM why did you not rank us and then go and solve those problems and I think we're just starting to do that and I think lots of other companies are cottoning on because I mean GMO from Vel recently said that like he gets more signups from chat GPT than any other any other source. So like linking listing right for agents is super important
And so everyone is trying their own ways of doing this. And so what might have changed in the last few weeks is that lang chain have gone and like worked out the answers to worked out ways to answer some of the questions that LLMs are asking or want to want to hear the answers to. >> Okay. Good. Good. At least someone is. I knew I like them. Well, I think that's it's like given that they're probably this much of the market and Open AI is this much
It's not it's it's minor minor minor pleasure. >> Can I look at your web search tool? >> How was I doing it? You mean when I changed it? >> Yes. Yes. >> So I when I changed it, I just did the same thing as we do in the deep research the plain example. So I just this one you mean? >> So we have a section in the docs on built-in tools. >> It's on built-in tools, right? Yeah. So it's it's here. But we should have an example
>> Yeah. You're you're basically doing this. >> Yeah. But if you look in the deep research the plain deep research example that will be that will be using it. Go on. and then I'll I'll change it over and and get it to do what I print out the right answer. >> Sorry, you have to you have to >> Where was that? Sorry. >> We don't yet
We're actually wondering about doing that. Obviously, we support the the search tools built into the model, the like down the upstream model that you're using. Um, we don't yet have like other search tools is actually something we're we're thinking about adding. Um >> each provider. >> Yeah. So, so our gateway doesn't basically doesn't try to uh so what what the other gateways do is they try to basically unify around a single API because at the beginning of this year it made complete sense. Chat completions was like completely preeminent in how everyone did everything with AI and so converting to chat completions was a good idea. Then in February, OpenAI uh that same weird meeting where they had us into their office to tell us that they had copied our uh SDK
They also announced they told us about responses. Um responses basically since everyone was centering around chat completions and making it super easy to move off OpenAI. They then released responses. It seems like the obvious interpretation of why they did it. Um but then they've moved most things to responses but not everything. So audio is still only on chat completions but like thinking is only in responses. And so all of the gateways now are in this weird position where like do we move to the new one that a isn't complete and b they might do another rug pull or do we stick with the old one. So when we built our gateway we chose to just do no and then the other thing that's happened is clawed code has more and more become like the de facto way of doing code and so people actually want to use anthropics API and people can can we can I use GPT5 but through anthropics API and so there's basically two answers
One is that you don't do any translation and you're like if you want to speak anthropic you got to make anthropic requests. Uh or you do a like many to many everything translates to everything piece of work which is a like enormous piece of work and going to be super buggy. >> Yeah. As in we don't try and translate the request. We just try and extract the cost along the way. >> Um was there a question here? Yeah, I assume there's like >> Yeah. Yeah. So, I mean all that you're really doing is um you see here where we register a tool here
It's just this like ultimately that's as simple as doing a tool. You can also have a have a tool type. So, if you um sorry, one moment, let me find the right place in code. If you look at the tavilli tool um it returns an instance of this tool type which is a data class which is how you define you can define more complex stuff on tools and then you can define tool sets which are kind of the step above that where they can dynamically register and remove tools as your agent is running and then MCP is an implement implementation of tool set. So if you look at an MCP example um where we um register the agent you you see here we're just we're just calling tool sets uh equals uh in this case one MCP server and this is this uh MCP server standard IO is just uh an implementation of tool sets. Let me run that example again and see if we get. So that should now be using Tavi to make the requests. And so if we come and look at that in logfire um Oh, we didn't instrument http which would have been nice
So if we had come back in here and we do log fire instrument httpx, we can see those outgoing HTTP requests to tab as well if we wanted to. I'll let that finish first. I think that's got to running the Yes. So, it's now running the analysis agent. >> Yeah. Yeah. And that's what I mean like that's taking 40 seconds. That's the kind of that's why I'm talking about at the beginning about getting into the league of like genuinely slow things that I mean obviously an individual call to an LLM is not going to survive a like shut down and restart
You have to remake it even if it was just about to return. But if you've got six of them in a row, you don't want to have to restart all six of them. >> Yeah. And as you say, if you use Tavilli, it comes back with ustop. Um, and if I run it with with instrument httpx, you get um, so if you instrument httpx since that's what's used by the underlying uh, LLM SDKs, you get um, what you'll see is you get the like uh, you can see the raw request to the LLM and exactly what the request and response is which when you're getting frustrated by the framework whether it be padanskai or any other framework what's it really doing you can go and see exactly what it's doing here so you can see uh is that right um yeah so you can literally see how it's registered the no that's complete bollocks let me find out where I'm looking for but if basically you'd be able to see exactly how it's how it's how it structured the request so if you see here you see how we structured the open AI request and you'll see we Um then have this is the the raw request to Tavilli um and how like its exact response what what data it returned. So you can like dive right down into exactly what you're getting back. >> Can you show again? >> Yeah. So I just did instrument httpx capture all and that is enough to get the httpx stuff
So that's the kind of thing that you obviously wouldn't get if you're just using temporal's view of the like execution as it were. >> Sorry, >> you said >> you wouldn't get that at all because that's just showing you the individual workflows and activities rather than like subtasks within them or networking requests within them >> if you want. Yeah, I mean if you're using logfire, you get that for free as it were, as in you don't have to add much more code. But if you you wouldn't get this if you're just using the orchestrator's view of tasks to understand what's going on. >> So we don't go inside temporal for that. um we have to go inside all of the so so most so if you look on if you look on logfire.instrument we have support for instrumenting like hund like like most libraries you would ever go and use most of these under the hood are using the existing open telemetry instrumentation for that library just making it slightly easier to use so if you look inside what we're really doing inside instrument httpx Oh my god. Um, it's got a bunch of overloads, but like if you look at the actual implementation, we're basically importing instrument httpx from from uh I'm pretty sure if you look okay, we go one level deeper. If you look at this is this is an unusual case because we're doing a bunch more stuff but ultimately we're uh when we get to it we're doing a lot of work to set it all up but ultimately we're calling instrument instrument client which is um part of the open telemetry instrumentation for HTTPX
In this particular case we have that capture all keyword argument which then does a whole bunch more work to capture header request and response headers request and response body and stuff like that. um okay I mean those are the examples of temporal I I would love to show people deboss which is an alternative lighter weight a bit newer framework but I don't know if we have I don't know what how long I'm supposed to have or whether it's time for a break or what we're supposed to do. Alexi, are we supposed to have a break? Can we have a break?