Durable Agents: long running AI workflows in a flakey world | Samuel Colvin, AI By the Bay25
Yeah, thanks. Thanks so much for having me and for introducing me and covering most of the things I'm going to talk about already. Um, yeah. So, I'm Samuel. I'm the creator of Pantic, uh, the open source library and now run, uh, Pantic, the company. Um, today I'm talking about um, durable agents, longrunning AI workflows in a uh, unreliable world. This is obviously part of a wider topic. I actually wrote this slide before what Alexis just said, but um yeah, on the wider topic of AI and engineering and reliable AI in particular, but what I'm going to talk about on um durable agents won't make much sense without a bit of background on what we do Pantic and and basically our world view that leads to to what we're doing and why we care about about reliability
So, what is Pyantic? I'm going to put this in here and hopefully you can still hear me. Um uh so this is the first the first part of what Pyantic does is is Pyantic validation. The OG library that started it all that I started way back long before AI in 2017 which does something very boring and rudimentary. Whenever I have to speak to VCs and explain what we do, they're always slightly disappointed by how mundane what Pyantic does. Um but basically it takes typins which are around in Python and enforces them. So you see here we have a have uh the city model um with three fields. We pass it in some data which is uh valid but obviously isn't all in the right types. So although the date is a date, it's not in the form of a a Python date object
The location is not a tupole and the values inside it are strings not integers. But if we run this code um uh we get back a valid instance of of city. So, podantic does does the val validation. It does coercion which is uh I'll come to why that's important in AI. Uh we also do stuff like JSON schema and serialization which I don't show here but but which are also relevant. Now, Pantic is used, I guess, in three main ways, in uh web APIs, most notably fast API, um in everything else in in in Python development. Basically, anywhere where you care about type safety, but your code has to interact with the real world. Uh people find themselves using Pyantic
It's downloaded. It was downloaded 560 million times in October, which makes it the 20th most used Python package or most downloaded Python Python package in October. If you remove what are basically transitive dependencies, we're somewhere in the top five. Um, and then the the third thing it does is validate structured outputs for MLMs. That's why it I guess has lots of lots of attention right now. And so that brings us on to the second part of what we do, which is pyantic AI. So, Pyance AI is an agent framework um but built on the idea of of effectively what what Alexi was talking about of of reliability over hype. Um there are an awful lot of agent frameworks
I I have a kind of passion for collecting examples where people who claim to be excited by what models are going to do claim to think AGI is about to happen do things which fundamentally bet against the model. And one of those things is building systems that are not type safe. If you think AI is going to write any of your code or analyze any of your code, then type safety is absolutely profoundly important because it's the it's the most useful thing for coding agents to basically check their working as they go along. And so um uh Pyanskai is basically the own only type safe agent framework. Um you can't see it here in markdown, but agent will be generic in the output type and in the dependencies type. And so in this very simple example, I don't have tools, but if I had tools, the dependencies you can access within those tools are typed. We work very hard to to do that stuff correctly. Um certain other agent frameworks have basically copied and pasted what we have done, but they've mostly broken the type safety because they didn't understand why that mattered, which I find extraordinary
Um but yeah, we can we have a very simple example here where again we're going to we're going to generate uh an instance of this structured data. Fun fact, OpenAI's um uh JSON schema for for tools doesn't accept tupils. So we have to use uh inline typed dict for the location. Um but yeah, we're effectively again generating that structured data from in this case an unstructured sentence. And so if we go and run this example, an AI is on our side, which it may or may not be. Yeah, we get we get back again uh a valid instance of the of the data. So we're what we're doing under the hood here is we're using tool calling for structured outputs. Pyance AI also supports native structured outputs for models that support it and prompted structured outputs
I won't go into lots of detail on that now, but come and talk to me if you have have any more questions. Um and yeah, we do a lot more agantic stuff, a lot of integration with MCP uh and the the durable execution which I will get to get to a bit later. Third bit of what we do. So, Pyantic uh validation and Pantic AI are completely open source and MIT licensed. So, we make money out of first of all pantic logfire which is our observability platform. So, you'll see the exact same example here. The only difference is I have added these three lines to to implement logfire and I've set up locally connected that to to my project. And so, if I go and run this example um you'll see I printed run the wrong example I think
Uh if I run this example, you'll see the URL is printed out. And if I open that, you see that we have a in this case very boring trace because there's only two spans, the agent run and then the um call to to OpenAI. And then we have the the result output. So you'll see here in my code that I have used logfire to log the result um of the result.output, which is the instance of city. Um, we do clever stuff to introspect stuff like strings and allow you to to do logging as if it was any other as if it was standard print but we'll display it nicely in the terminal but but more powerfully we'll give you access to the actual structure data which you can then go and query on uh inside logfire but also this this tracing view which we'll see the the value of in a minute. So yeah, this is full observability, logs, metrics, traces, but we are used more and more by AI mostly because that's where new projects are starting. Um, and then the last bit of what we do, which we only announced last week, is Pantic AI gateway, which is effectively a way to to use one API key to connect to all the models. You can either bring your own key or or basically buy inference off us
Um, and so the only change I've made to this example is putting gateway slash at the beginning of the model identifier. And if I go and run this and the Cloudflare outage has res has restored, which may or may not be the case, we may get it to to run the same code. Yeah. And you see we get the structured output. The the nice thing here though is without changing any keys, I can literally change this to OpenAI uh GPT5 or I won't want to wait half an hour 4.1. And yeah, we get the we get the same result. So, pinantic AI has great support for switching model in one line of code and gateway basically takes the next step of that of allowing you to do orth and uh rate limiting and observability in one place as well. So, what's the point of all these things? The point is that in our opinion AI is still just engineering
There are an awful lot of people out there who for fun and profit have tried to persuade you all or tried to persuade lots of people that AI is so profoundly different, so new and so creative that it doesn't need engineering practices. That AI somehow runs in a vacuum and is independent of all of your engineering needs. That is, in my opinion, Am I allowed to swear? Well, anyway, I have done. Um, I think that AI actually makes engineering even more important because AI is so profoundly important to your business or to society and it still it still runs on all the same rails as existing engineering. It's still you still need to care about uh the database calls that are being made in your rag application or whether or not the orth header is set correctly uh or the latency of some HTTP request whatever it might be as well as the stochastic nature of the the model itself. So you need good system design. You need to be able to refactor your app quickly. So you need type safety
You need observability to work out what your existing application is doing as well as what the hell this AI is doing. You need durable execution because where we could have been reasonably confident in the previous era that our database um query which was fundamentally the only slow bit of our application should return in under 20 milliseconds now we have this call to an LLM that might take 20 seconds might take 300 milliseconds. Uh I know with chat GPT5 if you ask it to write a long story about a short horse it reliably takes 173 seconds to return. So this is not we cannot just assume this is going to uh come back in a couple of seconds and so so that's where durable execution um comes into this. Durable execution um is a way in a unreliable world of unreliable network requests, unreliable end models of nodes which are running our task which might die at any time to build a system that might just succeed or indeed ultimately where we could have some confidence that it might succeed. Um so it deals with as I say intermittent uh network failures with uh a node that might go down halfway through a run. Um and in the case of temporal in particular, which is probably the leading durable execution framework that that we integrate with, you can also have workflows where you basically write procedural code that can run over hours or weeks or even years. You basically sleep and they will take care of basically pausing that process and then resuming it sometime in the future when you want to let's say send the next send the next um uh happy birthday email in a year's time or send the next thank you for signing up email a week after
Um so we support with impantic AAI um three uh methods of doing durable execution. Temporal who are probably the the best known and big incumbent in this space. Deboss which is a more open-source um based uh durable execution framework based on Postgress although they're a company again so they have some commercial bits of what they do. uh and Prefect who are a I guess previous generation originally a workflow orchestration platform who have realized quite rightly that what is now called durable execution was previously called workflow orchestration and that they can they can do the same things um I'm going to the example I'm going to give today uh works on on temporal and temporal is based on this idea of workflows workflows nice thing about presenting from this is I can fix my typos as we go along uh and activities so I have to say I spent about six months this year talking to Temporal, talking to people about how we did durable execution and how we were about to integrate it into Pantic AI and I never really understood how their thing worked. It all seemed a bit magic to me. So I'm going to hope to get you from where I was six months ago to where I was about a month ago in the in the space of this talk. Um this uh mermaid diagram written for me this morning uh by AI may or may not help you. Hopefully running the code will will do the other bit
But the idea of workflows and activities is that we have uh the the core um sequence of our application is entirely deterministic. We don't do anything random. We don't do any network requests. We don't do any file system access. We don't access a database. we work through and we do a sequence of deterministic things and then we have activities which we call out to to go and do those uh IO bound tasks which may or may not uh return differently and then effectively if we want to rerun our workflow if it has crashed let's say and we want to rerun it we basically record or cache the answers to each of our activities uh and we basically number them off one by one such that when we rerun our workflow we can basically slot in the results and we're not doing any clever what line number was it or some kind of reference to what the input arguments were. We're just remembering one by one what the different activities were and so we can slot in the results and that means that we have to have a relatively well we have to have a very clear delineation between where the workflow is and where the activities are and we need to make sure we do nothing that isn't entirely deterministic within our um within the workflow. um that in involves generally editing your code quite a lot or or building it in a very particular way but we have effectively gone and done that hard work within Pantic AI to make the to to separate the workflow from the activities
So you can run a pideantic AI agent and it will uh the the the the core loop of the agent of an LLM calling tools in a loop will be the workflow and then every call to an LLM or to a tool will be wrapped in an activity so you don't need to go and do that work. So we're the only agent framework as far as I know who does this properly. OpenAI agents does about 50% of the work which means it makes them it completely unusable in my opinion because they don't wrap tool calls in activities and virtually every tool call will be doing IO. Um so this diagram is all very well it gives you some idea of what's going on but I think to actually understand a workflow we need to look at some code of a of an actual example. So those of us those people here who were in my workshop yesterday will uh recognize this code. Um this is two AIs playing 20 questions against each other. So um we have an answer agent which uh knows some secret value. Um and which will return structured data in this case an enum of of different outputs
This is more than just yes or no because AIS are surprisingly bad at this and I want it to actually get to an answer. Um but it's kind of um categorical answers. Um and then we have down here the question agent which has a bit more uh bit more context bit more instructions. Um and then I talked earlier about how we can register tools with an agent. So we register one tool here with the questioner agent uh which is ask question. Uh I've put in here uh a random failure to simulate the idea of network failures. Given the Cloudflare issues today, maybe I don't need this. Maybe we'll get real ones
But to be sure, I've put in that 10% of the time we'll get we'll get an error. Um and then within this this question the the ask question tool, we are calling um the answer agent with with our question. We'll get back here. As I said earlier, we care a lot about type safety. So if you look at the answer agent, it is generic in the in the answer type. So then when we call it and we get back the result, result is also generic in answer. When we access uh result or output, it is both in typing time from from the type safety work and from pyantic doing the validation guaranteed to be an instance of answer. And so we can return answer here
Um, yeah. So, we we we have that we have that nice type safety and we can we can uh have confidence that our app is actually going to work the way we want it to. If I went and asked Claude Code or coding agent of choice to go and um refactor this, it could run typeing and be confident that it had got it right without you having to spend 20 minutes running your code to have some chance of it working. So if you look at this this agent, it's completely unchanged from the example if you ran it without without temporal without the durable execution. The one difference is we wrap it in this temporal agent which is effectively gives you back uh an abstract agent um which will have the same signature but but uh inject the correct uh activities etc. Uh, and then this bit and I've told Temper all this. They're they're they're set up in Python to basically define workflows and run them is a little bit ugly but it is what it is and it's not actually that much code but we basically define uh this workflow which has a particular decorator and a run method. Um, and then when we want to come and run our code um we set up a client which is going to connect to the temporal server
Temporal have a local server you can run. They make their money out of out of the cloud server that will take care of stuff in production. And then we run our worker. Um we have have to register some plugins for the two agents which takes care of the activity stuff. Again we register one workflow as as it controlling this one workflow. And then ultimately in this code here we go and start off a workflow to run the 20 questions. When this comes up it will in this case because we we execute this code execute workflow. It will start a new workflow
It will also resume any existing workflows. So if we get halfway through running a workflow and uh our application dies or I kill it, it will start another workflow. And this this code here is basically for the sake of the demo. I can run it and just wait for the result of an existing workflow because if I run this uh multiple times to show you it continuing a workflow, it will start another one and it gets very confusing what's going on. So this is only you wouldn't need this code in production. you would just run the workflow and it would take care of whatever incoming tasks and of of resuming any existing ones that are going on. Um, and yeah, I'm just printing out a specific job ID in this case so that I know what to what to resume. And I've set it up with with logfire uh because logfire um is just open just a open telemetry receiver and temporal emits open telemetry, we'll get a nice trace of what's going on
But if I come over here and I run, let me just check. So, so I'm running temporal locally in this case to the temporal server locally in this case. Uh, is that big enough that you can see what I'm writing? I'll assume the absence of a no is a yes. Um, and if I run this and again, AI and children never go together in in demos. Um you can see it it working through and it it's asking questions and getting back an answer printing them out and this is this is uh chugging away and uh you see we get a broken exception just just on Q and it recovers from it. That's temporal taking care of effectively errors within activities and continuing the workflow retrying the activity. Um uh and we can come over into logfire and you can see the trace of what's going on here. So you can see the individual steps of the the questioner uh agent and then within that we have tool calls and within the tool calls that's probably quite small for you but I uh I I'm loathed to zoom in and make it hard to see what's going on in general
Um, we have some nice stuff like we we can see the token counts and the price of individual steps and of our entire run. Um, and we can see maybe I can zoom in here and just show you roughly what's happening within within a given answer agent. So, it's it's got the system prompt so it knows the answer. It's given a question in this case, is it alive? And it replies kind of. So, it's done it's done a reasonable job. And we see this continuing to to run. And indeed, it has succeeded and it discovered the answer. But let me run it again
and then be mean to it and um uh kill it some way through. So, we'll start it off again, new job ID, and it's going to start asking questions. We got to step three, let's say, and then let's say the node that's running this particular workflow goes down. We've stopped. We we if we were in we weren't in a durable execution context, we would now have to restart that run all over again. We would have to pay the time of running it again and we would have to pay the the inference again and and I guess the compute again. Although that's that's kind of minor. So let's just rerun the same thing
Um in this case, as I say, I'm going to wait on this particular job ID just so that you can see that instead of starting off another job, we're just going to hang on the result of this particular workflow. And you'll see immediately um well you will see quickly that we skipped steps one to three and we went straight on to step four. Now we didn't actually skip steps one to three. Instead temporal took care of providing the results into those ones. So if we come back here and we look at this this workflow, what you will see where it ran is that the the answers to those first bunch of tool calls and the first bunch of calls to the model took 11 milliseconds, 2 millisecond, 1 millisecond. Now that's obviously not actually calling the LLM. That's just looking up the cache for the answer. So it's able to basically recover from failure by basically replaying that first set of results from the cache and continuing
Um and oddly it's answered it's got to the answer this time. Yesterday when we were running this in the tutorial it kept failing completely and never finding the answer and getting itself confused. Um but this this time uh um it's worked pretty quickly. So, Temporal is, as I say, the like the 300 pound gorilla in the in the durable execution world. They they made their name out of uh Uber using them for all of the updates to trips and they're now used for the deep research agents of definitely of OpenAI and I think I think Anthropic again. Um but as I say, there are other there are other options. They're not the only one. I think there are at least two other durable execution frameworks who have PRs against pyanci
Um so and we're happy to support as many as we can. Um I think it is more the like well first of all the specific principle of using durable execution to make uh LLM applications reliable and not having to to restart them when they go wrong. This this task as you saw took uh a few minutes but if we're running something that might take half an hour we really don't want or or hours we really don't want to have to start it again when it fails. But even higher level than that, this idea of actually following engineering principles within AI and not thinking that AIs are somehow magic and don't need um good engineering practice. And that, as I say, is stuff like um type safety, observability, durable execution, um separation of concerns, using protocols where they work properly like open telemetry, like MCP, uh and others. Um I have That's all I have. Thank you very much. I may have some time for questions
[applause] >> Yep. >> Sorry. Uh yeah, the slides are um here. Slides is a perhaps a strong term. The markdown file and the code uh in this repo along with my other talks. >> Thank you.