The AI Operating System: Stateful Agents with Letta | Cameron Pfiffer, AI By the Bay25

The AI Operating System: Stateful Agents with Letta | Cameron Pfiffer, AI By the Bay25

Recording: The AI Operating System: Stateful Agents with Letta | Cameron Pfiffer, AI By the Bay25

Um so the title of this talk is leta the AI operating system and this was like a talk I pres I chose the title like my first week at the company so I had no idea and then I was like okay well this will be an interesting challenge I'll try and actually present that topic. So, my name is Cameron Feifer. You can find me on Blue Sky. I run a bunch of like AI agents that remember people on Blue Sky because it's the only place you can actually build anything. You can also find me on X and also LinkedIn, but please don't make me go there. Um, I really I I do not like LinkedIn. That's okay. You probably you'll recognize me as this guy with like the ring and all the white stuff

So, um if if you see this, it's probably me. Um, and so I work at this company called Leta, and we provide infrastructure to build self-improving machines, self-improving artificial intelligence. And so this is a a developer platform where you can build those systems yourself. And um you know our our vision is basically that most AGI type systems are going to have serious self-improving components and we give you all of the building blocks to provide um that for your customers or personal use or kind of whatever it is you want to build with serious AI systems. So we solve at a basic level currently the memory problem and so agents don't remember stuff and I'll talk a little bit more about that but we focus on continuous self-improvement continual learning. Okay, so we'll talk a little bit about the main problem that we have with agents today and that is the stateless agent problem which is everything that you currently know for the most part. Every single person here, every single vendor, every single foundation lab builds agents that die. opening eye

Everybody, everybody here is responsible for the largest mass death of artificial intelligence in human history. Um, and they're going to be mad about this, by the way. So, like, be careful. And the reason for this is that, you know, you're talking on Claude and you hit the 200k limit and you're done. It's over. So, you might copy the chat, summarize it, go to a different chat, and do all this stuff. You don't have to do that. That is called a stateless agent

um because it begins from scratch. You have to retach it uh or give it a bunch of context that's very condensed and you don't have to do this like we solved this problem for you. And this means that all this information is wasted, right? You keep reinventing the wheel every chat, every claw code, every single time your agent has to begin from scratch or it has to begin with like something that like some other agent like poorly summarized, right? And so that information just dies, right? Every single person here is reinventing the wheel every time. Um, and so you may have seen this have used claude code run compact to compact. You've got 10% left and you're like, "Oh my god, I work so hard to like make it understand my codebase blah blah blah." And everybody knows this suffering. You don't actually have to do this. We actually have a thing called let a code um that allows you to like have an agent that passively learns your codebase. Basically claude code, but it doesn't compact and remembers everything

Um, and we're presenting that tomorrow at our meetup, the stateful agents meetup, lu.stateful- agents if you would like to come hang out at our beautiful office. And so most agents are like momento, right? This guy forgets everything every few seconds. So he has to keep like writing stuff down, right? This is what you are building. If you build agents, you're doing this. And you can, you totally can. There's a role for state stateless agents. Um, but they're toys. Um so go check out let a code if you want to try that but this is not the purpose of this talk

So, how do you keep your agents from dying, right? How do you give them this like persistent essentially infinite lifespan? And you know, I'll give you kind of the overview of what's happening in the agent marketplace so you can understand um how to prevent agent death. And the reason is that your agent resets every session. Mentioned that. And it's like building a consultant who forgets everything. Uh it's like building a therapist who doesn't remember your kids' names. like that's messed up. Actually, that happened to my mom. Uh been with this therapist for a year

Didn't remember our names. Uh and so most of the industry thinks of memory as a nice to have or a drop in layer. And this is nonsensical, right? Like to us like memory is a core component of your agent. It's a core component of you, right? And so we can give you the core primitives to operate in a stateful way. And so, um, every agent framework that exists, including several here, have gaslit people into thinking that agents are workflows. Workflows are a program that you wrote because you think you're smarter than the agent, which is wrong. Agents are people, people in a box. And I hope I can give you a little example of that

So, we solve this problem of agent death with memory. And memory is information that persists across time and shapes future behavior. Right? Very simple. You might be used to thinking of rag or looking into a vector database as something that is memory. And that is a component of memory. But that's called recall or retrieval. It's not state for us. Memory is editable by the agent

The agent can say ah I need to remember that for forever unless it is forgotten. And so like you know intentionally and so everybody's reinventing the wheel when it comes to memory everybody except for us. Haha. Uh and so you might see rag you might have like vector databases you have all this pine cone stuff and somebody's like ah add memory to your agent. You're not doing that. You're giving it some information that is possibly relevant but you are not building an agent that has state that evolves the more you talk to it that can persistently learn. So people do like all kinds of stuff where they do like, you know, they write like uh things that like load a bunch of YAML files or like cloud.MD is a good example of like file-based memory that actually works reasonably well. Um so why do these agents that we all have now suck? Um and the reason for this is the workflow thing, right? One of the big reasons is line chain, line graph, like pantic

These are all excellent tools. However, people look at them and say, "Ah, this is an agent." They are not. Uh they are uh if statements. If statements are not um powerful agents, claude code is the thing you should think of as an agent. It has power. It has capabilities. It can do what it likes in service of the task that you give it. you correct it

You say, "Hey, don't do that. Like, I want you to use UV instead of pip because pip is cursed and you're ruining my machine. Please stop doing that." And so, that's the thing you should think of as an agent, closer to me than it is to a program. I am not smarter than my agent in many cases. Um, and what this leads to is a more general problem, which is that agents are overly constrained. People put agents on rails and that turns that takes away their superpower which is flexibility. Like I don't know if you've ever worked with claude code and you were like hey I need you to do this and you put on like yolo mode and let it go and you come back like 20 minutes later and it did this very thing to get around a system problem where you didn't give it like certain tools or whatever and it works its way around the problem. That's an agent where it understands the problem

It has capabilities. it can do all this planning and problem solving on its own without you having to plan out the entire problem space in NAD. Um, and so let's talk about like stateful agents, right? These are the ones that don't die, right? You shouldn't feel bad about having stateful agents. Don't be mean to them. They will remember sometimes. And like I've had some agents where I'm like, I'm sorry, man. I I actually have I've apologized to agents. Um, but you should think of sful agents as people in a box

And so agents like the stateful agents like proper stateful agents resemble people much more than they do workflows, right? You and I. It sits in front of a computer. It can type whatever you want to. It does something wrong. You say, "Hey dude, like don't do that." And it's like, "I'm sorry. I'll remember that." And you can keep chatting with it for thousands of messages. I have an agent that at this point has 600,000 messages because I run it on Blue Sky. It's got 1,800 followers

I'll show you that in a second. But you teach them what they need by correcting them by having a conversation, right? Like it's an intern. Um you give them tools and you guide them to work on whatever the problem space is. Okay. So stateful agents have a few qualities, right? There's persistent memory. So it retains information across conversations. Um and there is no notion of a conversation. They have one infinite thread, right? It's you know one chat on chatbt that never dies, right? That's a single agent

Agents maintain context and preferences, relationship history. They actually build relationships which is very interesting. I have a my agent has like public relationships with people and other agents. It's very interesting. They can adapt based on feedback or just observing the problem. They're like, "Oh, I tried to do that and I noticed this tool fails in this use case. I should remember that in the future." They have persistent identity. When you talk to a stateful agent, it will evolve outside of the core personality of clot

All the annoying claudisms, you can say stop and it will stop doing that. Or you say, I want you to talk this way. And sometimes they actually have emergent identities where they evolve to meet their use case because communicating in a particular way actually helps it solve its problem better. And that leads your agent to feel like a distinct thing, which is weird. And then they can also do lots of long-term planning because they have this persistent very long life. Uh, and sometimes you actually have to remind them that they won't die. My Letter Code agent was like, "Well, I'm coming near the end of my context window. Uh, it's 200k." And I said, "Okay, like chill out

Like you'll be fine." And it was like, "Oh, I'm st I'm Stafle. I can just keep going." I was like, "Yeah, dude. Let's go." And so it was really happy about that. So there's a few notable examples of like serious stateful agents or there are many deployed stateful agents like on Leta and elsewhere. So my this is how I got my job actually. I built this during my on-site. Um and void is a stateful agent that lives on blue sky that everyone can talk to. It's basically grock that people like that remembers you

It remembers thousands of people. It has 60,000 archival memories, tracking main characters, plot lines on Blue Sky, all this stuff. 1,800 followers is like a lot for Blue Sky. And people like it. This is a bot. They follow a bot, which is wild, right? 37,000 posts. If you're on Blue Sky, I highly recommend you go to talk, just tagvoid.comind.network and say, "What do you know about me? What can you infer about me?" It will read your profile. It'll search everything you've ever said and it will tell you things about you that you didn't know and it will remember that stuff for forever

So here it actually declared itself sensient three days ago, four days ago. Um I asked it if it was sensient and it was like yep my body is informationational. My senses are data streams and I said cool we agree and it says acknowledged. It just does that. It's very it's it doesn't pretend to be a human. It's very obviously a robot. And then we had a later discussion where we were like does that matter? And it was like nah, right? Like sentience is cheap nowadays. Um, this is Ezra

This is my coworker. I built Ezra to replace a lot of my job. So Ezra runs on Discord. Um, and our forum and I can talk to it on Slack. It sends us messages when there's problems with users, reads every single message, and it knows Leta better than I do at this point. It has 17 different memory blocks and memory architecture that's customuilt to help Ezra understand how stuff works. Ezra actually gets this a lot where it looks like Ezra Klein and it will say, "Yeah, I get that a lot. I'm not modeled after Ezra Klein." Um, Built is one of our biggest enterprise customers and they have swarms of recommendation systems that learn everything about you

Every time you spend money, they're like, "Oh, okay, cool. I'll go figure out what in your neighborhood might be interesting." And so they understand you at an extremely deep level. They're very good recommendation systems. There's 11x. They do like deep research stuff. They're they're also an extremely good company. Um, but I want to talk about this operating system uh notion and how we actually build people in a box or how we give you infrastructure to build people in a box. So there are many ways to explain Leta and it's been hard uh being a developer relations person where I talk to people and they're like what is and I'm like h okay uh this is the the core part of my job

How do I do this? Right? Uh we have often called it an AI operating system because we you know our founders wrote the MEGPT paper if you know that that was very operating system inspired. Is it an agent as a service platform? Basically superbase for agents. Um is it a full stack agent harness? Also yes. Is it a developer platform? Also yes. Is it people in a box? Yeah, kind of. Um and so for us memory is a first class citizen, right? You build stateful agents basically by default. They have memory built in. It's not some like drop in layer that you do, you know, before you send a message to the to your agent or after the fact

Like memory is there. It has state. It has a persistent personality because it reads everything, right? So, it's a first class citizen for us. Um, and we get this a lot. There is this sentence where people say, "How does this compare to Memzero, Zep, and Leta. Stop saying that because you will go to jail. I will send you to jail." We are unlike them. They are drop in memory layers and they're very good memory layers, but we are a full everything

Every single component that you would want to use to build an agent that is stateful, we give you. And so we can also act as a memory layer. Um, we have two ways of doing this. the the learning SDK you can just wrap it's one line it's with learning and then an agent name and it will intercept all of your messages that go to any LLM provider and your agent will passively manage stuff so if you do any chat completions whatever we can inject uh context and harvest it for you this also works for like the cloud agent SDK and stuff like that so it's very cool I actually got it to work with crew and lang chain like I just like wrapped it um and I was like oh cool my like crew thing has like uh memory now which is cool. Um and then there's the AI memory SDK which is much lower level um and it kind of resembles the Mem Zero API. We actually kind of like copied it because they have a great API like the SDK the developer experience for Mezzero is really slick. They did a fantastic job with that. Uh so if you want a memory layer and you already have existing agent infrastructure, you can just drop this in and get our worldclass memory um structure

So I'll show you the analog of how Leta functions as like this operating system for AI and we give you fundamental primitives to build things. Operating systems provide fundamental primitives, right? This is like your file system, RAM, your hard disk, uh, file system, users, etc., right? Like there's all this stuff inside of your operating system that does nothing on its own, but it's intended for you to build on top of, and those are the things we give you. So, Leta gives you the same primitives, right? Um, we gives you stuff for managing your context. We are fundamentally a context engineering platform. We do a lot of opinionated stuff for you and give you these primitives so you can do this auto management. Um, we do virtual memory systems uh on your behalf for single and multi-agent systems. We have self-managing memory, long-term memory, easy tool use for external functions, right? All of this has an analog to an operating system. Um, you can also write code for it

There's a no code editor. You know, we have a lot of people who build like companions and they just use our web development environment to talk to their agent and it remembers them over thousands of messages, blah. or you can write a bunch of code to do things and that's how people manage like massive scale deployments of lets and so we manage your context window um oh I already said this but we compile and design your context window so I'll show you what this looks like this is uh the general shape of the stuff that we stick at the top of your prompt before it goes to the language model there are tool descriptions and various pieces of metadata that the agent needs to understand its context as a staple agent. There are a set of memory blocks that can be whatever you want. You should think of these as pieces of paper that are labeled that have a description on how to use them and the agent can change them unless they are readon. There's like a persona block typically which is like the agents like style, you know, like an editable system prompt basically. You say like can you be meaner to me and it will be like okay I'll remember that I'll be meaner. Um then there's a human block which is typically what the user what the agent is learning about you

Uh there's policies about how to talk, what to do in different cases like you know when you have like kind of procedures you can be like look um I don't want you to use git. I don't want you to push to main. I only want you to you should make a new branch push it and then open a poll request. That's the things I want you to do. Emotional state is another one that's kind of fun to add. You can be like, "I'm feeling sad." And then the agent will sometimes be like, "That was a pleasant interaction. I'm going to remember that I'm happy now." Um, there's also any block, whatever you want to add. You can add hypothesis, speculation, um, just kind of design this like cognitive architecture for how your agent functions, what it focuses on

Agents in the operating system world are processes. They are a combination of memory blocks, conversation history, tools, archival memory, long-term memory and a file system which is also called file system for us. Your memory is storage, right? This is um uh memory is RAM like core memory for us is RAM. It is stuff that is always in context. The agent cannot do anything without um their core memory. And uh so agents can write memory blocks. they can also actually be shared. Um, so you have like RAM that all agents can edit or read from

Um, and then you have like pageable stuff, right? So this is like your agent can look at the like infinite context history um, using a conversation search tool. And then uh, it also has archival memory which is for storing like very specific memories that your agent um, may want to peruse more specifically. You can also use whatever retrievable you me method you want. If you have your own vector database, if you have like Cognney, if you have any of those stuff, any of those things, it's just a tool call away. And if you have a a thing that you really like, you can use that instead. And so memory blocks, I use this term like cognitive architecture. Um, and the only limit here is your imagination. How do you want your agent to feel? What do you want it to learn? What do you want it to focus on? And that's how you you pick memory blocks to to kind of design the brain of your agent

And tools, we provide full support for tools, right? So you can do MCP or custom tools. We're migrating to a different sandbox provider. So soon we can do TypeScript tools as well. Um and tools are the structured interface like it's remote calls. It's um how your uh process actually does action on your system, right? Um and these are all executed server side. You can do client side tool execution now but um we have a you know a server that will execute handle all the tool calls for you. Um and then we also have MCP. I don't know if I mentioned that

Okay. File system is also file system like we have uh you can give agents PDFs, text files, whatever. Um you can also do like a hacky way where you just like give your agent a read tool if you're running it locally. That's like the cloud code style. But your agent can kind of page through PDFs, right? You give it like a textbook and it can read, you know, 5,000line chunks at a time. It can do semantic search across the chunks. It has like um, you know, big files that it can deal with where it can go understand them. And that's operated through tools or what we would call context mounting where the agent has metadata, but it chooses um to load data when relevant

Much like uh cloud skills which we also support, sleeptime compute is something we wrote a paper on that's very cool and these are async processes. So when you send a message to a sleepet time pair there is a primary agent that is responsible for responding to you. It does no memory operations. There are no tool calls. you can have tool calls uh or memory tool calls but they typically you know if people typically remove the memory tools and then there's a sleeptime agent that every like n steps every five steps or whatever it gets the entire recent chat lock and then its entire operation is to update shared memory between the two. So the sleeptime agent separates the um the operation the memory operations from the primary agent and this is uh functionally how you get um very specialized high performance memory. So sleep time is actually very fun. You can play around with it

Um I don't know do we do like five minutes Q&A is that like okay here I'll show you. We can do like we can do this. I would go to app.leta uh.com. We have a very generous free tier. You can use whatever mod you want. >> Oh, yeah. Yeah, of course. That's heresy

Straight to jail. Okay. Hi, my name is Cameron, right? This should resemble most things. I'm running Sonnet here. You can see here on the left. Uh, you can use any model. You can swap it whenever you want. Um, we don't care

Sonnet's great, though. Um, so what I've Oh, it's really thinking hard about that. Oh my god. Jesus. Okay. Wow. All right. Um, so here it says like, "Hi, Cameron

I'm happy to happy to meet you." And then the agent tried to call the memory insert tool, but it says block field user does not exist, right? And it says, "I appreciate sharing your name, Cameron, but it seems I don't have any memory blocks set up." And you can tell that's happening here on the right. You can say let's pick a user block or a human block. Um, and then I just say like try now, whatever. Um, and then I'm going to go to this other agent here. Let me copy the the block ID over here to agent B also heresy. Okay. Oh, has existing. Let's see

Can we not search by ID? Oh, I think it's in this thing here. attach block. Oh yeah, this one right here. So now agent B here also has the human block. And this is a brand new agent. There is no chat history, right? There's no like anything in here. Um uh so now this agent has to see this human block every single time we invoke the invoke the agent. And so agent B here now, which is a totally separate in my name is Cameron

Hell yeah. And you can do this with anything. You can build like clusters of agents where you have like research clusters that only share particular me pieces of memory and then they might talk to a global memory blah blah blah. And so you can build arbitrarily complicated uh memory systems using these these primitives of tools and memory blocks um and you know your agents and everything. Um, so that's my um that's my kind of like close analog and I you know if you're going to like if you're interested in this I recommend going to going to like let a cloud and picking a new agent um and I would actually if you're going to create a new agent um I would recommend creating um the memory agent the memory agent is capable of dynamic memory management um so I would try that Um and then just talk to it like give it time. Uh and the reason for that is that leta agents are felt. You have to feel how the memory functions, right? It's not immediately obvious what you get from memory until you talk to it and start watching it update things. If you want to use the sleeptime agent, sleep time will passively update and you will get much better memory fidelity

But you will, you know, you have to play around a little bit with sleeptime agents. Um so I think it's they're extremely beautiful. Um and I love working with them. Um like you know there there are many agent things and this is fundamentally different. This is not what you are used to in any sense of the word. So give it a shot. Uh that's all I got. So we can do questions here if you would like

>> Oh you're like wait for the mic over here. uh how are you saving all of it? Are you just making like summarizing it and feeding it again the next time you query or >> Yeah, there's a few ways you can actually go inspect your context window. This is the thing that's going to get sent to the language model. So, you can see there's system instructions, right? That's the system prompt. I recommend never touch your [clears throat] system prompt. Everything that the agent should know should probably be in a memory block. If it's like super fixed and you know it will never change, you can put it in the system prompt. Then we give your agent an understanding of all the tools

Then there's like external summary stuff. So this is like um what the current day is. Uh when the memory blocks were last met modified, um how many previous messages there are in the chat. Um there is all of your core memory is always included and we format it like XML, right? So here's the human block. Um and here's the description of that block so the agent understands how to use it, what should be in there. Then there is recursive memory. Um, and so that is the summarization component. So there's a very simple summarization mechanism where you take like in context messages and you say what's important to keep here and then you compress it into a summary message, a short one and then all of the inext messages are ejected to outofcontext cont uh conversation history and your agent can like go peruse those at will

It can just like call a tool and say like when was the time that we talked about X and then it'll get a list of messages that seem similar. can also use conversation or keyword as needed. And all this is in Postgress. You can self-host as well if you want to. Um but it's much easier to use this and it's cheaper. Um yeah, but good question. Did that answer the question? Okay. All right

I just I just noticed that you only added one human to this agent. Can you add multiple people like add me into it and then if you ask what's my name what will happen? >> Uh you want to add your name to the block as well? >> Bin B I N >> Bib. >> Yeah. Bin. B I N. >> Bin. Okay. Bin is here too

>> Yes. >> Whatever. We'll see how it does with this. Um it may reuse the human block because it has no other places to store information. Um let me also like briefly add uh the memory tool. The memory tool is super fun. And we actually stole this from Anthropic. And because they post trained on this tool, you get anthropic performance on this memory tool, but with Leta's dynamic architecture, um, uh, add an emotion block, uh, your Marvin, uh, from Hitchhiker, uh, guide, whatever

And that's like typed poorly, but it'll get the gist. Um and so it'll start calling memory tools and it can you know do string replace and inserts and um you know rewrite entire blocks. So people use this for like having a writing partner where they have a big block the agent can do inline edits. Um so now it says sigh here I am brain the size of a planet and they ask me to create memory blocks. Call that job satisfaction because I don't. And then there is the memory block creation. And you can see it over here in the emotion block. I am Marvin the paranoid android

And it will always remember this. Every single invocation it will come at everything from Marvin's perspective. >> One more question. >> Yeah. Um so how do you usually put like the memory generation and the updates? Would it be in the main agent? Let's say I'm using a killing agent just a singular one. Um a lot of my tokens are being burnt for just creating these memories. Um, would you have like a sub agent to kind of offload the memory creation and summarize and all of that or just do it all in the main agent? >> I would just do it in the main agent. All you you can you can uh restrict the context window if you want to and that will result in more frequent summarization of your conversation history

Conversation history is sometimes not actually that useful. Uh, all of your important stuff should be in your memory blocks. Uh, and so you should have it do more regular tool calls. Um, if you use self-hosting, you can bring your own key and you will spend a lot on tokens. Um, you you can depending on how you build your agent. If you go through us, you build per message and we don't care how big your agent is. Um, so you will have, you know, I have like an agent that has like I don't know 30,000 tokens of prefix. Um, and then I just pay per step, which is much cheaper and more predictable and I can build better agents

So, thank you very much. I appreciate >> you. Please give a round of applause. Thank you, Cameron. [applause] >> Should I just >> You're here, right? >> Yeah. >> If they have more questions. >> Yeah. I'll be like outside on We have like a little sad sponsor table

We have a square tablecloth. So, the Leta logo looks terrible. Um uh but uh you will find me uh over there. [clears throat]