Challenges of Ambient Agents: Building Governable AI Systems at Scale | Fred Patton, AI By the Bay25
My talk is about the challenges of uh ambient agents. Um I'm a developer advocate at Offzero and background is in event processing. So um ambient agents are particularly exciting for me. So um if I say anything resembling a forward uh looking statement, I have little inside knowledge so disregard it. And so yeah, so framing the challenge. So um you know there's there's a lot of [snorts] of um complexities and and issues with uh with co-pilots just you know reacting to prompts. And so when it comes to ambient agents um you know these issues magnify. And so we're going to be looking at um what ambient agents are, why they're a problem, um and you know, how do how might we mitigate those challenges? So, you know, sustainability, most AI projects fail
Uh it's not even funny. And so um you know like with the cap theorem you can just two of the you can choose um two of the three things and make your choices but when it comes to um ambient agents you know they have to be fast enough to matter, cheap enough to run and governable enough to trust. There's really um no negotiables there. So any one of these will wreck your project. So um Harrison Chase um came up with the term ambient agents uh you know these agents running in the background and listening to signals. So typically an event or something and taking action and so that's quite different from you know the request response pattern of your chatbot. And so um [clears throat] the um the fact that um intent is no longer explicit is going to create a lot of problems. So we'll jump in there
So yeah, admin agents must act um whenever conditions arise. Uh it could be tracking differences in logs, deployments, tickets, and you know, if we think about the context window, the more context in there, um you know, the more ways to go wrong. So that's particularly a problem if you have agents with a kind of a wide purview and grabbing signals from everywhere and needing to take action. So um they need a lot of help there. So they have to be able to interpret uh the noisy signals and really just get the minimal context that is um that's valid and then they have to be governable right when we have um processes running in the code. Um someone has written that code so we we know um what that code does. But with an agent we know what we told it to do but we don't know how it's going to do it. So, as we'll see, determinism makes this a lot more tricky
So, um things boil down to uh the intent. It's now implicit. So, um it's not it's not like a user asks for something and the amate agent has to take action. It's based on um looking at things over time, trying to remember what makes sense and taking action. And again the lack of determination determinism and then uh the problem with authenticating and and authorizing. So uh here's just a a small example of of you know how intent gets revealed. Um an agent's looking at your your logs and there's bad signals in there. Then it's going to go and correlate that with recent deployments
Okay, it sees something, it can take action, roll it back, notify Slack, open a channel. So, that's kind of example. You you you're not going to have a bunch of if then statements in there because it's it's going to be um a lot more fluid than that. So, it all comes down to context. That's why context engineering is is so important. Um you're having your event streams coming in. Uh they're raw signals. it's not like ready context for an agent to take hold of um and in that event depending on what the underlying issue is what parts of that context in that event may vary
So you you have to basically build some kind of heruristics to understand uh for that state coming in and what that event seems to call for what aspects of that context you pass in. uh we'll use things like aggregates, counters and sketches to kind of reduce the amount of information coming in. So we don't actually we just lose we just want to lose noise but maintain all the vital signal and then we want to present to the LLM a reasoning case. So clearly connecting your um AI agent to an event stream would be a very bad idea especially financially if you were your pocketbook. So how do we deal with continuous context? [snorts] So yeah, that's a fire hose and that's ne you want to be spending tokens on. So we need to basically um maintain efficient state and act fast enough to stay relevant. So what we're really looking at is that once you have um an event bus involved, it's a stream processing problem. So all the tools we're using for stream processing to basically come up with real-time state we absolutely need for an ambient agent to be able to make effective decisions in a tractable way
So one way to think about it is is this context aware governor pattern governor pattern um you've got these raw streams and we know uh we know the event stream is not sufficient. So we need code that can basically um pull the context from that event stream structure this structure the states. So there's got to be stream state because just a particular event is probably going to be missing important context. Just think of out of out of order messages and um sometimes it's a pattern of things like this event might be okay normally but if it was preceded by another event or followed by a different type of event then that would be a very different thing. So um it's a complex event processing problem. It's event processing problem. So, so having some code in there to basically translate those raw signals and provide the agent actionable context is going to be what we're looking at. So yeah, imagine a fluid stream
So we're basically we're going to manage deltas. Maybe it's certain thresholds that will trigger an event. Um a certain accumulation of events. We basically need a way to um emit at a much slower rate but an actionable rate. Like once we've seen enough to get a better idea that that there's um something for the agent to do, that's when we need to pass it. And we need to basically also at that time be sure about the policies involved and what kind of information is in there so that um the agent only has what it's supposed to have and is only doing what it's supposed to do. So there's a lot of mechanisms for maintaining the stream state. We can look at windows uh tumbling, sliding and so forth to kind of aggregate from that slice to give the agent just what it needs in that moment
We can look at reducing the dimensionality with a different kind of sketches to basically keep that stable space and just hold what's necessary for the agent to make the decision. uh staple patterns like I mentioned complex event processing joins enrichments whatever we can do to basically uh tame that payload uh increment incre uh incremental aggregates so you know counters histograms moving averages those type of things to basically enrich that event with the with the stream context so that the the agent will have um more context text and then anomaly signaling, you know, um that's that's a big one because we're going to um you know, surface deviations, you know, without full recomputee. Basically, if if there's a anomaly signal in there, there's there would be basically particular um uh compensating actions that an agent would want to be able to take and so want to be able to know about. So fortunately there are a lot of tools for these things we wouldn't be reinventing um the wheel. So things like Flink for uh most all the things but yeah event time windows incremental aggregates complex event processing uh solutions like materialize for incremental views stateful SQL red Panda because it's always super fast and Kafka streams you can do some processing on top of it you can do that with Flink too of course but yeah K table sliding windows KSQL DB again on the Kafka side uh processing a lot of information and none of that stuff takes tokens so it's just what we want before we get in there um finel vector DB foss temple patterns on the py python side so a lot of different tools to to use for this processing to basically tame that event stream so that it's going to be um actionable for your agent but if you don't want to do um a high load and you say okay it's it's really basic I don't want to be pulling in all infrastructure. You can do more lightweight things with a lot of various libraries like uh reddish streams with reddish bloom to kind of just do things in the in the runtime state of the agent. um T digest uh river cream you know especially with the percent uh the [clears throat] quantiles uh duck DB you know um awesome library and you know lots of sketches if you want to basically you know which things you're trying to basically squeeze down you could just directly use uh some sketches there so let's get to I anything in governance. Yeah
So, you know, think about um social engineering, right? Um I'm not going to say whether I've ever been had or not, but definitely there's lots of um examples of AI agents being had and it happens in various ways. Um for context engineering uh manipulating log metrics requests um signal pollution you know poisoning the upstream stream text context altering windows feedback loop exploitation you know repeating patterns over time so the agent slowly shifts its definition of normal without noticing is like you know boiling the frog frog slowly. um emergent intent hijacking um you know shaping events so that the agent believes you know something else is happening dynamic boundary manipulation. So there's all kinds of different ways because you could say yeah we're not worried about someone hacking in and talking to the prompt but there's a lot of ways to basically get things into the event stream that your agent will see. And so there's there's still an attack surface to deal with. But so there are variety of ways of manipulating an agent's perception, but you can't trick a scoped access token. So that's why we want to be able to rely on uh security protocol protocols and in identity and access management. So offzero [snorts] for agentic governance um is basically we need to know if if we think back in in terms of um uh rag and all its kind of evolutions
You know, we started with basically large language models and their training data, right? Ask it all kinds of things, ask it to do all kinds of things and it can do it, right? But when it comes down to your company's information, real-time events, uh, it had no answer for you, but it would give you an answer and be very sure about it, but it doesn't really actually have a real answer for you. And so, you know, we had all these different um rag approaches to come up with how do we bring just in time context to give the agent that information. Now, when we're looking at agents doing things for humans, for users, and users that have a bunch of different applications and their data ac split across those applications, you know, the different silos, um this is definitely a case where we again need to have um a way to kind of pull in the right context. But this gets very dangerous because uh you are hitting your users applications and you know if you give that agent um the credentials to that um user all sorts of things can happen. And so you need to basically be able to make sure for the agent what user they're talking to and then a clear way about creating um short-lived access based on what the what the user is trying to accomplish, what they want to do. So identity is a big part here and then the governance being able to define things like the fine grain authorizations and other kind of policies that you have and then the ability to enforce it with um on the security side with PKI and and other type of methods. So you know identity determines who can act. So you might your agent might have an identity, but that identity if it's working on behalf of a user also has to take into account that specific user you're acting for
And it's not um it's it's not that that agent can um if you think about how intent merges over time and it's it's not explicit. And if you take the same agent and the same logic and the same trainings, but if it's been dealing with me over time, um that agent is really going to be a different agent than the same agent that's doing something for someone else. So there's going to be drift and there's going to be basically what the agent has learned. And so really identity at this point becomes a a compound concern of of the agent and who they're acting on behalf of. And then on a governance side, when it comes to the users's data, what they're granting access to and on what terms and then how do we manage it? Like every time if we ask the agent to do something, it gets more permissions and it keeps accumulating, then it's it's going to be the same thing as if you gave it credentials at the end of time. So things have to be scoped and short-lived. And you also need to be able to partition your permissions so that the agent at any time doesn't have um wide range access to to do completely unrelated things and basically have a wide surface to expose to an attacker. So agents must act on behalf of users
So identity and delegation matter. Without access control too, you will lose um context about who did what, when, and why. And so it's another reason that that on behalf of user has to be part of that compound identity. So there there are various ways to um uh protect against a lot of these things. um you'll you'll see cases where people have gotten illicit access to databases, pulled down a lot of records that they shouldn't have been able to see. So finding a fine grained access control is one way to instead of just having these very coarse permissions, you'll be able to say things like, okay, this person has access but not from an unknown IP address or not outside of work hours or not from a different location or not if the a certain type of case isn't open. So there's there's really a rich range that you can do for your domain and your model. So that um a lot of those rules will protect your a will protect your agent when it's basically getting scammed to do something that it shouldn't do
It'll go ahead and try it, but you know such permissions will limit what it can do. That's undesirable. And then sender constraint tokens, right? you and if your if your tokens are stolen um normally someone is going to be able to impersonate your agent and go ahead and do lots of bad things, right? So by using sender constraint tokens like with DOP or MTLS and methods like that, you're basically um having these all the subsequent requests backed by a private key so that um if the attacker doesn't have that private key, then even though they've stolen the token, they won't be able to do anything with it. again um being able to keep the user in in the loop with async off. You know, use things like the client initiated back channel where you can um push approvals to the user's phone so that they can kind of see what the agent wants to do. it wants to do like maybe a stripe transaction and you know verify whether you want it to go through and you know token vault I mentioned you don't you have all these different apps of the users none of those credentials should go into the agent right that's just a big risk it's going to be if that agent is gotten then there'll just be a you know all the things that they can do with your information and your identity so with token vault what it lets you do is that it holds those credentials and then when the agent needs to take action, the user can kind of authorize in a more of a fine grain way, you know, what they want to do. And so you don't have, you know, everything exposed at once. And then again, keeping things minimally scoped and not just accumulating permissions over time, but revoking ones that are no longer needed for ongoing tasks
So if we think about if you go back to that governor pattern and now that we have some ways to kind of enforce it, it looks a little bit different. We have the external signals, we have the the user's data involved. And so you'd have the governor basically managing that context as the signals are coming in and packaging, you know, an actionable case for the ambient agent. And then the ambient agent is now governed by um [snorts] identity and access control and then you know be to be able to call the external APIs. Now there are some also some complimentary solutions that I' I'd like to mention really quick. Uh there's you know behavioral policy enforcement things like marshmallow AI um work on that. It's totally complimentary. there's prompt based security something like um meal pond research so al all together to get a complete picture we'll complete let's take a quick look at that so for complimentary solutions from marshmallow AI we've got uh behavioral policy and learn norms so you know it's using reinforcement learning kind of student teacher patterns so things like uh continuing inferring normal agent behavior from real action traces the student teacher model where your Asian is the student and then there's other kind of of context to from the teacher to observe that okay this makes sense uh this is correct and that it reflects what's been done over time and it reflects policy and it actually adapts over time so that it um uh it gives consistency there and so norm discoveries you know it spots repeated behaviors across similar situations and highlight which ones match what consider what um people consider good adaptive behavioral patterns
The system automatically tightens or loosens what actions the agent is allowed to take and then you know working with the distinct existing identity permissions uh identity rules decide who's allowed to act. So FGA offzero other solutions. So it's very pluggable that way. Uh Mil Pond too has a solution where now you're going to say this is for like prompt based attacks and it's a and if we're talking about ambient agents you'll ask me okay why are we talking about prompt based attacks? Prompt attacks can still appear in the event layer. It really how you handle prompts and how those come to events should be um uh very intentional. You never want to just basically expose those prompts. They they should go through other processing and policies before doing so. But then the other thing is that even with that taken care of when your ambient agents are talking to other tools, talking to your MCP servers, um here's still a case where that communication those requests going to those tools can also um have non-normative behaviors in there that these tools can look at
So basically it gives a visibility into every agent query delivering insights and optimizations. Um it intercepts these calls to determine which ones are allowed and so it will proactively block them. Now sometimes it's actually benign like okay someone has made a dangerous request where they're not trying to do something illegal and so it will go ahead in some cases to kind of mask and patch those so that the call can actually go through and then you know it's in interconnecting all the prompts and agents so that you kind of get this nice behavioral layer in there. So time. So an interesting thing is that a big thing we want from AI generally is to um scale people right scale the workforce. You can say okay we want to reduce the workforce or we want to make people more powerful. Um and so we we give a we give agents tasks to do things and you know we give them credentials. And so if you give a human a wide range of APIs and a wide range of permissions, the the human will basically use the um the ones that it takes to do the task
Uh you know, a human has some kind of normative understanding and or they can know that they can get in trouble or or whatever. Agents are different. All the tools you give to the agent, the agent can use in very novel and surprising ways and take full range of that thing. And so if you think about it, you want to automate tasks that people have been doing correctly over time. So you probably don't want a lot of um random and novel solutions generally. I mean sometimes it's good and and we can capture those but um you you want to be able to basically learn from that audit history and make that enforceable you know within your a system like using something like fine grain access control fine grain authorization. So, um, that's what we're looking at here, like driving norms from audit history and making them governable through things like FGA and then making them accountable. And this is a case before we talked about where AI agents are um doing things on behalf of users
But here the ambient agent is having a boss just like the other people doing the work. And so if an agent is doing things where it's used an uncommon set of permissions, it's it's trying to call an um a uncommon set of um APIs, then this is a chance where it can get stopped, have to basically explain itself, request authorization from a user and then if it so happens that that is valid, um it can go through and then otherwise um it will get blocked. So it's a way to kind of scale scalably utilize your audit history when you're letting agents take on work that humans have done in the past. So that's you know making norms enforceable. And I think [snorts] that's really the the main thing there is that basically creating this this loop where we can um let the agents [snorts] learn from the humans and then the humans learn from the agents in the case where agents make breakthroughs. So I think that's good here and thank you very much. >> So we do have time for a few questions. Anybody has any questions? >> The real question is what's for lunch? >> Yeah, probably
Thank you so much, Fred. I appreciate this. >> Thanks, everyone.