Challenges of Ambient Agents: Building Governable AI Systems at Scale | Fred Patton, AI By the Bay25

As AI systems become continuous and event-driven, ambient agents represent the next frontier—and the next headache. They live inside systems, not chat windows; they listen, adapt, and act without explicit prompts. This shift breaks many assumptions about how we scale, govern, and even measure AI. This talk explores the emerging design challenges of ambient agents—from streaming architectures and adaptive token governance to the elusive art of defining Minimal Viable Context (MVC). We’ll discuss how these agents blur the lines between automation and autonomy, and how techniques like audit-mode discovery, sampling for continuous learning, and scoped delegation can restore control without killing initiative. By the end, attendees will understand the core tensions in governable AI systems: persistence vs. efficiency, context vs. privacy, autonomy vs. accountability. These tensions aren’t bugs—they’re design constraints for the next generation of scalable, trustworthy AI. Ambient agents don’t just respond—they observe, remember, and act. But with that power comes complexity. This session takes a systems-design deep dive into the realities of building continuous, event-driven AI at scale. We’ll unpack: - Why context becomes the new scaling law for always-on systems. - How to define and enforce Minimal Viable Context (MVC) under cost and governance constraints. - The role of streaming algorithms and adaptive token budgets in controlling operational drift. - Techniques for scoped delegation, fine-grained identity, and session-level security (DPoP/mTLS). - How sampling a…

▶