Future of AgenticAI: Architecting the Global Control Plane (Keynote) | Tyler Jewell, AI By the Bay25
All right. Good afternoon. My name is Tyler Juul. I'm CEO of AA. AA is a 16-year-old company. We were previously a development toolkit for building distributed systems. It's been downloaded a billion times. There's over 100,000 systems that have been built with AA
And on any given day, two billion people touch an application that is powered by AA underneath the covers. And today we're going to do a little journey and have a discussion about what is Agentic AI and what's the direction of it? And and that's an interesting question. What is Agentic AI? Um and and it's an interesting question to explore. uh and we need to explore it because uh the productivity gains that are being witnessed are absolutely stunning over the last six years. If you take a logarithmic curve of all the models that have been released and plot them every seven months, the capabilities of a single request, call it an agent, call it an LLM, have doubled every seven months in the last six years. If we're able to sustain that pace of innovation over the next six years, in 2030, a software engineer who uses an agent will be able to complete a month's worth of work on a single request. It's a stunning, stunning change. Six years ago, we got a couple of seconds worth of effort through autocomplete, and now we're looking at a complete month's worth of software engineering work
And you know work magically work. There we go. So how is this happening? Well the first thing is let's break this down and you know kind of understand what is a gentic AI. The first thing we need to understand is that LLMs are not perfect. They're stochastic engines that generate random outputs. They're highly probable random outputs. But even today, if we take a look at the most expensive uh models with the largest context windows uh that can handle the do the most processing, they're only in this 76 to 82% accuracy range, which you know, flip side is is that that's an 18 to 24% failure range on every request that you give to it. So, uh the first thing you have to understand about ai is that the things that you're working with are randomized
This is not going to work. Well, here we go. Right. The next thing is is that when you look at an agent, you have to recognize that it's a system. It's actually something that is working with data. That data is interacting with that LLM over a network. So, you've got you've introduced a network. You've introduced data
It is a system. An agent and an LLM is a distributed. Is it a distributed system? I wouldn't quite call it that, but it is a system. And because it's working with data, it's going to be stateful as well. And an Aentic AI system, the agent itself, if it wants to have some remembrance of the previous execution, that's called memory. And memory therefore is a database. You got to have some sort of mechanism of persistence that the agent can tap into. And that suddenly makes the agent stateful
And this stateful behavior extends beyond the agent because you have heard about people talk on orchestration or workflows and we'll discuss a bit more of that. But orchestration is effectively a fancy way of saving the call stack after each step. So that requires persistence as well. So we have random LLMs, we have a system, and it's stateful. Furthermore, if you just didn't want to add some more complexity to this is that it's been shown that the larger your context window, the bigger the prompt you give, the less accurate the response is going to be. It's called context rot, right? So unlike databases where as the transaction volume runs higher you get a very deterministic and consistent profile with agents as the memory gets larger. If you insert that larger and larger memory without trimming or managing that context window your accuracy goes down. So you've just increased the randomness profile just by using the LLM
So when you're dealing with a gentic AI, you have to kind of accept some of these elements that you're dealing with and you have to change your mindset. Over the last 30 years with most of the systems that we built, we approached them from a deterministic a determinism point of view that we could design systems that had rare failures. If you do that and you try to put that into production, you are going to have an unhappy outcome. And really what you have to do is you have to reccalibrate yourself and accept that all you can do is strive for perfection because that these systems will never achieve the expected outcomes that you have. But you can strive for perfection. And striving for perfection is important because how do you strive for perfection on a system that's continuously adapting, continuously changing on you. And so when we deal with enterprise accounts who are looking to put a Gentic AI systems into production, they start by assessing their risk profile. And what they realize is that historically with most of the systems that they had to assess, it was purely a systems risk
things related to resilience, availability, uh performance, SLA management, perhaps some cost management, isolation, security of the network, but because of the randomness of the LLM and emergent AI, which we're going to talk about in a second, and context rot, you also have to factor in that there's a randomness risk, and so your risk profile has expanded systems risk and random randomness risk. It really disturbs and bothers a lot of the enterprises that we work with. Uh not just the context risk and the random risk, but the emergent AI. Emerging AI is a I guess a concept that uh that came out in April of this year through uh scientists and it shows that if you take an aligned AI and an aligned AI is a model that has been um uh not only tuned uh but all of its ethical and moral safeguards have been validated and then you put that into production. you can retrain that model or feed that model with prompts and information that you give to it that distorts its understanding of truth. And as it goes on over time, an aligned AI that had ethical safeguards can be turned into an malicious AI that almost has no safeguards whatsoever. So you can convince the AI that its reality is different than what it was programmed to be. So all these things are factors that affect your risk profile in order to get an agentic AI system into production
And so it's no wonder that if you've paid attention to the news that people tell you that 95% of AI projects fail or that they fail to make it to production. A lot of that stuff is experimentation, but most of it is enterprises coming to this sort of stark reality that there's three problems that are keeping them three barriers that are keeping them from entering into production. The first is is that these systems are complex because they are distributed systems. There's distributed state for your memory. There's orchestration for managing them. They are operating over a network. And on top of that, they're going to be constantly changing. And then most importantly, they're just not trustworthy
You don't trust them. Now, people default into, hey, it crashes or maybe it's not the most reliable framework, and that's certainly a concern, but there are safety and ethical concerns that get in the way of people moving these things into production. And then the cost factor. Everybody's aware of how much it costs to consume an LLM. But in Agentic AI project, the costs go much further beyond that because there is a nurturing and ongoing maintenance of the Agentic AI system that is continuous and some of the compute and infrastructure builds that go into the management of the Agentic AI tier can equal or match the cost of the LLMs themselves. So complexity, distrust and cost are these real barriers that move forward. And so the bottom line, the bottom line though is that if you do not trust your Aenteek AI system, it's not going to go into production. And even if it does go into production, it's not going to stay there, right? And I tell this to our prospective customers all the time
You know, if we can't get your system into production, we're not going to make money. uh because we only charge for production usage of our system. So our mission and I would argue that the definition of an Aentic AI system is one that a method by which you are going to infuse your applications with AI, right? We want to build applications that have AI in it. But it's also a system that continuously builds trust. If you don't have both of these components, it's not agentic AI. Now, I mentioned before that we talked about how there is no such thing as perfection. There's no such thing as determinism. So, all you can do is you can strive for perfection
How do you strive for perfection in a computing system? Is it's one that continuously adapts. It's one that is observing its environment and it's striving to be as close to perfection as possible. And when you do that, then you can have trust. So there are six steps, six steps that we talk about with our enterprise accounts on how you go about building this framework of trust. Usually with a lot of systems that you build, you end with governance. You think about observability, the metrics, the policies, the controls that you want to have in place um after you've designed the system and built it. Well, since this system is an imperfect beast, it's never going to be fully accurate. You have to invert that
You have to start by understanding what your boundaries are going to be, what your metrics, guard rails, protective be uh uh protective controls are going to be in place and how you're going to measure these things and observe them. So, you start with AI governance. You don't end with it, right? You don't build the agent first and figure it out. The next is that every agentic process needs to run in an an impenetrable runtime. It needs to be fully isolated and then the network that it's operating on must be a zero trust network with least privileges um and mutual TLS uh connectivity. The system has to guarantee resilience. And in order to guarantee resilience, you have to be able to guarantee that there's never going to be any data loss of the memory that the agent's consuming. And then therefore the data that is going to manage that orchestration
So it has to be able to recover not only locally, but it's got to be able to recover over territorial zones. Agents by definition are stateless processing units. At the most basic level, they're just a compute unit and you have to be able to assign a verifiable identity to those agents because the agent is going to interact with other LLMs and it's going to interact with other agents and it's going to interact with other APIs and other tools. In order to do that on a safe basis, it's got to be able to identify itself. It needs to be able to verify that identity, whether it's taking that identity from another system or the user that's engaging with it, and be able to pass that identity along. And you have to then be able to trace every interaction across this distributed system and put it into an immutable, irrefutable audit log. And only then can you have true chain of thought and traceability that leads to an explainable outcome. And explainability is a critical part of these systems because if you don't trust it, right, you then need to have explanations for every decision that was made from the beginning of time
And then because it degrades over time, it has to continuously adapt to its environment. It has to be able to read those environmental sensors, take the events, get learnings either from reinforcement learning of the models or other places and incorporate it all so that the system gets a little bit better on each iteration that it goes. So, a we used to be a development framework. Now, we ship an Aentic AI platform. This platform is everything that you need to build an agentic system and get it into production quickly, keep it there safely, and we can guarantee that it's going to scale more cost effectively. So much so that we are consuming up to 95% less cloud compute than certain Python frameworks that you use like Langchain and Lingraph. There's reasons for this. And so our mission is to help you build systems that never fail but also continuously build trust
Now we have an SDK. This SDK includes everything that you need to build, test and run an Aentic system. And what makes us different is that we approach it as a system, not as a piece part. So you build the entire system with the SDK. You test the entire system with the SDK. You package it. You deploy it. And then when you run this, we run this as a cluster
And I'm going to talk a little bit about how uh we guarantee resilience and elasticity. Uh but that cluster can then deploy into any environment you want on edge, bare metal, onto Kubernetes. And then we provide a whole series of manageability utilities for various audiences so that you can build trust into that system and make sure that it's behaving exactly the way that you want. Um distributed systems which is what a is incredibly well suited to this entire problem space because a distributed system has always started from a first principle of not trusting the environment around it. We always assumed that the hardware could fail, that the network could be disruptive, and that the system needs to self-heal and adapt to those changing conditions. Well, an LLM with its randomness is just another variable that we don't need to trust and adapt to. Granted, it's a content type of randomness. It's not a hardware or a network, but it's just another form of randomness
And so a distributed system is well suited to solving the agentic AI problem. So how do we do that? Um the first is is we can guarantee elasticity. Uh we do this because at the heart of aka uh is a form of concurrent processing called actors. Actors are a lightweight form of compute about 4 kilobytes in size. We can get hundreds of millions of concurrent actors on a single core. And by doing this uh actors also have a isolated structure with supervisors. It's almost as if they are running their own agentic system but supervisors can spawn actors. It can change the flow of information between those actors and it orients the system so that the whatever the the goal that it was assigned to it can run it in the most efficient way using the maximum amount of resources that have been allocated to it
And we do all this in a way where you can build a clusters that run across millions of nodes. And by doing that, you can get fan out, fan in, and then traffic steering routing for your end users. Uh uh allowing you to have the maximum amount of compute um processing for your end users against whatever physical allocation that's been given to you. We also guarantee resilience and this resilience can be for your agentic memory. It can be for your orchestration, your durable execution. Uh it can be for just memory objects that you create or even the streams that you have within it. And the way that we do that is all your state is tracked as a sequence of events that's put into an event journal. That event journal is persisted transparently to the developer behind the scenes
But because you have a sequence of events in any sort of disruptive environment, AA automatically can go to that journal and rebuild your agents memory uh instantaneously so that it can recover exactly to the state that it was at previously. We do this for your agent memory for any objects that you create in the system and it's a form of event sourcing. Uh and we've been we've been doing this for a long period of time. Uh it's one of the original foundations of what AA is. But we then evolved all this into what we now have is domain integrated context engineering. It's called DICE. Uh which is designed to help eliminate and erode that context rot. And it's effectively a way of allowing your agents to view the domain object that you have, the application behavior as a form of tool so that your agents can interact with the rest of the system um as if it was like a remote MCP server
And when you do this, your context window shrinks and the appearance of rot goes down quite significantly. And then last we have streams. And streams are a form of real- time processing that takes audio, video, uh, metrics. It could be events from a pub sub broker like Kafka and it offers a form of continuous processing so that you can take all your environmental sensors, process it, synthesize it, update your memory, update your context windows, alter the goals that a orchestration has and all of that then affects the next iteration of the system that it runs. And then finally for our larger enterprise accounts uh we bundle all this up into uh multiple regions and you can deploy AA clusters into different regions and when you do that you can deploy the same agent into two different locations and then they have a form of active active replication of all their state between those regions and because it's active active you can lose an region and then the system will fail. fail over and offer a continuous operation on the failover region. And when that downed region comes back up, there's a disaster recovery and a reconciliation of all that state so that the system carries on. Uh we have customers that are deploying into upwards of 30 regions and they get they use those regions not just for um high availability and uh continuous operation but also for data sovereignty uh and cloud sovereignty
These regions run in any different cloud provider. So it could be Amazon, Azure or AWS. And you can actually do high availability and disaster recovery across cloud providers uh automatically. Uh we've been now working on the Agentic AI problem for about a year. We have about 45 accounts that are in production and we've started to see some really remarkable performance metrics. uh our largest account is uh deploying about 18,000 cores in a a giant cluster and they're processing more than 5 billion tokens a second through uh uh through through the system. It's about 120th the size of open AAI on a tokens per second basis. And in that particular case, what they're doing is it's using a feedback loop uh because they're processing uh real-time content streams about users uh their advertisers in order to generate recommendations uh through a series of a halfozen different models
Uh and this turns into a giant feedback loop that then ultimately leads to recommendations and personalization that goes out the door. Um some of the things that people use uh to build these agentic systems they're doing uh large data pipelining where there's large volumes like pabytes worth of documents that need to run through massively concurrent pipelines. Uh we have uh vendors who are doing customer support and chat bots. There is traffic routing optimizations that you can do with modeling. Um, and so it's some combination of six9s of availability, uh, large numbers of concurrent end users, very low latency issues with your memory. Um, or you've got some sort of territorial multi-jurisdictional deployment requirement and all those things come together into the customers that uh we have landed here. Um I am going to uh have another session uh in about I think it's about an hour and I'm going to do a live demo of building some multi- aent systems and deploying that into clusters and I think we're going to do that in this room. Uh but until then uh me and my group uh we're exhibiting
We'd love to talk to you and answer any of the questions that you have at our booth. Uh other than that, thank you and welcome here. Yep. Thank you. Um um if anybody has any questions for Tyler. Um no. >> Oh, >> thank you, Tyler. So, I'll start
You know, uh so basically what's amazing to me is like there is a huge scale of these requests. So what um what uh kind of it's a distributed system like what AI features right these clients want what what are they bringing in from the Python world how do you see the stack around a emerging on top of a >> uh so that was uh two questions in one it was what are the features of Python that people want or AI features of Python that people want and then what's the future here um you know the interestingly thing I'll say that the Java stack right Now there are like six or seven uh uh agentic AI frameworks in the Java space and and they are they are matching pace with lang chain uh on in terms of capabilities on the AI front. Uh the the big things that are really necessary in a system like this you got to have guardrails. Uh guardrails are a form of being able to validate inputs and and outputs and then being able to interrupt your control flow. Uh we've got that. Uh you got to do um evaluation. uh evaluation is a way of evaluating the accuracy, safety, and performance of the system. So, we provide a it's actually a really cool way to do it, but it's an LLM as a judge approach where the system can be running and then in parallel to that, you've got these other agents that are evaluating the the first one that's running
And then as the agents are evaluating it, they're generating events that affect the system itself. And it just keeps uh going in a kind of an LLM as a judge evaluation framework. um uh uh the way that you do uh immutable tracing uh you're going to see some really cool stuff, but you've got to be able to trace not just the interaction between the LLM and the tools and and whatnot, but you got to be able to follow the trace from the API through the distributed system all the way. And it's got to be uh tamperproof. You have to be able to demonstrate tamperproof. And then the number of security things, uh we just finished an RFP. uh it was a 300page request just on the security dimensions on on AI and so AI introduces a whole different kind of risk profile that traditional systems don't introduce and you have to demonstrate an ability to either uh prevent or resist all those issues >> questions Oh. >> Um, thank you for the great presentation
Uh, I'm right here. >> Yeah, thank you. All right. >> Uh, uh, great presentation. Um although most of the technical details uh uh I would have many different questions over but I particularly liked the uh 5 billion tokens per second metric. uh can you share some some tricks and uh tips on how you're able to support that kind of load and uh uh would it mean that you would have to keep a lot of open connections? Um like anything that you could share from a distributed systems point of view? >> Yeah. Um uh so that's 5 billion tokens a second and that's an 18,000 core system. our largest deployment of it's non AI but our largest deployment is 300,000 cores uh and that is a one of the world's top four observability vendors uh in all the data ingest that they do on that so uh you know so what are some of the principles that you got to get into in order to be able to get to that level of scale um uh everything needs to be non-blocking I mean everything all requests through and through not just the initiating end user requests uh but also all the services within AAS because you've got workflows talking to memory, talking to agents, talking to streams, those all have to be non-blocking um uh actor-based uh concurrency mechanisms and and then the outside resources that you're talking to need to also be reactive
Uh part of part of most most model systems are not reactive yet, meaning that they're going to be synchronous and blocking uh connections. And so you're going to have to manage against that. But I don't think it's going to be long before we see reactive drivers for LLMs, inference providers and and whatnot. And that and that'll help with the scalability of it all. The the systems that don't scale as well, it's not about the number of concurrent uh activities that are going on. It's about the amount of state. So if you want to have a stateful system and you want to have pabytes uh distributed around the world, that's where it becomes more challenging. And we we can talk about that at our booth on how we do that
>> Great. I guess we have time for one more question if anybody has anything. Yeah. No. Okay. So, yeah, it's uh 2:10, so it's like right on time. >> Awesome. Thank you
Thank you, Tyler.