Live Demo – Develop and Deploy a Trustworthy Multi-Agent System | Tyler Jewell, AI By the Bay 25
All right. All right. Do we have this? We are good on audio. Can everybody hear me? Alrighty. Okay. Back again for part two of uh session. This time we're going to just go a little bit deeper into what we had talked about before. uh you know again uh the biggest challenges with Agentic AI systems is the complexity the trust and then the cost of these systems
So um and and we make a platform that makes it possible to build entire agentic AI systems run them and then manage them and you can do that quickly safely and cost effectively. uh you know what we're going to do today uh if we have the time 30 minutes is uh not a lot of time uh we're going to try to get through five uh different sequences of experiences uh the the first we're just going to start with a hello world agent one that uh helps us with our language our international and foreign language and in order to do that agent we're going to need to deal with memory and we're going to show you how memory is distributed uh It's going to need to have a streaming response and we're also going to have a mutable tracing on that. And so this is just going to be a hello world agent. Then we're going to up the game a little bit and convert that into a three node cluster. Uh a cluster with resilience so we can see how an agentic system can uh fail over. Uh it's going to have a traffic routing for uh end users going to the right node and a node failure scenario. And then we upgrade that to a multi- aent system where we use a orchestration to manage multiple agents. There's going to be a toxicity agent, a haiku agent for creating interesting uh uh poems uh and an analysis a sentiment analysis agent and they're all going to be working in cooperation with each other
And then if we get enough time, we'll take a look at how multi-reion replication with high availability and disaster recovery works. And then at the end of all that, we'll show you how we can use cloud code to build these agentic systems through automation as well. Build and uh compile them. So, so let's get started. So, the hello world agent uh first in in a I realize that this is not the greatest uh screen in the world. I apologize, but but this is an entire agent in in a land. And this is Java code. And and what it says up there at the top is uh it's a class
It extends agent. And uh there's a string and it's a very long string, but that's the system prompt. That system prompt effectively says uh start the response with a greeting in a specific language. Uh give some instructions like subsequent interactions uh should be in different languages. Uh so we're going to translate and offer a greeting phrase. Um and then at the end pinned all the previous greetings. So there's there's about 10 instructions here that go into this uh system prompt and and the entire agent is effectively just a method. It's a it's a greet method
Uh it takes a string parameter which is going to be the user's prompt that comes into this and then it uh just returns what we call an effect and and an effect system in AA is a pretty slick mechanism. um and in effect is a way for an application developer to give a set of instructions to Aka and then AA is responsible for executing those sets of instructions and so we decouple the application developer's intent apart from how AA goes to achieve that intent and in this particular case the effect it's pretty simple it's just saying use this system message use this user message. Uh there's a bunch of other things that you could define the models. Uh you could do some dynamic templating, the guard rails that you want, the memory configuration. So there's all sorts of other things that you could pass into this agent, but in this case, we're just keeping it as simple as possible. And uh and then what you're going to see is that uh every every application in AA is structured exactly the same way. So the entire system is given a very streamlined sort of project structure. That project structure has folders for the application itself and for the test framework
They're separate folders. And then we also separate out the APIs, the uh workflows, the agents and your domain logic into separate folders. So there's a very strong separation of concerns. Um and and there's really only way to do these things within the product. So, um, if we if we go back, we have a couple of, uh, consoles here. They're empty at this point in time. I've got them in the folder. And, uh, I I'm just going to, you know, build build this
And so, it uses Maven. And this is a local laptop. The only dependencies that you need are Java and Maven. Um, and and that's pretty simple. and and even though there's no Kubernetes uh or anything else running on this laptop, we can build and run these entire systems. Uh and so that's pretty cool. And the binaries that we create here are the very same binaries that you would go and put into production. So there's you don't need to worry about creating specialized development environments
And it it it built just fine. So uh we're going to go ahead and run that. And and that's just an exec Java. And all you're going to see is that it's a micros service uh that's launching. This particular micros service has an agent and then we've put an API, an HTTP API on the front of it and and it's going to run on port 9000 um as it gets booted up here. And there it is. It's running on port 9,000. That's all well and good
So we want to go and take a look at this and and what we've got. And uh one of the things that we have in our CLI is a local console for managing the system. And we booted that and we it shows us. Okay, great. It's running on port 9,000. And what I'm going to do is I'm just going to grab uh our console address and I want to actually open up where is my browser. I want to open up actually a private window and we'll put that in there. We can zoom in
And so it's black on black. It's a little hard to see, but that's uh this is our service on port 9000. And if I open that up, there's two components in there. There's an endpoint and then there's an agent. So the agent we saw this endpoint was something we didn't see. Uh so I'm going to click on that. And when you click on that, we actually provide for you a request builder. So this is a Postman style request builder
This is against the live API that's there. Um, and so if I do a drop down in here, it says, "Yep, there's a hello method that's there." It gives me the syntax of what it is. My name is Tyler. Uh, I am from Oregon is nice and neat there. And I can pass in he header parameters. But before I go and do that, uh, it gives me the option to copy it as curl. And uh that that's really great because one of the things I can do is we can go ahead and bring up another console window here. Zoom in on it
Is that big enough? Maybe get bigger. Zoom in one more time. All right. We'll just paste that curl. And so you can see the curl. It says, you know, there's the appended. It's Tyler. I am from Oregon
and it's going to send it to that uh URL and it runs there and and I got a response. I got a response from that API and that API called the agent and it says hello in English. It's great to hear from you. Uh Oregon is actually a beautiful state. That that's great. And you know, and it's got a record of all the previous greetings in here. So that was great. So, uh, I could actually bring this up one more time and oops, and I could say, uh, my wife is from California
Pass that along. And if you notice, we get another response in French. And this time, it's got a record of the previous greetings. And so there's some memory with that. We didn't have to set up a database or anything like that, but it's got some memory because it remembered that we talked to it the last time. So we'll go back to uh that particular interface. This is the local console. And this time I'm going to sit there and say uh and this is a true statement
I am heading to Toronto tonight on the redeye uh in coach. H that's a bummer. Anyways, you hit there and it sends it. So, it executed that request again. But in this particular case, because it's a distributed system, you need to have an immutable trace. Uh, and now we have that. And what we actually do is we do distributed tracing on the system. So, it showed the request here
It was a post request that came in. I can actually click on that post request and it gives me the entirety of the details that was there. Um uh so you can see that I got a response um and now it's in uh Japanese and so it added to the response here on that and we can take a look that this total time processing was 1.44 seconds. All of that was put into the agent. Uh we can look we inspect the agent that was here and what happened was that the agent without us having to configure anything it had its own implicit memory. Um, and that memory was implemented as an event- sourced entity behind the scenes. And so it's actually a a durable record of all the interactions. And so a command was sent to that memory for getting history
And that was 2 milliseconds. So that data is in memory. It was cached in memory and it grabbed that history. Um, and then it actually went to Gemini. So it called Gemini with the LLM query and that was where the bulk of the time was. It says it's 1.4 seconds there. And then afterwards there's another command called add interaction where it takes the response from Gemini and sends it to the memory and for persistence on that. And so boom, that was 6 milliseconds
So it was 6 milliseconds to write uh that new entry into as an event into that journal. And uh you know we're we're good to go. And then it responds. And so we've got another interaction down here. And and that's all well and good. And so we can now come back to the agent. And in this agent view as an operator, I can see that there's actually been three interactions, which is exactly what we've had with this. This particular agent doesn't have any guardrails configured
So it's not really set up well for a policy. And if we need to, we can come into one of these things and it'll show us what the system message is, what the user message was, and then what the response. So we can trace that whole interaction with it all. All right. Well, uh, we'll go back to our code and we are in the hello world. And one of the things I'm just going to bring up VS Code on this project and just give you a little sense of what the endpoint look like and the endpoints in the API. And I'll just open this up. We'll zoom in
Zoom in. Keep zooming. How big do we have to get so that people can actually see it? All right. And um in this endpoint, it's also just a pretty simple class. It's got a hello method, which was the post. This is what we used, right? And then it accepts a request. And what's interesting in a uh everything has a location transparency. So in order for this to be a massively scalable system, uh your services need to be able to run on any different physical node
And so since your services could be running on any different physical node, uh the API requests that came in and the agent may not be on the same node. So they need a way where they can discover each other and then talk to one another. And in AA, we have this very simple concept called a component client, which is a way of allowing one component to talk to the other component. And this just says, hey, I'm going to invoke an agent that this is going to tell a to go find out what agent it is or to go find that particular agent. Um, it's going to pass in this particular user. So, it's this user, think of it as a session ID and and I wanted to invoke the greet method on that agent, which we had seen, and then just execute it with the text that was passed in. So it was find this particular agent under this identity uh execute this particular method under that agent and then invoke it with this particular parameters and that's the whole that's the whole API and we got that whole system from just these two files that was there. Um close that right go ahead and uh close this as well and I'm going to just kill this the oldfashioned way
And now it's like okay well we've done that as a single node but what if I want to run it as a cluster and I want to get you know clustered resilience for this thing. Um and so in that particular case we are going to uh run a utility we called a local cluster for running clusters locally. Uh but we could set this up inside of Kubernetes if we want to deal with the configuration. But in this particular case I'm going to run uh three nodes. And what it does is it turns around and just verifies the compilation. Uh and then because we're running multiple nodes, they need to have shared persistence. Uh so it it sets up a shared persistence model behind the scenes. And then it launches all three instances
And so it's running all three instances. I can hit the number and we can see that they're all running. And over here the local console told us that the previous node that we had exited uh was uh previously exited. Um and that none of the other uh nodes had come up if Oh, there they are. All of them come up there. So now we have a three node cluster. Uh and they're all discovered each other. And now that they have the cluster uh they can actually share information among one another
So, if I go back to our lovely browser that we closed out, create a new private window. Zoom in a little bit. Bring it up. And there we go. Now, also we have three nodes here. And so, I could go to 9,0001. We can go to our endpoint. We can invoke a request
I am from Oregon. I cannot type. My fingers are on the colder side. Send that request. Well, that's interesting. That is very interesting. Uh oh. That is the demo gods for you
Yeah. We all good. Why is it not finishing? I'm going to blame this on Gemini being down. Oh, there it goes. All right. Uh, so that executed. That's good. All right, that was on agent two, right? And and so if I need to, I can come over here to node one or actually what I can do is I can come over here to node two
I can say shut it down. So it is, you can see that instance two is now stopped in here in the console. If we give it a second, um, it's also going to show that it stopped. There we go. And if I go into the agent, I can see that the previous interactions are still there. And then the newest interaction is still captured as well. So even though agent 2 is down, the other agents um have the resilience and they still have access to the entire history. So it's a single cluster and you got resilience
Um, all right. So uh let's quit that. Let's also quit this. Okay, we will exit this. Now let's go back to our nifty slides. That was hello world and we've clustered it. And so uh let's get into aentic haiku where it gets a little bit more interesting. It's m multi- aent
So in this particular case, we're going to have two APIs. Uh one of the APIs is for accepting the request that it's going to initiate it. And we want the user to type in some inspiration, an inspirational quote. And then from that quote, the first thing we want to do is we want to check the inspiration to see if it's toxic and then to also make sure that it's happy. We're not really interested in sad haikus or mean haikus. And then once we've done that, those are going to be two agents that do that. And after after we check on that, then we're going to take that accepted uh inspiration and then use another LLM to generate an actual haiku. And then based on that ha coup, we're going to ask Gemini to create an image that is a reflection of that concept
And then after all that work uh since there's a bunch of uh information that's span across those different agents we're going to create a view and a view in a CQRS representation of the data and it makes a readonly representation of that data that can then be pushed out over a server sent event which over a second endpoint. So that's that's the structure of this system. uh the code looks a little bit different in this particular case. There's a workflow that gets implemented as opposed to an agent. Uh but it also has an effect. Every component inside of AA has an effect. In this particular case, the effect is uh applies to the starting of the workflow, the termination of the workflow, and each of the steps. And what you do with that effect is on the start here, it's basically saying every workflow has a state
So you're going to update the state of the workflow, right? Let's give it a empty an empty state in this particular case. And then after we're done executing this start step, we're going to transition to the check message quality step. Um, and with certain user input, we're going to pass the user input along. And then we're going to reply that this start step is done. And then the rest of the workflow is implemented as these other steps. You just write these methods and inside those methods are other effects that basically say inside this step I want you to do these things and then transition to another step. And there's other things that you can do. You can do retries if it hasn't been addressed in a certain amount of time, what to do if there's a failure event
So you can have all all kinds of logic and and behaviors on each one of these steps. And then uh the view after all these sort of steps execute uh the view is going to be triggered um when the haik coup is generated. And so you'll see, it's probably a little hard to read, but there's a method here and it basically says on change and when there is a Haiku generation event that is triggered, that method's going to be executed. And basically what it does is it says grab some data, grab some information, and then update the views row. And think of the view as a private readonly table that has information about the Haiku system. And so on on each haiku generation, it's going to grab some information, put it into the table, and then way down here on the bottom, there's another method which is used by the API, and the API can actually query that table through uh it's a get all method, and it actually queries that table, grabs that readonly data, and sends it out. And so without the developer having to interact with a database, they we've basically created a separate path for all the write operations independent of the read operations. And so uh we go back to our uh wonderful system and now we're going to go to our agentic haiku app
All right, we'll go ahead and make sure that it compiles. Um, this particular one, uh, I'm going to should be okay. All right. Excellent. And, um, yep. We just need to do some extra tracing. So, I'm going to run a script instead of just Maven running it. And, and there's some extra tracing that it tags into that
Uh but when this time what we're going to see is that it's going to boot up and there it is. It's on port 9000 which is great. Uh we can bring up the AA local console and it should tell us that that service is running. Okay, great. That's running. Uh but instead of going to the local console this time, one of the things you can do with AA uh is you can actually host applications inside of the Agentic system. So I'm going to actually just go to localhost 9000. So this is actually the URL of our server of our service that we built
And so we've actually loaded the application inside the API. So this is a static location. And so this is a little bit of a clientside application uh that we built for this AA haiku generator. Somebody uh volunteer a line of inspiration that we can pass in to this. Oh my goodness, that was so original. Thank you. May the force be with you with one period or three. All right
All right. We generate and off it goes. Uh and you know it's already uh gone through the sentiment analysis. It's gone through the toxicity analysis and now it is generating the haiku. And there we go. May the force be yours. Guiding presence ever near. Walk with strength and grace
All right. Words that have never been better spoken. And then afterwards, it sends it off to Gemini's multimodal. And this is the image that came up. Oh, look at there's a force. There's force rays. You can't see it, but there's a little nice rays there. Okay, that's quite nice
Very good. Very well done. Very good choice, sir. All right. Um, this time we go back to our local console. We can see that there's an Agentic HighQ service here. This time when we drop down the components, there's more going on. We have our endpoint
Uh we have the agent for generating the haiku itself. We also have the sentiment agent. We have the toxicity agent. And we've got the workflow. And I I think in this particular case, let's go take a look at the workflow. Uh because workflow has a different state. It's got its call stack here. And I might be zoomed in too far at this point
I apologize. So, we're going to have to make it a little bit tinier, but we can see that it was successfully completed. Um, we can actually open up and look at the various state objects and how it was updated. So, this is uh because each step has its own state. And you can see that it took about 19 seconds to execute. So, it was a long running process. Uh, but we'll come in here and we can take a look at that workflow. And it actually shows us each of the steps, what their state was, how many attempts and retries there were on this, and what the input and what the output was, and also the execution time on this
So we can see it was the check message quality, analyze the sentiment, generate haiku, generate image, and and pretty straightforward. Now the only thing we didn't do with this is that we ran uh the check message quality and the analyze sentiment uh serially and and if we wanted to really you could do both of those things in parallel. So uh we could rewrite uh rewrite it so that those two steps execute in parallel. It was just an implementation choice to do these serially in this particular case. And it looks like we could save ourselves about 1.2 seconds if we had done that uh in in parallel going that way. All right. Um I am being told that I'm running out of time. So if I go back to my wonderful uh app, uh what you'll see is that if we the next step that I would have done is I would have taken that Haik coup and actually deployed it into one of AA's clouds
And when you do that, you get uh some really incredible monitoring capabilities so that you can track failed requests, successful requests, processing time distributions, P99 latencies, um uh token counts, uh LLM behaviors, and so that's just automatically being tracked. And uh and when you deploy it into our cloud, you can actually take that same binary and deploy it into separate clouds and all of their state, all the workflow state, all the agent state would have been automatically replicated between those two clouds and you get a high availability and disaster recovery. And if you come up to me afterwards and and you want some more information, uh what I'll do is I'll give you uh this last a link to this last video. Uh and and this particular video uh on this moment, this is actually a multi-reion HA scenario where uh we're running one in Google Cloud. This is the same app. What one's running in Google Cloud, the other one's running in Amazon. and it's mimicking uh and showing you the replication in real time of everything that is drawn in one of the environments is duplicated. And then all those a bunch of little tiny numbers inside those screens, those numbers are the latency numbers of what it takes to do the replication and then verify um verify the accuracy of that
Um and so this is all this is all capable and we do this at at sort of terabyte level scale. All right, I only had, you know, one little hiccup, which is not bad for the for the demo gods. I'll give you that. And I'll leave that up there and then open it up for any last minute questions before I wrap up and say thank you very much. >> Great. Thank you. Yes. Does anyone have question for Tyler >> here? Yeah, you have a question
Um, is this like a open source thing that people can like use or is it um >> I I couldn't hear her. >> Is this an open source thing that people can use? >> Is this an open source thing? So, uh, uh, we we used to be open source and we learned the very hard way that that's not a very good business model. And so, uh, uh, this product is, uh, it's in business source license, which is a source available license, which means it's free for development. Um, so pre-production, uh, unlimited usage and then we charge we charge usage in production. Uh, our cloud service, we just have a flat fee for $100 a month on our cloud service. Uh, but um, uh, we have all kinds of, uh, discounts and freebies for startups, uh, startups, research, uh, institutions, u, students. So, we do a lot of a lot of free stuff. Uh, but it does require a commercial license
>> Question. No question. By the way, I just want to say uh it open source can be a great business model, but when your software works as good as our software works and there's no bugs, people don't need to pay for support >> and it's not such a good business model. [snorts] >> All right, >> buddy. No. Okay. >> Thank you very much. All right
Yeah. Excellent. Okay. Thank you. Thank you, Tyler. Yeah.