Uber's Multi-Agent Platform | Jamieson Leibovitch, AI By the Bay 2025
Recording: Uber's Multi-Agent Platform | Jamieson Leibovitch, AI By the Bay 2025
All right. Hey folks. Uh, so I'm going to talk about uh, Michelangelo. Let's, uh, move on to the next slide. All right. So, uh, just to reiterate, I kind of already had an intro, so fantastic. Uh, so I've been at Uber for about four years. Um, I'm a graduate from a school in Ontario called Ontarot Tech University
Uh, maybe let me grab this actually might be a bit easier. Um, I have a bachelor of engineering. Uh, I interned at Uber ATG, which was Uber's self-driving startup. Uh, I was actually in the hardware embedded org. Uh, so this wasn't even in like the distributed or AI scene yet. Uh, last year I worked on Uber's Texas SQL project called Quico Copilot uh, for uh, Presto SQL generation. Um, and I'm currently working as one of the agent platform leads as a full stack engineer. All right
So, the first thing that we we always like to cover first is how how Uber believes the north star should be, right? So, just like AVs, what we've done is we've broken down levels of autonomy from levels one to level five. So, level one is essentially our responders. These are your Q&A bots that you get like for example like on websites that you might have had maybe 5 10 years ago. Uh you go to any site that kind of just looks up FAQs and these are really your L1 level autonomy. From uh chat GBT onwards uh you have these like L2 assistants right so these are these kind of popped up a couple years ago. So it's able to leverage a set of predetermined tools and system instructions in some form of agentic loop. So of course uh I'm sure as many folks know if you've been attending the talks this really isn't enough. So from then we have L3 which are our collaborators
So having multi- aent systems that actually collaborate with each other and also have some form of AI planning reasoning and some form of human the loop. So from there uh we can even push the bounds. We have L4 experts such as uh AI agent uh engineering teams that actually build a new complex feature. So for this it might require things like planning and refining strategies in real time. And of course as we push towards AGI artificial gen uh general intelligence we have L5 autonomous teams uh for AI manage such as an AI manager that can build its own teams of new coding agents uh that can create uh basically teams to invent um invent new systems or solve problems that wouldn't have been defined otherwise. So uh let's talk about the evolution of Uber's agent platform a bit. So what we've done is uh basically broken it down into L1 to L4 for now. So with L1 we have uh our model gateway
This is really just getting access to LLMs at Uber. So the model gateway is a very key part to even bringing LLMs into Uber and many other big companies I've talked with do the exact same uh thing where they have some form of gateway that allows you to talk to all the vendors. So this allows us to do uh auditing, compliance, legal, privacy as well as PII redaction. In 2024 onwards, we started working on uh two products called agent FX, which is our internal SDK, as well as assistant builder, which is a noode platform for building uh L2 assistants. Uh this allows for basic orch orchestration, rapid prototyping and access to internal tools and data that otherwise would not have been available through products such as chatbt or Gemini. In the early 2025, so early this year, we started working on our agent platform. So we've had a lot of learnings from 2023 to 2025. We learned that single agents aren't enough to solve complex problems
So what we've done is we built uh a system uh that allows you to build multi- aent graphs, visualize, debug and trace them uh have evaluations inhouse as well as AI guard rails. In addition, we've expanded our model gateway to support different types of vendors as well as OSS uh models uh in addition multimodality. As we close out the year, we're basically at this point of uh L3 autonomy. We have upgraded our noode platform assistant builder to a v2 product called Uber agent builder allows us to build uh autonomous agents and also event-driven agents. So if you don't know Uber is one of the biggest uh consumers and contributors to Kafka in the entire world. So having a a form of event- driven agents also known as ambient agents is a is a very uh very clear pathway for Uber. From there we're also working on model gateway 2.0 O upgrading and uh upleveling our model gateway to provide first class API support for all vendors as well as additional plugins. So at the current state of Uber's agent platform, we have uh basically three three different uh verticals
So what we want to do is provide the best-in-class tools for creating, evaluating, debugging, deploying uh high-quality agents that will rapidly scale safely uh as well. [clears throat] So basically what we have is uh our rapid prototyping with Uber agent builder allowing uh engineers or non-engineers to come in and and build AI agents at scale and rapidly. Then when you're ready to move to production uh you can uh basically write these applications using our in-house SDK called agent FX. And lastly to continue to iterate of course we have evaluations but we also have the ability to visualize and debug the applications using a product called Uber agent studio. So just to kind of circle back on agent FX a bit. So what is it? What is it right? It's our core agent SDK. As I mentioned in the previous section, we had started it back in 2024. At the current state of things, it's our core agent SDK providing a standard agent interface for all agents at Uber
In addition, uh the ability to have a set of standard online service APIs that will connect to our internal tooling which allows users to visualize and debug. Also uh this set of this uh standard interface that we provide also allows us to uh enable standard offline use cases such as batch processing. So at a high level this is what we call our agent runtime. So agent runtimes have come up a little bit more recently uh in in different blogs. But what we mean by our agent runtime is the execution path for agents at Uber. So what we've done uh again is we have our uh at the very bottom of the diagram on the right we have our SDK. This provides all of our base interfaces for uh all agents at Uber. In addition, we've adopted an open source protocol uh called the agent protocol in combin uh in in collaboration with Langchain AI
So the agent protocol is a fork of the langraph cloud APIs also known as langraph platform uh which is published by langchain and allows us to to use a standard set of APIs for online services. Uh the benefits here uh as you can see at the second layer provides interfaces for threads which are sessions runs which are part of the executions messages that are artifact of the executions but tied to a thread. In addition our agents and these are all built on top of our core agent SDK. So what we've done is defined uh a very low-level abstract interface called base agent. Uh this is basically just execute and execute stream. In addition, we have interfaces for accessing uh data sets or vector data sets, our evaluation interfaces as well as access to tools and MCPS. All of this combined together is exposed through our online service API via agent protocol. We also have a uh UI that we connect out of the box to all of these uh agent effects services
Uh basically this pluggable chat UI you can either bring into your uh native application at Uber uh internally or externally. Um and they all connect to your uh your agent effects application out of the box. So this powers uh two key products from Uber Michelangelo which is uh Uber agent builder agent studio but again can be used in end user applications. So just to kind of like touch on why did we pick agent protocol. So uh you know there's a lot of different protocols available now back but back in January of last year there really wasn't that many. Uh since then multiple uh protocols have popped up. So you know MCP has gotten really popular. Um Google 808 and a few other really uh really interesting APIs
But at the time there really wasn't anything. And we needed to solve the same problem as I'm sure many other folks. standardization. So what it gave us the ability to do was provide chat history management uh making it easy for us to manage the presentation layer as well as a series of low-level APIs essentially dictionary in dictionary out almost every agent SDK uh lang chain crew AI ADK they're all basically just dictionary in to dictionary out and this was one of the things that we really liked about this API at the time as not every agent is necessarily a chat agent and lastly it also gave us the observability to view individual executions So at a high level uh just to show off where we are today uh on the left side we have our our no code product which is uh form based called Uber agent builder. So it allows us to uh create prompts that have connected connectivity to all of our tools and users can essentially uh do at blank to uh to to prompt. So for example if I want to hand off to another agent I can call at fu. So whenever I ask about the weather, it might be a bit hard to see on on this uh it's a bit might be a bit small but it says at fu uh whenever you're asked about the weather and you have access to tool A b and these are done through the prompting on our our noode platform and this allows you to build uh effective noode uh multi- aent systems using uh basically a form on the right uh for once you start needing to move towards production we have our agent studio agent FX and eval platform So as you can kind of see in this uh this video here uh you have the ability to visualize uh your application as you run through it. So this uh this agent here is basically our uh our toy pancake agent
It can only talk about pancakes. Uh so as as I talk through it, you can watch it u basically traverse through the graph. In addition uh you can see the presentation layer information on the on the actual chat window. that that chat window is our pluggable UI that can be that's used in both Uber agent builder uh and user applications and agent studio. So just to dive in a little bit more. Uh so this is an example of how you can prompt on Uber agent builder. Uh it's again multi- aent. So I can uh go through all of the native noode agents uh and even some code first agents as well if I want to hand off
So uh let me see next one. Cool. Uh so on the slide as well uh just to to call out uh so this is an example of adding a new agent. Uh we support multi- aent drive tools which is a feature that is very powerful for users. The be the ability to select some fields that you want to show to the LM but some that you don't actually want to. For example, if I were to build uh an agent that calls a Jira dashboard, uh I don't necessarily wanted to look at every possible Jira issue. I just wanted to narrow it down to maybe some query or some uh project name. So the drive tools is an example of the ability to do that
Uh, we also support MCPs out of the box, the ability to have files, Google Docs. Access controls are built in. Of course, we don't want every every user to access every agent unless the owner wants them to, as well as the ability to connect to our pluggable UI. Um, so this is the agent studio debugger. Uh, just a bit bit more in depth here. So, you can see an example run that I had. So, this is my my pancake one. You can see uh it's probably a little bit hard to see actually
Uh, so I it called the tool. You can see it on the UI and then it it talked about stuff and I can even uh kind of narrow down on which node uh had details. So I can see all of the the graph state uh within this uh this console. So essentially uh this this UI hooks up using this agent protocol that we talked about. Uh it's part of the agent effects SDK out of the box. Meaning that if I were to scaffold a new application uh today, it connects to this uh UI right away and is used as our native development tool. Uh in addition it also provides node level observability uh and tracing through open telemetry. So every agent also has the ability to uh go view it in our trace UI
Uh so you can see this trace button up at the top here. Uh which then takes you to our in-house tracing. Uh one of the benefits of this as I mentioned it connects to your local. Uh I don't think you can see it in this. I think it might be cut off but this is actually connected to my local laptop for this particular image. So I spin it up on my local connects right to it but it also can talk to my deployed agents so that I can actually test in a staging or production environment. So it's it's really good for demos. Uh we also have version history
So we have the ability to uh go back in time look at previous versions of the agent and also tie evaluation reports to that. So whenever a user goes into our codef first platform uh makes a change such as like a prompt or even a graph change or really anything uh you can tie an evaluation to a particular gitshot which allows you to then uh look back and see the progression of evaluations. So this is an example of the eval report. Uh again kind of cut off at the top here. However uh I had selected a particular eval data set and then it shows a particular evaluation commit. uh in this case uh one of the the ones here um so of course this is offline as well meaning you don't actually have to deploy anything you can run it on your local machine uh but we also have it for for uh as a pregating operation for CI/CD for agents so uh wrapping up uh we're basically working on our path to the next level agents at Uber so what do we have right now we have our agent services in the bottom left corner uh eval debugging and tracing monitor ing uh guard rails provided by our security team uh the ability to connect to agent to agent either through MCP or this agent protocol uh long-term memory uh background executions code execution uh and of course computer browser use is also very interesting so we have a product at Uber that allows you to uh connect to your browser so it will talk to your uh langu agent FX or um uh agent builder agent and it allows you to interact with your browser Um, of course, uh, this is all built using our in-house SDK. Uh, but of course we do also have the ability to bring your own SDK. So, Langchain, Lang Graph, uh, Crew AI, all of these are powered by agent FX
So, you can bring your own framework if you want and again it just has to hook into our native interfaces. Uh this has allowed us to build products such as Uber agent builder uh as well as expose an agent registry. On the right side we uh have an example some Uber services such as uh Jira up as our deployment system uh M3 for metrics uh logs and then we also have our data lakeink uh which is powered by Presto Hive. Of course we can also take in additional Uber data uh through Confluent Slack Docs. You can either have this powered through open search uh or another MCP server. And all of these are powered through a product we have called MCP gateway. This essentially exposes Uber services uh as well as custom MCP servers uh to Uber agents at uh well at Uber. Uh and then of course this is all uh accessed through our our model gateway supporting uh all of our major providers such as uh you know open source like Llama, Gemini, uh all the GPTs uh Claude, Mistrial, Quen
All right. Uh so I think that's pretty pretty much it. I think we have a bit of extra time for Q&A. >> Do you have time for asking questions? Thank you. >> For your nextgen stuff, is that do you see that really being like the base or part of your L4 L5 journey of maturity of agents and and does that go a lot >> the nextG stuff meaning here? Sorry, you clear? >> Exactly. >> Yeah, of course. So, I think I I think one principle is very true, right? um you know some some applications that use more like a static graph so for example like lang graph are very powerful for determinism but as um okay that's weird [laughter] I don't know why that's doing that sorry um as as we start pushing towards better models and they become more and more intelligent uh you're going to start seeing and we're already start seeing at least at Uber more dynamic applications so being able to spin up agents or talk to other agents uh through our registry so if you have a series of deployed agents being able to intelligently decide what steps to take is really going to push you towards that L4 autonomy. Uh for example, uh G again given the registry of all agents, if there was a query expert at Uber, uh instead of trying to do your own querying, the agent might eventually realize, hey, I can just reuse that existing query agent uh as long as that agent's high quality
And again, the the key thing here for pushing towards L3 to L5 would be eval. You need to analyze the quality of each agent. uh for example, why would you want to uh utilize another worker who has bad output, right? So that's an example of uh even in humans. So we got to make sure that in our registry um you have that that sort of quality score published not only for humans to view which agents to use but also for other agents themselves. I hope that answers the question. >> Thank you so much. >> Thank you. >> Um anyone else have questions? No
Oh, here. >> Hi, thank you for the talk. Does the do the gateways and I guess this would probably live inside um maybe not agent effects but maybe uh agent studio. Do you do any sort of cost monitoring that's built in in terms of token usage and broken down by teams? >> Yeah, that's a fantastic question. Yeah, we do. Uh so we have a dashboard that tracks all this cost by token usage. Um I think the the one of the challenges with the the model gateway approach is that uh the cost changes for these models all the time, right? So uh for example, there have been a few times where the uh the token cost has gone down or sometimes even goes up but typically you know typically goes down, right? So we do track them at a more like per token count. Uh, of course we we have the ability to still utilize vendor rate limiting, meaning if uh a particular person is using too much, the vendor may say, "Hey, you no longer allowed to." But yeah, to to basically answer the question, we do have uh cost monitoring built in
Uh the model gateway is essentially a proxy. A lot of the time it's a pass through especially to something like OpenAI today but we have the ability to track all the cost and usage uh on a per consumer basis which is great for auditing for legal security but also for uh owners who want to know how much uh their agent costs. >> Anyone else questions? Oh, I see. That's that's a good question. So, to I think for folks who may not have heard that. So, uh if we're tracking cost, do we also track the outcome such as productivity? Um I think that's that's actually a really good question. Uh because we do track productivity for some of our products. Like for example, Uber is a pretty big user of cursor, right? Uh that's one way
uh that doesn't I think that goes through our model gateway uh I'd have to confirm that one actually um but there is but we have other ways to measure productivity but for individual agents that are built on our platform uh again the key it's kind of hard to determine productivity increase like for example the product that I worked on last year called query copilot uh we we didn't use tokens to necessarily uh improve the or sorry like the I guess the responses themselves to uh say oh this is going to make a user x more productive right uh what we ended up doing was uh we analyzed uh multiple user behaviors. So we have uh like Google Analytics and stuff on our on our side to kind of view who was using it and who's not. Uh we also of course heavily relied on our own evaluation saying like hey given this particular type of question how long does it take? Does it does it actually run? Uh so I think the the outcome is one of those really tricky things again back to the whole quality eval discussion. uh if you want to try to uh determine if if the agent will be good quality, having a good golden data set with a with a long like a very large series of different types of questions helps uh at least be a proxy to productivity increase depending on on what your uh area is. >> Good. Okay. >> Um what's like the easiest way to get started with this or like to play around with this tool? Is this all closed source? Is there anywhere we can go try something? >> Yeah. Uh unfortunately at this moment it is closed sourced
Uber is uh definitely looking at potentially open sourcing agent FX. Um I don't know if we have any updates on our our current strategy. Uh for now we want to focus on internal uh use cases and then eventually uh consider moving to an open source world. Uh so at the current moment no uh unfortunately you can't try agent FX out uh unless you join Uber of course. >> Okay. Okay. Great. Thank you
Um you must be recording a lot of uh chat conversations as well for evaluation and uh even in production etc. Um are you using a different kind of data store for storing this for either compliance reason or because it's not your typical log data from infrastructure or etc. >> Yeah, that's a good question. So um what we're what we're trying to do is rely on traces pretty heavily for that. So for example uh conversation interactions are are very useful to have through uh oel right and then there's a lot of different providers that allow you to annotate uh these traces. So uh I know L langsmith is one uh Arise I think uh OPIC might be another uh so there's a few platforms that are available uh uh that we're currently investigating to basically double down on this annotations which then allow you to feed back into building building and improving your your golden data set. >> Great. Anyone else have more question? Yeah
No. Hey uh thank you for the presentation. Just piggybacking on his question. So for some of these uh multi-turn chat interactions that you have, how do you build your golden data sets and then are these synthetically generated ones or uh how do you like when you trace these uh data sets and create your own data sets, how do you annotate them and how do you build them? >> Yeah, that that's a very good question. So I think uh just I'm gonna restate it one more time uh just to make sure I understand. So basically, how do you get started with building your golden data set? Is it something that we we generate or is it something the the owners come up with? Is that essentially the the question? >> Yes. >> Yeah. Okay
Cool. So that's a that obviously depends on who you're who's building, right? A lot of the time for uh top use cases or even even non-top use cases, you kind of have some kind of use case in mind, right? And we definitely think it's it's one of those like iterative uh kind of things. So you don't start off with a perfect golden data set. you typically have a series of questions in mind that you want the agent to answer. So let's go back to my my Jura example from earlier, right? I wanted to be able to look up information uh maybe to close tickets, something like that, right? So given a series of questions, I should be able to to determine that the outcome uh is that the ticket was closed. So I can either have API calls uh in my my offline evaluation pipeline or I can use the agent response itself. Uh we we do have two techniques. You can either manually approve it or you can also use LM as a judge
Uh LM as a judge uh does have some drawbacks of course especially uh it has a bias. Uh one technique I've seen uh people use before is using two different model providers completely to help with LM as a judge. Um this is something I've tried out. Uh I' I have seen a slightly better results. But the reason why I say that is because if you use the same model to generate the responses and also have the model evaluate it, um you can have like some kind of statistical bias there that might actually say it's a better answer than it actually was or or vice versa. So so one technique I've seen folks do is try a different uh model for it. Uh but of course LM as a judge, you still typically need to review the results until you're pretty confident that the the overall model isn't going to produce a huge bias. Uh this is just goes with any kind of quality testing
You could have uh a very weak golden data set that doesn't capture enough scope and the LLM gets all of the answers. Uh this this happened with our text to SQL where we had maybe 50 questions. It was able to answer them pretty well. We throw them in front of users and then uh they go uh it can't answer my question. Uh and then we had to hit the drawing board and you know take that kind of feedback and and kind of circle back. So I think really the pipeline of uh trying trying out a couple questions, put it in front of some test users, take the take the history, annotate it and then build sorry and build an improved golden data set from the uh the user answers is probably a very good pipeline for building and improving your golden data set. >> Thank you. >> Thank you
>> Um we have about four more minutes. Anybody else have questions? No. Oh, >> I'm curious about how how much adoption are you seeing across the business with non-engineers building agents? >> That's a very good question actually. Um, so we were recently discuss discussing the exact same thing. So, of course, we don't expect many non-engineers to jump into our Python SDK. Uh, most of them are going to gravitate towards Uber agent builder. So I think the number uh when I last checked is something along the lines of like two or three thousand internal users using uh Uber agent builder. Um uh some of some of which not everyone of course is is necessarily building but the adoption rate uh is still pretty high
Um we do see a mix of engineers and non-engineers. Again, engineers who are maybe wanting to try out the prototyping approach first, uh we typically recommend start with the simplest thing, right? Maybe prompt engineering or chat GPT will solve your problem first and then moving towards a more complicated approach such as uh using our in-house tool or uh agent FX or SDK. So I think overall the adoption is definitely uh you know I think in the next year we're working with some key partner teams uh at Uber to basically um uh pushed it towards non- tech users even more. So very likely the Uber agent builder product will be the uh the non-technical product for most people but of course engineers are still going to use it. question. >> You don't have to answer specifically for Uber, but how big of a team do you need to support um this kind of architecture? >> Oh, I see. Yeah, that's a that's a very good question. Um I I mean our team is currently pretty small
Uh we we are working on expanding that team. Uh we've had maybe about 10 people for this year. Uh which is a fairly small amount of people. uh we were able to pull through and get a lot of this off the ground this year. So I think uh given the number of requirements that we have come in for next half I would say uh you probably want a good team of like maybe 15 to 20 people for for Uber minimum. Uh of course more people means less work for me hopefully. [laughter] Uh that that's not true I'm sure. Uh but we we have a lot of pretty pretty high priority requirements at Uber
A lot of different uh organizations within Uber. A lot of our verticals like Uber Eats, rides, not necessarily external but internal as well for business reasons are looking at using uh agents as a whole. And of course, I'm sure as uh at a size at Uber is it does come become pretty hard for a team of 10 people to maintain this. Uh we do our best and we do use agents ourselves to try to answer as many questions as we can uh internal documents and and whatnot. Uh but I would say that for a size like Uber, yeah, definitely more than 10. [laughter] >> We have um one more minute. Anybody else? >> Thanks for the talk. Do you find that these kind of tools are being used to support like engineering asks and engineering workflows or more kind of business or product workflows that are not as technical? >> That's a very good question
So I'll I'll basically state for engineering uh use cases right now. I think the number one use case has been uh on call assistance either analysis of alerts or uh the ability to help with support channels uh at Uber. So we use Slack pretty heavily for for support. So whenever there's uh a question uh agents are a perfect use case for for answering these. And then for non uh non-engineering uh on call use cases. Yeah. So again, this is kind of back to that point of many folks are looking at using uh something like Uber agent builder to get started uh to either analyze information, generate new content um or uh basically like I think probably our number one tool is our our data lakeink tool. So either running queries, analyzing the data, generating new queries, etc
So a lot of it is uh very businessheavy for non-enge but then for edge use cases of course uh either prototyping new products uh new applications or again most popular one has been on call assistance. >> Okay I think we're at time now. So thank you all for your questions. If you do have more questions I think you can still go to the interview room and ask Jameson about whatever you have in your mind. So thank you very much and thank you uh to Jameson for the great presentation. Thank you. [applause]