Transcript: Jamieson Leibovitch, Uber, on Reliable AI — Interview with Alexy

Uh I'm Jameson. Uh I'm a senior engineer with an Uber. Uh last year I was the the lead for uh Uber's product called uh Query Copilot. It was our our text to SQL application for uh Prestos SQL. Um, it allows users to generate queries to get insights way faster than they would have if they had written it themselves. Uh, this was my first uh real deep dive into the agentic world. So to me, reliable AI is about having agents that uh answer questions effectively or are able to solve the problem uh with high quality. So uh ensuring that I know that the agent is able to uh take an action and I can trust that it will do the right thing. Uh a lot of this comes down to uh high quality evaluations uh ensuring that the the actual score of the agent and you know we have confidence ahead of time before we actually launch into production. Um so uh I think I think one of the things that help really power reliable AI is a is a good evaluation pipeline as I think I previously mentioned. uh being able to go from a series of questions, pass it to the agent, be able to iterate over it and see a see a trend. Uh for example, every time I change the prompt, does it get better or get worse depending on my answers? Um I think as the community gets better, maybe something uh either open source or um you know, maybe something allin-one that allows users uh or engineers or even non-engineers to be able to build effective quality uh agents. Um yeah like in the last five five years now we've we've basically exploded from like uh very basic agents uh such as like ChachiBT to even more complex ones. Um so again like we we've had especially at uh within our own company we've had no AI in the last 5 years to the explosion of AI in the last uh two or so years. Uh the models continuously get better. uh we've had uh single you know basic agents uh with some basic tool calling and data fetching uh to multi- aent static systems to now multi-agent dynamic systems. Um I think the the next step at least I can see in the next year or two is uh more like task management oriented being able to collaborate with each other more effectively. But the the problem of course is this all hinges on the models uh themselves. If the models get insanely good that you don't even need any of this, this will change the entire scene. Um, maybe we'll go back to a single agent that's able to do everything. Um, it just it's really hard to determine uh based on the speed of how how rapidly these these agents are getting better.