Agents + Data Streaming for Real-Time Enterprise Intelligence | Peter Corless, AI By the Bay 2025
So yes, we're going to be talking about AI and data streaming which is in a way an oxymoron. Why? Because traditionally AI systems are batch trained and data scientists live down on that side of the hall and data engineers who are dealing with data streaming live down on that side of the hall and sometimes they see each other at the lunchroom or in a coffee break. But traditionally these have been opposite sides of your campus. So let's take a look at how we're going to be able to cross these streams. Who am I? Well, I've had a long and storied Silicon Valley career, but the thing you should know is I drink root beer and I know things. Uh, and I like to torture AI all the time. This is my own little benchmark. It's a microbenchmark called a D20 test
I play role playing games and so I asked them, can you draw a d20 with a 20 on top? Which for people who play D and D know that's a critical hit. And you can see that chat GBT did damn good except that there's duplicate 14s on this die. And so people would take you out back and shoot you in a western town for cheating at dice, right? So this is a problem. So it got the geometry right, it got the topology right, put the 20 on top, the number sequence. No, wait a second. There's not just duplicate 14s, there's duplicate 12elves, a duplicate 16s. Holy cow, this is a die. This is a loaded die
And of course, there's one number here, the 16 that has a visual artifact on it. So, is that really a 16 or is that 116? And I've tortured LLM after LLM and they all fail in the D20 test. Many of them, as you can see, can't even draw an icosahedron, which is a platonic solid of 20 sides, all comprised of equilateral triangles. I have yet to see an LLM pass the D20 test. And if it can't even draw an a realistic D20, why are you trusting with a billion dollars of your infrastructure in the next year's budget? So that's what my test is. So again, let's start at the batch job. Transformers came on the scene in 2017. All of the symbolic logic systems of the past decades since I was in undergrad at Carnegie Melon
All of them are thrown out the window when the transformer model comes along and everybody goes, "Oh, you don't really need context and you know history and all that kind of syllogisms. Who needs that? Let's just do a random word guesser tool." and we all pretended to learn what tokens and vectors and marov assumptions were. Right? I'm a guy in marketing. I do not have the mathematical background to actually understand these algorithms. But we've all had to deal with them. And as this onslaught of LLM has hit us in the market, they've grown by orders of magnitude, bound after bound after bound. Chat GBT1 was only at this point 117 million parameters. And we thought that was hotspit
That's what changed the world. They're now at 50 trillion parameters for chat GPT5 and 5.1 etc. We don't even know. They don't even publish how many parameters that they have. But what we do know is that it's not a single monolithic model anymore. They've had to split it. We knew since chatb4 that it really was basically five 220 billion parameter models all running in parallel. That's called a mixture of experts
We'll get into that in a little bit. But we know that these things are getting bigger and bigger and bigger. So, how do we even conceptualize this? How do we deal with this onslaught? Well, the other problem too is that just getting bigger does not necessarily mean it's getting better between chat GPT 5 and 5.1. There are actually regressions on certain benchmarks. the AIM 2025 benchmark, the TOW benchmark for telecom and retail both saw degradations between the models even as they're getting bigger and supposedly more precise. We also have seen it when a model was deployed and within the first quarter of it being deployed, it has fallen off that there's been drop off in quality as it's being used in production. This is not like your typical API. If you nail up an API and it works today, it'll work tomorrow
LLMs don't necessarily retain precision over time and again they're all batch trained. So you will see many slides like this. I'm not going to go through all of these but this is my homework and I can let this be your homework too. All of my slides are already available on my LinkedIn profile Peter Corass on LinkedIn and so you can have access to all of the research that I put into this presentation. Chapter two, the coming plateau. We are running out of data to screen scrape. You can see where chatbt is here in red. I added it to the chart
When this chart was made, chatbt hadn't been released. And so, as you can see, we're getting to the estimated stock of human generated public text. And we'll run into that hard limit sometime in the 20 late 2028s or 2030s, 2032. Eventually, we're going to run out of stuff to train these models against. Well, people say, well, why don't we break into private data? Why don't we that's the next frontier. We'll get private data, right? But you're already seeing a backlash of people not even wanting their public data being used to train some models. You're talking about ethically sourced data. [snorts] I don't want my kids photo to be used to train your system
I don't want my photo to be used to train your system. I'm an art student. I don't want my art to be your system. I am a business. I don't want my trademarks and my copyrighted materials showing up in your system. So if anything this publicly available data we might even shrink the amount of inventory we have to train against there's obviously cost inefficiencies there's energy intensity the amount of energy needed to maintain these models at scale and so far the real time feedback mechanisms have been limited to reinforce learning from human feedback which means if everybody's asking why this certain billionaire is such a quackhead head, the system will say, "Oh, I guess that rich person is a quackhead because that's what the inputs are training it to believe." So that's a biased reinforcement. And so we have to figure out a better way of training these systems in real time, building these systems in real time, nurturing these systems in real time. And it all has, you can't take a year or a calendar quarter to make sure that this LLM is not lying to you
So here's some resources for the coming plateau. Moving on, adaptive strategies for LLMs. Like we said, uh I said mixture of experts. Hey Mo is the latest strategy where you have a number of experts. One might be a butcher, the other one a baker, another one a candlestick maker. Each specialist in their art. And so there's a frontendian gateway that routes the query to one or more backendian expert systems. And so for instance, one might be specialized on Java, another one on Python, another one on Russian language translation
Right? So the mixture experts is domain dependent then. And then the router needs to make sure that it's balanced, that it doesn't just keep on going to Joe on the third floor because Joe knows Python better than any of the rest of us. Otherwise, Joe gets oversubscribed. And there's a lot of other problems with this strategy, including overfitting, branching. There's a lot of problems with it, but this is the best that the industry has right now. Another strategy is continuous training. either continuous pre-training, continual fine-tuning, but it will not avoid hallucinations. It will not avoid cat catastrophic forgetting
It doesn't fix misalignment. But this is the state-of-the-art again to deal with fine-tuning. They've tried to sample down how much you need to fine-tune. So low ranked uh adaption Laura has been used to uh shrink by about a 10,000th the amount of data that's needed to do fine-tuning training. It's a wonderful algorithm and I think people should know more about it because if you use Laura you can actually use pluggable modules for specialist type tasks. You can train something in a special task and then you can use that module for specific inferences. So per inference you might invoke a certain module but many pe many people don't even know that Laura exists never mind how to invoke it at runtime this is the latest thing that's actually I say latest the the concept of it goes back to 2020 not long after the transformer model but again I said we threw out all this symbolic logic AI but 2020 people are saying well maybe maybe there was something in that garbage pale that was actually valuable and so Last week there was an AWS paper where they combine an LLM with a symbolic logic system. The symbolic logic system maintains policies and a verifier model to then take a look at the LLM's output and it is going to go through these syllogisms to make sure that premises match conclusions
And so this allows you to to combine the best of both types of artificial intelligence because as we know LLMs can learn but they don't reason. These older systems reason but they don't learn. And so this is a new emergent model. Again this paper is a week old. I think they used a Ryionaire data set to train it against and they used the policies of Ryionaire to figure out like would this actually provide maybe a coupon code more than the plane ticket that you had. So why is it refunding you more than what you paid? So it's an example of how you can use domain specific things. This is also designed for highly regulated markets. Think about this for finance
Think about this for medical care where you need to have strong policy management as well as the creativity of an LLM. So again, a new model that's coming out. It's there, but many people don't even understand the sciences behind it. I'm barely learning. I'm watching video after video. This is my commute now. Here's an example of some of the validation. Can you do um this translation was ambiguous
Uh this is impossible. this answer um this answer is just simply invalid. No, this is satisfiable or this is totally valid. So it's using again just PC C type logic all the symbolic logic that we tried to forget from our undergrad or graduate school programs right but it's still there transformers did not get rid of this and plenty of resources for you to read all available on my LinkedIn chapter 4 that's all on the LLM side but now I'm an application developer I can't fix the LLM I need to deal with it as a blackbox in some ways So the first thing we had and this is only February of last year the concept of rag came out in a paper. We had called it different things before then. We knew that we could query a system of record and combine it with with LLM output but the concept of rag was minted in 2024. It's only been that long. And then as soon as it came out it was obiated when model context protocol shipped
And that's only a year ago today practically. And so now you at least finally had a standardized API. So you're still doing that system of record lookup, but at least the SDKs are all going to match. At least you're going to be using JSON. At least you're going to be using like a standard interface to your favorite databases. So it improved upon rag by standardizing the process. And then our company said, "This is really awesome because we do a lot of gRPC internally, a lot of gRPC." Um, but we wanted to turn that into MCP servers. And so we did and we open sourced this
So if any of you folks are dealing with gRPC APIs and you want to turn it into an MCP server, have at it. And this is the kind of acceleration. There's an onslaught of capabilities that we're all dealing with as technologists, but we're hoping to help by as open sourcing as much as we can to give you tools to build your next generation of applications. A side benefit of this, just as an example, is that if you're using a REST API, every time you're calling that LLM, you know, maybe sometimes you might be pounding a really big system 10,000 times per second, and each one is dial the phone, ring, ring, ring, and then you hang up and you dial again 10,000 times per second. GRRPC can keep a connection nailed up so you're not dealing with that kind of connection overhead. And I think we need to be thinking about applications of scale. Not just vibe coding in your laptop, but how do I go to production? Next adaptive strategy. In April of this year, Google announced a how do agents talk to other agents? Um, but right now, um, there hasn't been a lot of broad adoption
I've been taking a look at the videos. We saw them in April, May, June. Every dev rail was like, let's talk a right. But are people using it in production? And I think the answer is no. If you are using ADA in production, hello the internet. Anybody watching this on LinkedIn or YouTube, etc., please let me know if you're using ADA in production because I'm looking for that. I'm like Dioynes on this one. And when I took a look at it, this was my feedback
I c I wrote this. This is the ADA AI systems capability profiles. I wanted to understand if I wanted to do A to A, does that agent talk German or Chinese or English? To what grade level does that system talk at? Does it speak Python or JavaScript or C++ or Rust? Can it produce or ingest PNGs or SVGs or GIFs or MP3 or MP4? which forms of cancer will it be able to diagnose? We don't have a grammar to explain what an agent is or what it can do or why I would even care to call it from an Atoa protocol perspective. And as you can see here, Claude can only do SVG graphic outputs. ChatGpt gave me a wonderful PNG and Midjourney gave me a animated GIF. And that's kind of important before I call it and say, "Can you do something?" and says I I don't I don't do that output. Why am I wasting my tokens? So this is the kind of grammar that we need to build. It needs to be an extensible method to even understand why am I even using this AI in the first place
And if we take a look at the Google trends of the past two years, again, you can see the blue line where rag takes off in early 2024, but it never really peaked very high. You can see where MCP came off in uh late 2024. And that's really what's grabbed attention. But AAA remains near a flatline. And this is just people googling. So I can tell if nobody's googling it, probably nobody's running in production. Here's some resources. Now, we all want to get to production, but you can't run in production if you can't do observability
And I think we have some great examples of some vendors down in the hall today that can help you with observability and evaluation on on AIS. And here's your moment of zen. If you have AI, you need observability for AI. And if you have observability, you can use AI for your observability. It works both ways. But today, we're going to focus upon observability for AI. Traditional stacks look something like this. you go all the way from the instrumentality on the one end through the data collection to the storage the query engine the dashboards etc
So this same architecture that we've been using for classical systems need to be reinvented for AI you need to gather metrics from your AI systems you need your logs from the AI systems but more than anything else I think what people are seeing is traces from their AI systems and traditionally traces have been the hardest thing for a lot of uh traditional uh observability systems. So and then I would classify I would split the uh observability of AI into two different buckets. classic observability which is your performance tracking um your resource utilization typical tracing all that kind of stuff is what I consider classic observability which is any sort of distributed system that you're m uh managing but the new things the net new for AI is you need evaluation so data quality monitoring bias detection explanability data drift these are unique things that you need to gro above and beyond sorry to use the word grock but you need to understand these above and beyond what you've been doing for classic just system observability these are endemic to dealing with AI perplexity which is does the AI like does it provide you a logical next word for the sentence it's constructing so 1 2 3 banana that would be high perplexity right I would have expected at least an integer value so where where did the banana come And there are again uh OPIC is downstairs. I've been talking with them here. Um but there's Langfuse, there's Laminar, there's uh Phoenix from Arise. There's a number of different packages and these are just the open source. There's there's another closed source vendor downstairs. There's tons of them springing up and you need to decide what is important to you in doing your AI evaluation and observability
Langfuse does this great job at doing the distributed tracing. So you can see that middle column that tells you how many seconds this thing was in operation and what was the uh kind of the flow between the different systems and processing that it was doing on the back end. How did it arrive at this answer? Important thing again we've been talking to these people they can run optimization algorithms against your agents and prompts. So faster iteration loops uh easily curate and to remove failures from your loops uh and to support text in multimodal agents. Multimodal is like a huge thing in and of itself. Phoenix Arise does this really beautiful data clustering, right? Visualization. I've loved 3D graphs ever since I was in undergrad doing them Pascal at Carnegie Melon, right? But we all need great ways of visualizing data in novel ways specifically to the AI domain. [snorts] And now we get back to how do we do all this in real time because data streaming is everywhere
AI is everywhere. This is the mapping of my company Red Panda. This is the the the current uh verticals that we operate in. And if you just take a look at the top five like financial services, manufacturing, cyber security, adtech, gaming, these are definitely real-time industries. You can't wait till tomorrow for financial services. Like you know gamers will let you know in a few seconds lag, right? These are highly real time operations and underneath them is observability. Under everything there's observability. Every use case has observability as a shadow use case for it
And above every use case, there's an AI use case. How do I make this more understandable? How do I filter and factor and analyze what we're doing? How do we improve it? How do we build it in the first place? And underneath everything, there's a streaming use case. Under everything, there's a streaming use case. And so in a typical financial services system, you might have microservices and those are streaming telemetry off of them. Logs, logs, logs all the time. Metrics, operations dashboards, um trading desks, mobile users having their apps crash. That's totally unacceptable for somebody who's trying to sell their stocks right now. So there's all this real-time telemetry happening all across the industry
Poolside is a company that's specifically using AI with Red Panda to help facilitate what they're what they're doing in trading model iteration. This is incredible stuff. This is big brain stuff. I'm so I mean I'm jazzed to even be in this industry. It's it's transformative what we can do with technology these days. Manufacturing floor, they're no slouches either. They're doing real time factory floor systems. Uh they have desktop workshops, you know, sorry, say worker desktops
They have intrusion detection systems, right? They need to make sure that factory is protected from people, bad guys around the world trying to bring that factory down for gaming lag, right? Where is it? Is it lag because this guy has a last generation uh uh uh PC? Is it lag because the servers that he's connecting to have problems with the network? Is it lag on the backend servers? Is it lag fetching from the database? Where is that lag? How do I get that lag to near zero? If I can't solve that, my customers will churn. They'll find another game that's their's favorite to play tomorrow. And so this is the landscape that we're all faced with. AI, observability, streaming, and analytics on top that all need to harmonize in a modern data application in an end-to-end architecture. And this is why if you come listen to us at the booth, I'm not going to make a pitch here, but we perceive and you can perceive in your own organizations that there needs to be an agentic data plane, a way to unify data streams in real time, a way to do governance in real time, a way to do deployments in real time, that there needs to be some control mechanism that is specifically designed for the world in the age of AI. It's got to start with connecting your systems together. So we have Red Panda Connect. So if you have a system that's kind of it's funky, you know, we don't have standard connectors
You can write one in red panda connect and get telemetry streaming off of that. You can get your change data capture streaming off of that. If that's UDP package, you can get that streaming into the system as well. And not just your gazintas, your gazadas too, you know. So where's the target for this data? Where do I need to get this data to? And this is a toolkit because you can't spend a year if you're doing vibe coding and that takes you hours. You can't spend a year to get the pipelines set uh nailed up. It's got to happen at the same speed of thought that you're doing your vibe coding at. And that's my rant
Thanks very much. [applause]