Reliable Agentic AI, the Open Way | Mark Collier, AI By the Bay 2025
Recording: Reliable Agentic AI, the Open Way | Mark Collier, AI By the Bay 2025
So anyway, we we put these foundations together and that brought together Linux, OpenStack and Kubernetes. But within the Linux Foundation, there's a lot of other really critical open source projects that are fundamental to AI that live in a number of different groups, including LFAI and data, the PyTorch Foundation. Some of them are are part of CNCF, which you may be familiar with is the CloudNative Computing Foundation. So, when you put all that together, um, you really have some of the most important building blocks for the future of AI. And one of the things that I fundamentally believe is that you can't separate AI from infrastructure. A lot of people think of them as somewhat separate domains, but I come from an infrastructure cloud background. And when I see what's happening in AI, I can't help but see the impact on the infrastructure. And if you look at what's happening, AI is putting incredible new demands on infrastructure
So just in the past year, Google has shown 50 to 100x growth in the number of tokens tokens they're processing and you can see that when you look at the investment going into data centers. So all this physical buildout, you know, we are all have this insatiable appetite for tokens, right? We can't seem to get enough tokens. And I think what is widely reported a trillion here, a trillion there, it starts to add up and we're building new power centers and everything else. But one of the biggest things that actually is scarce very supply constraint and the physical nature of infrastructure just takes a lot of time and incredible amount of capital is actually power and so one of the things I think people have not realized is that the only way we're going to be able to address that gap and deliver this type of um efficiency required is actually through software. So there's a lot of, you know, talk about every new Nvidia generation. People line up like it's a new iPhone trying to get the new Reuben chips or Blackwell or whatever it is, but you're probably going to get 2x, maybe 3x out of a new generation of Nvidia. But what both Google and Microsoft have recently been discussing is that they've seen 30 to 40x improvement in efficiency all from software. And when it comes to the software innovation that we need, really there's no way to do that without open- source
And you really can't have an AI strategy without an open source strategy. This is something I really believe fundamentally. And I think this is another example where people maybe think of these as separate things. Well, I need an AI strategy. Of course, we're going to use open source, but you know, what do these two things have to do with each other? And I think they're they're really inextricably linked. And for me, when I think about all the different areas in AI, it helps helps me to kind of break them down into different groups and pillars and think about the various open-source communities that are building these and what is the state of those communities, what is the state-of-the-art and the tools that have been built. So for me, the three pillars of open source AI and kind of AI generally are are training, inference, and agents. And inference is essentially just running the model right on the infrastructure
Training is, you know, building the model. And then agents are kind of no one knows what an agent is, let's be honest. Um, no, no clue. But I can tell you that uh training is a much more mature uh market. And I have a little slide that kind of illustrates this. This is definitely made up, not based on actual data. But if you think about like the size of the community, the number of people that are going to build or are building apps that are in some way plugging into like LLMs or other AI systems, it's millions. It's essentially like every developer will eventually be part of that community
And then if you look at the other end of the spectrum, the number of people that are training models, it's a very small group of people. Now it's highly leveraged and we're dependent on the continual progress on models. But in terms of like the open- source landscape in my view, if you look at something like training like PyTorch has 80% market share. It's very well proven. It's incredibly reliable piece of software and it keeps changing and innovating and if you don't want PyTorch, there's other other options that are also open source. But the agentic world is much much earlier, right? And inference is kind of somewhere in the middle. So when we look at uh one of the impacts of the training uh ecosystem and open source community producing something like PyTorch and the methodologies and I was just at the PyTorch conference down in Moscone a couple weeks ago and it's incredible what has been accomplished there. But one of the effects of that is we're actually seeing the rise of specialized models
And I think that you know um Satcha had this great podcast a few days ago and he said I can make the argument if you're a model company you may have a winner's curse. You've done the hard work and other people can essentially kind of very quickly come along and build build other models. And a lot has been talked about in terms of different models from around the world and open models versus closed models. But I think specialized models are actually kind of a unique flavor of that. And I'll give you, you know, a couple of real world examples that I've I've learned about recently. But the upshot of this is that if you have data in your enterprise, you have data in your company, and you have the access to these kind of um proven systems for training models, we're going to see a lot more models. We're not going to see three models in five years. We might see three million models, but we're going to see a lot more than three
And there there's some specific reasons for that. But the first example I'll give you is Uber. So Uber is training 20,000 models a month. And um you know Uber's somewhat unique in that they were born as an algorithm company. Um and I think you know in 10 years everyone will perhaps be an algorithm company but I think they're kind of showing the way forward and they can uh train them for specific tasks and the cost to train a model or with reinforce reinforcement learning or or smaller models it's actually coming down and yeah a frontier model might cost billions of dollars but you can build a useful model for a heck of a lot less and so Uber is really showing that. Um, Airbnb recently shared that they are running the Chuin model which is uh one of the uh leading models out of China from Alibaba. And I've been practicing the pronunciation. I don't know if I've gotten it right
Often we call it Quinn, which I've been told is not right. But um in any event, this is just another example where you have a company and again it's kind of an algorithm first uh native company because that's how they were started. So you expect they might be a little bit ahead. But if you assume that everyone's just using open AI, I mean that's just not the case um when you talk about enterprise adoption and these kind of specialized models. And so some of the drivers here are really much more cost-effective. And often times, you know, if a model's good enough, it might be just slightly short of, you know, state-of-the-art, but it doesn't that's not a reason as a business case. Well, we have to be on the state-of-the-art now. We want something that solves the problem
And when you with reinforcement learning and other techniques, you can really start to make a model that in fact is more effective, not just less expensive. Um, you also don't necessarily need the latest GPUs or if you have last generation GPUs that you that you bought, you know, you can repurpose them. And then of course, you know, on prim uh control over where the infrastructure runs, whether it's in cloud or other otherwise, you get a lot of control out of that. So the second pillar is inference. And one of the things that's wild about the inference world right now from an open source perspective is the rate of innovation. I've never seen anything like this ever. Um there was there was a recent example where um so Llama CPP which is a local model uh really cool piece of open source software for running local models. They uh you know this little DGX Spark came out
this cute little box you can run on your desktop if you have an extra $4,000 lying around and that uh in a few days after it was released, Nvidia committed code upstream to this open source project and increased the speed by 40%. And these examples just come every day. I'm sure we all struggle to keep up with the AI rate of progress in news, but the the the key point here is really that in open source, this kind of progress is compounding. So you have uh lots of different inference engines, lots of different engineers that are contributing from around the world to advancing this stuff. So every uh advancement sort of compounds the next one and that's why we're seeing this kind of you know takeoff type type of a scenario. Um now the the downside to that is because it's not as mature um as say the training ecosystem there's a lot of complexity and if you're an enterprise uh the idea that you have to update your software every time you know a new model comes out is uh it's a little daunting. So we have a lot of work to do to make this stuff easier to consume. Um so who who here's heard of VLLM? Okay, looks like at least half
So, VLM is the the most popular um not only the most popular tool for inferencing or serving models, it's actually the number one open source project in the world in 2025 according to this just released report from GitHub at Octoverse. Uh number three fastest growing. So, it just shows you inference is really kind of uh uh king right now. And um agents, you know, I said I didn't know how to define it. Um, sometimes I think it's just another cron job with a a fancy rebrand, but um, Andre Karpathy, who knows a lot more than I do, has said it's we're on the decade of agents. So, you know, we're getting started on something and we're going to really kind of have a software 3.0 as he calls it. And if you haven't seen him speak, he's amazing. So, highly recommend that
Um in terms of you know coding agents specifically you know this is another thing I I took away from Satcha Nadella's recent interview but he said that this AI could be the biggest category bigger than knowledge work which is kind of crazy because Microsoft's entire trillion dollar business is like tools for knowledge work. So um I think that you know it's not going to replace junior developers um in that sense. I agree with the AWS CEO. said that was the dumbest thing he ever heard. But it doesn't mean they're not useful. And I think there's been a lot of great talks here about how to make that useful. And uh you know it's coming. I think in terms of the open- source ecosystem and what is going to be required, it's quite a bit different
you know, discovery, context, security, there are new challenges and and I would say the only way we're possibly going to address these challenges is through open source, through open collaboration events like this. Um, there's just no possible way we could tackle this with like a single vendor or a single kind of industry, even single country. We need to do this all over the world. Um, and one more example I'll give is I recently was talking to a anonymous person at a hyperscaler cloud provider and they were talking about how they had allowed an agent to generate code which deleted root fs which not a good idea. Um, and so because of that they have a new policy that any new code that's generated um that goes really anywhere in their any system production or otherwise has to run in kata containers. I don't have time to go on to all of the details on Kata containers, but it's very cool open source project um that helps secure uh it's sort of like a sandbox. It's like a lightweight virtualization. So, you know, we're going to need open source to mitigate that that challenge of agents
So, the three takeaways here and I think I'm going to be on time is uh that training, inference, and agents are the three pillars of open source AI. And within those, you see in the training world because it's more mature, we're getting this ability to create these specialized agents. I think that trend is going to absolutely take off. You're going to hear a lot more about it. Um inference is driving the efficiency gains when we run the models, but also complexity. We got to help manage. Um agents, we're going to have more protocols. You know, MCP is getting getting more um it's it's improving and getting closer to to a general release version
and you know sandboxes like Kata and and just cooperating is going to be very very important if we're going to tackle the agent world. Um I very much believe open source will win if we build the strong communities and the Linux Foundation where I work you know that's what we do. We build strong communities that create software for for production. Um we want to make sure the AI native era is open the way we've been working so hard on the cloud side. So let's work on this together. Um you can connect with me. Sparky Collier and various social networks and uh you know look forward to meeting you all. Thank you.