Beyond GPUs: Why AI Agents Need a Need New Hardware

Beyond GPUs: Why AI Agents Need a Need New Hardware | Vasanth Mohan, AI By the Bay 2025

Recording: Beyond GPUs: Why AI Agents Need a Need New Hardware | Vasanth Mohan, AI By the Bay 2025

Um, awesome. Great to be here at AI by the Bay. Um, and and excited to to be giving this talk. Um, really quick, just quick show of hands. Um, I I'm assuming I'm assuming everyone here is familiar with GPUs, right? How many of you have heard of kind of other other types of chips or other types of architectures that are out there? Okay. Handful. All right. So, that that's that's super helpful

Um and and we'll we'll go through through everything and I think there'll there'll be plenty of things that we talk about more technical some some that I think will hopefully help uh give give everyone a base understanding of everything. Um just as a really quick intro, my name is Vasant Mohan. I I've been at SNOVA for a little over a year right now. Um I and lead developer relations as well as product marketing. Um, and so day in day out, constantly talking with developers that are building on our platform and our cloud and using our chips to to build the next wave of AI applications and agents. Um, the the slide that I wanted to start with just at at a broad level and I had to actually update it earlier today um just because uh Google Gemini just released their their latest model is is kind of just an overview of how AI models have been evolving over the last year. Um this is uh this is a chart from um a really great analyst firm called artificial analysis and um this was um the they they constantly keep updating these charts on their website uh day in and day out. Um but this this was kind of taken as as a snapshot from from their u uh Q3 2025 report

And what you see is that over the last year that there's been amazing amazing exponential growth in these AI models. But as you start to look towards uh the the right or right right side of the screen, what you notice is that the intelligence which is what what is being measured here of these models is starting to to taper off and and you you actually start to notice that all of them are kind of starting to converge around very similar levels of intelligence. We're not kind of seeing that same level of exponential growth in in intelligence from from these models. That's not to say that that that it we're kind of in a bubble or anything. It's it's just that we we need some sort of ex extra boost to to improve the the model performance and intelligence. And you're not just going to get that purely just from working with these models in singleshot prompting. So really I I think the LLM intelligence is really just progressing incrementally. Now what we saw this morning was Google Gemini released their own uh their model

But again like in that chart it's only really an incremental improvement. uh as I was uh on the drive-in, I was just kind of listening into um one of the talks from them and and what they were basically saying is they've kind of improved the tone. They've improved some basic agentic capability. The cost is a little bit higher, but could you still use the existing models that are out there in the world today? The answer is yes. And so what we're not going to be getting is that same level of intelligence exponential growth that we've seen in the last year. Instead, what we're starting to see as the trend that that is emerging is that efficiency is more critical than ever. It's and that efficiency can come from a wide range of uh areas of the stack all the way from the chips all the way up to your agentic applications and how intelligently you actually use these models. And and when we're talking at the chip level, one of the the the trends and and headlines that you might be seeing is that we're at this uh inflection point in terms of the amount of power that's being uh constrained uh in in primarily in the US, but you see this across the globe, too

And that we need more power to run these data centers and run all of these workloads. So, a a metric that you might see that that's floating around is the number of tokens per watt. And I think that's a it's it's a decent metric for for measuring the the the the challenge that we have at hand. But I think there's an actually better metric that we um were part of a report uh a report with Stanford just talking about this. And what they've described in their paper which I definitely recommend anyone takes a look at and reads um is this metric called intelligence per watt. And I think a very similar metric that you see the the CEO of of Microsoft, Satia Nadala, talk about is tokens per dollar per watt, which is kind of effectively the same thing. But to to really summarize this, what what intelligence per watt describes is how much intelligence you can get out of the model for every watt of of power that you actually end up putting into the system. and and and what you can see in in their paper is they were looking at what's happening with local LLMs, what's happening in the data centers to really help address this problem

And they've come up with some very innovative ways to tackle that, whether that's using local for certain tasks where you can you can have much better efficiency and you have and use the data center for some of some of the bigger models and agentic tasks that that these these smaller models that don't fit on laptops could actually um actually need to be run on those servers. Um, and so that's where they came up with with the metric as as a means to to really help us quantify the fact that we we need to start optimizing for this to maximize the amount of intelligence that we're getting in this data center while we're power constrained and we're and taking power and building up power takes a long time. I mean that that was on the BG2 podcast. That's what uh Satin Adala was talking about is the fact that he has all of these GPUs. He has nowhere to actually go and plug them in. he doesn't have what he refers to as warm shells where he can he can put the power in. And so that that's how much of a problem we're at in in terms of the the economies of scale that that we're getting at with AI. So that's and and largely that kind of comes down to to some of the efficiencies that that you get with GPUs

They're they're really great for a lot of training use cases, but when you're looking at AI, there's a lot of different other types of use cases that are out there, especially when it comes to to AI inference. So at Salmanova, uh the the company's been around since 2017. They they've pioneered what they've called an RDU. So an RDU stands for reconfigurable data flow unit. Um so it's very different architecturally than a GPU. And we'll talk a little bit more about that in a second. Um and this is our fourth generation chip, which is the SN40L. Um, and being designed for AI allows us to do things that you can't really do with GPUs

And it allows you to have a lot more efficiency than what you uh what you get from the GPU paradigm. And to dive into a little bit more detail, I just kind of wanted to paint this picture here of like how how GPUs works versus how RDUs work. So, a couple things I I like to talk about right off the bat just from an architectural perspective is one these large language models have a ton of memory and need a ton of memory to to be able to run and having a lot of that on chip is extremely helpful for a wide range of use cases whether that's running these chips really fast in certain types of memory. So like S there three different types of memory if I just go back to the previous slide. So there's SRAMM, there's HBM and and DDR. Each of them have pros and cons in terms of kind of the speed and efficiency that you get when running these different models. And you want to use them very smartly and intelligently uh at different points of the time when you're running these models. So with our chips, we actually have a very unique three- tier memory architecture that allows us to optimally use the the different sets of the memory that you have on chip to run these models

The the other thing that really uh helps kind of from an architectural standpoint is so if you look at the GPU paradigm, the way that you it it works when running let's say the kind of the base transformer model which is what really kind of kicked off this AI revolution is there's a bunch of different matrix multiplications and a bunch of different data that that needs to get run through various different operations and every time you need to one many one run one of these many matrix multiplications or different um applications uh uh uh in math uh for for the model you have to go to the GPU, you dump a bunch of data, you have to go back to memory uh to to fetch the new set of data, run that through the GPU, and then there's a bunch of back and forth that ends up happening between in the CPU GPU paradigm. What that typically means is that you have kind of low operator fusion and and a lot of lower local data locality when it comes to really efficiently processing the pipeline of of data running through these AI models. what where the data flow name comes from in in the RDU and how we've we've kind of architected the chip is having all of these matrix multiplications all run on chip and then have them segmented in different part so that the data can flow from one operation to the next to the next to the next to the next and so you're not having this dance back and forth between the CPU and GPU that you t uh that you would typically have and everything can run a lot faster and a lot more efficiently on that chip. What that ultimately translates then into is a much more uh bigger savings in terms of the energy consumption. So if you see kind of what's happening in the headlines today is people are building gigawatt token factories uh or AI factories is is is another common term that comes out there. they're using gigawatts of data and that's because at the rack level which is basically the the server that ultimately gets deployed into into the data center uh for for the uh Blackwell series you're running at 120 kilowatts at peak consumption when you look at the Reuben uh next generation so so not what's what's currently available in the market today but but uh a year from now uh is their Vera Rubins those are going to be 600 kilowatts so what's happening is because of a lot of these inefficiencies at that chip level, you're you're translating into higher and higher power requirements that are required for AI and for required for us to get to those next levels of AI. And it it's purely because you have a hardware inefficiency problem. When you look at doing a completely different architecture like the RDUs, you get to a power profile as especially in the inferencing use cases averaging 10 kilowatts

So you'll see it kind of in that band. And the the the beauty of it is that uh they run extremely fast too. It's not like you're using less power and you're getting less performance out of it. What what you actually end up finding is when you're measuring energy, which is measured in jewels, it's power times time. And this is actually a much more important metric when it comes to actually figuring out how much energy is getting consumed. You you'll see watts reported a lot, right? Gigawatt factories, megawatt factories, etc. um but it's actually the energy that you use which is purely the power times time and in that report that when we talked about intelligence per watt they also introduced a new metric called intelligence per jewel and I think that in my opinion that's a much more valuable metric for me measuring the amount of energy that's actually consumed because if you're taking a lot less time to complete a task that means that you're you're you're using the the power and resources way more efficiently than uh any other system that that might be utilizing and trying to complete that task. And and what they found in that report is that actually with the SANOVA systems, it's not just the power that was significantly lower

It's more importantly and in my opinion more impressive that it it it's taking less time to complete a loss a lot of tasks, which translates to a 4x energy efficiency even compared to in to to the black wells. And so these are concepts that I don't I think there's a lot of confusion about in in the market right now like what's energy what's what's power um and and and I think it's very crystal like clear that like having a complete understanding of what's happening even at that energy level is what's going to be needed in that next wave moving into 2026 as we're talking about efficiency and and just for reference so at Salmanova we have our own cloud product um it runs a wide range of these open source models that are out there today. Um whether that's the the llama models from from meta, whether that's open as GPOSS model or even some of the Chinese models like DeepSeek. Um and what we're seeing is that our our speeds when you go and independently benchmark these things on on artificial analysis are significantly better and that is what ultimately translates to a lot of that energy savings that you end up seeing um out out of out of this architecture. And these are some of the many reasons why efficiency really becomes a really key and important theme as as we're thinking and looking at uh 2026. Um so now talking about agents, right? So we we've spent a lot of time talking about kind of the chips and what you can kind of do from an efficiency perspective. But when you're talking about performance and and moving away kind of from from the energy metrics, when we we think about agents and why why does speed and any of this actually matter, it's when you're starting to think about building agents that need multiple different LLM calls and different models that they're working on and chaining these requests together. each and every time you go back and forth to to the data center, it introduces a lot of latency and and if each request in and of itself is taking a long time, the end result of of what your angentic application will will result in is is going to be very slow

This is one of the reasons why when you look at a lot of deep research um tools that are out there when they're they're constantly thinking in the background sometimes that that takes minutes and and oftentimes a lot of these deep research tools uh will have to go out for hours or or at least half an hour to to generate your report. And that's because it's doing a lot of this back and forth in the background calling different requests, processing, analyzing, refining. Um, but imagine if you could have each of these requests that are kind of in this sequential chain happen significantly faster. That's why speed ultimately matters in in a lot of these agentic app applications and in use cases. If you can take things that would take minutes and bring that down to seconds, that unlocks a very different iterative process for coding. It unlocks a different process for creative writing, etc., etc. There a bunch of different what whatever the use cases are that you're you're thinking about using AI for. That's that's how you get to that next level

um with a lot of these agents. So uh just to to conceptualize that right so today um a lot of kind of the the use cases that we see in the market are really around these kind of chatbot use cases in the enterprise you'll see a lot of different knowledge assistant based use cases and we're we're getting into that process of chain of thought where where these models are thinking quite a bit but when you really get into agentic use cases it's calling out to a bunch of different tools it's calling out to to various different models to make those requests for you and the tokens that you need to to actually get to the end to end result that you actually care about are are going uh up to an order of magnitude of of of 100x compared to um what what you've typically got when you just go and go to chacht and and write in a request and and that's the level of scale that that's really causing us to really think more critically about efficiency too. It's not just about the fact that it's just kind of one back and forth to the model. It's that to build a lot of these agents and agentic applications, you're going to need a chain of things that are happening in the background to fulfill the requests that you need, but that that's expounding the amount of demand and and uh that that we need on the on the back end to to actually support all of these agentic workloads that that are delivering actual value. So, uh if if we're to break that down even further, right? So what what what is an agent and what what uh what what do we look at when when we're talking about agentic applications? So uh for for us the way we think about agents because I know that the the market kind of there's a wide range of definitions out there is you you tell this agent kind of the same way that you would a human a task and it is able to go out and successfully complete that task on your behalf. Um whether that task is writing you a report that you need to give to your boss, whether that report is generating some po piece of code for for for whatever you're building, your your goal is to give that input. Then it's going to just the same way as a human does go through think about plan what that it needs to do, execute by calling a bunch of different tools that it needs to to uh to use to to actually complete the task. And then I this is kind of the optional step is is ideally you kind of have that feedback and learning in a duration mode that then improves it for for the next task

Right? So very similarly I think that the easiest way to think about it is if I were to give this small task to to one of my employees what what would that be and what's that bite-sized thing that that I need this code that that's no AI agent uh for it to run and do. And the goal is not necessarily for you to go back and forth prompting this thing a bunch to to actually make something happen, but it's actually more about just giving that task and letting it run. Um, and then this is just kind of an example breaking that down a little bit, right? So, uh, again, like in this case, it's like analyze my CSV. So, like if this was kind of a data analysis use case, it's going to go in, it's going to generate, um, generate the code that it needs to run it. Um, we have a great demo that I'll I'll show in a little bit, assuming the internet cooperates with me, where um, you'll have our agent demo. It takes that that request. It then goes ahead, generates some code. It'll be able to execute that code with our partner Daytona, which is a coding sandbox environment

So, so you can securely and safely run code, and then formats that end result and spits it back out to you. And it's in doing that it's really then taking instead of you just going to an LLM asking it for information and and getting back a result it's now getting you into that motion of turning those thoughts that you had and actually executing and actioning on it. That's the critical thing that we see the industry starting to move towards uh as as that next wave to really improve the efficiency of these these AI models and really deliver a lot more intelligence and results from from these models and workloads. Um, these are just some of the examples that that we've started to see people use our agents demo for, whether that's in data analysis, financial analysis, coding, some of the many ones that I've talked about already. I think there's a lot of interesting things. Um, I think scientific computing is is it's very early, I think, but like you'll start to see a lot of people in the market today starting to think about, well, what what could that that next level of innovation be if these models actually figure out how to solve new problems that humans haven't even solved before? Um, or create more more value from business intelligence or even say predict uh kind of uh and manage my portfolio of of of stocks. Um there's a lot of things here that that agents are really gearing up to to to improve upon. Um and it's going to the the key here is that agents are actually improving the amount of intelligence that you actually end up getting out of these systems by chaining these prompts together and using them in effective efficient manners

So um like I said there's a lot more efficiency gains that that you can really have with agents. We our own engineering team just put out a blog on is it LMS that are actually solving these software problems or agents that are actually doing it and um may maybe not a shocker is that you if if you try to throw a bunch of contexts at an LLM it's largely going to break that they while they can handle um volumes and volumes of text into into um into the into the model. What what typically ends up happening when you do kind of these these big shop prompts is it ends up hallucinating a lot more then if you actually have an agentic task where you've broken down the problem and or more specifically the agent also breaks down the problem for you and then starts executing on it you you tend to get to a much better form of accuracy. Um the other area of research that um our engineering team put out uh believe three weeks ago and and more as kind of a sneak preview here for this team that they're actually open sourcing some of that code um tomorrow is uh around uh agent context engineering. So instead of kind of taking kind of the these fine-tuned models and then going back to that point on kind of that feedback and iterative learning loop, what they've put together is this framework where you have a generator uh that uh helps you uh to to improve the playbook of how that agent works. So if you think about uh these LLMs and kind of the system prompts that that they have at their disposal, what they found in their research is that when you kind of do kind of a naive version of of basic feedback where you take a task, you ask it to reflect on that task and then updated system prompt, it ends up reducing the the performance quite considerably in terms of the accuracy that the agent actually ends up producing. So with the framework they ended up producing um which which they call ACE agent context engineering you can actually improve um with a few more steps in that learning process with a generator a reflector and a curator to actually improve the accuracy and improve the overall performance of the agent as it's continually learning and working with the environment um and creating what they call like a context playbook. So there's so much more intelligence efficiency that we're able to get at that agents layer, at the at the chips layer

And so that's that's really why I I do strongly believe that 2026 is going to be that year where it's really thinking about efficiency and improving everything up and down the stack to to really get the most out of these models that that we have um in the market today. So um I I'm happy to jump into demos. I'm happy to take questions at this time. Um, but I I do want to pause here. This is our website at at Salmanova. AAI, so you can you can take a look at um and learn more about kind of the chip and and and our hardware and uh al also our cloud and and see see some of the value out of that. Uh I I will jump into demos, but I do want to pause in case there's any questions here that I can help answer beforehand. So you talked about uh speed, inference speed, how fast it is

Do you have like numbers comparing to black belt like prefill and generation what you >> uh some of that information is in that paper? I don't have them off the top of my head but yeah that they're they're they're included in kind of the paper and some of the research reports that that we we put out there. Yeah. >> Which paper is that that you're referencing? >> Uh so that's the intelligence per per watt paper. Yeah. >> [clears throat] >> So God knows uh if you can tell us who can use 10 times less electricity you know climate change we'll be happy to hear that news but having said that uh if all the capabilities of models are converging to a single point I think you know with the new exciting use with uh post-process reinforce learning methods that's the way you're going to be able to distinguish yourself from your competitor uh are you guys fully like integrated with PieTorch. I I didn't see you know Sanovo at this year's PieTorch. Uh yeah. >> And if not, do are you going to introduce yet another ML dialect? How are you going to be able to make full use of your hardware without having to yet learn another language for kernel development? >> No, that's a great question

So um right now when when we um when we release models it's our engineering team that that's kind of working with uh at the compiler level and at the PyTorch level to take these models and run that on our hardware. It's on the road map right now to to kind of expose that and make that a lot more easier. But some of that is also getting a lot of those chips more accessible into the ecosystem and also into some of the research uh uh um labs there as well. Um, so that's something you you you should expect to see kind of o over moving into into the next year. We don't have those live just yet. Um, so what we typically do, um, especially because this is just how the market's working, right? Is a lot of these models are open source anyway. So it's really easy for our team to go in and optimize them. And even that's true in the GPU ecosystem today is that what you get kind of base out of the out of the GPU is is quite slow and very inefficient

there are a lot of teams and labs that are going in and and going in and improving that. So, right now it's it's really kind of at our team and then in in kind of the future iterations that that we're planning to release in the future, we'll we'll make those a lot more accessible. Um I don't I don't have kind of the details on how they want to expose that right now, but that that is coming and and we know that that's absolutely critical for the ecosystem. >> Why don't we jump into the demo for time sake? >> Awesome. So, um I mentioned that that that agents demo. Um let me make sure that um uh it seems like we're we're up and running. So, um th this is uh available at chat.sova.ai. Um and this is uh a mo uh a open-source e easy to use uh tool for for really anyone

It's it's kind of our deep research playground. So, um one of one of the things that I was kind of testing and iterating was like get me um should put this here. There we go. All right. Cool. Uh get me the uh top routes uh for for airlines for Thanksgiving. So, very simple request. most most of the kind of the chat um demos uh that that you see out there

We'll be able to kind of go to the internet, find that information, and then be able to generate that for you. Um so, we'll we'll give that a second here just for for the internet to come through. We can see kind of the a bunch of those different routes that that are here. Now, let's say I actually want ahead and visualize this as kind of a an HTML page. So, uh, generate a, uh, HTML page that has a map, um, of the top domestic. So, like we talked about in in that slide where we we we talked about it generating some of the code and then going through um, and and actually taking what you had and turning it into into an action. This is where where agents a agents are actually making multiple calls on your behalf. And what you can see here right now is that it realizes in that sandbox

It actually needs a dependency. So it'll go and install that kind of a map to help you visualize um the the the list of kind of these these top routes that that flights are taking for for the holidays. Um so if if we go back um once once this finishes here actually just stop that here. Um, and you can kind of see kind of compared to a lot of other use cases, it's running insanely fast. And we'll see that a little bit more in the cloud if we if we go up to the top here. So, this is kind of a pretty simple example where where it's going out to the internet and fetching some data. Let me see if I can make that a little bit bigger. Uh, yeah, that looks a little bit better

Right. So, it's using kind of about um 9,000 odd tokens here. um it's only outputting kind of a handful here and then it's uh generating um kind of a couple LLM requests to go to go and make the um make the when uh when we go scroll down you can see kind of the amount of input starts to slowly ramp up as it's generating and creating a lot of the that's required for for it to uh generate that end report. uh and uh and then it's going out also has to call call various different tools here to to actually run the code then be able to actually help you visualize it here uh on a map. So this is where kind of that the tokens start to expound when you're looking at a lot of these agent use cases compared to kind of just simply going back and forth and and talking to a prompt and a model which which maybe takes on the magnitude of like hundreds of tokens. Now we're talking about tens of thousands of tokens here. Um, this is all purely powered by by by Salmanova and and our on our cloud uh running on our hardware in the back end. So, if you head to cloud.somnova.ai and this is also linked on our website, you can see the the the dashboard of of um of our cloud

You can play around with a bunch of these different models here. So, this is the model that we're actually using, which is the Deepsec V3 model. Um, if I give it kind of a very quick prompt like write write a binary search tree, it's going to be able to do that and it's going to be doing that really fast compared to what you would typically get out of uh out of the out of the GPUs. Um, so any of these models are available for developers to start building and playing and iterating with. Um, you can create AP keys. Um, you can um take a look at any of the pricing and billing metrics that that are available here on our site today. That's all public. And this is really designed to be kind of that place where where developers can start to take advantage of of our models and capabilities

Uh to your to your point earlier, um these these are built by our team, right? So they take the open source code, they optimize that with with kind of our internal tools. Once once we make that more accessible out into the open, then the the the goal would be for for the ecosystem to really be able to help optimize and and run these models on that hardware and get a lot more of those energy efficiency gains that that you were just talking about. Um I think I'm at time so um if there's any other questions I can I can take one more. Um otherwise uh I think thank you so much