Easy and Accessible LLM Fine Tuning with Manos Koukoumidis and Oumi | Agents By the Bay
Recording: Easy and Accessible LLM Fine Tuning with Manos Koukoumidis and Oumi | Agents By the Bay
The right model builds the biggest model. For your specialized task, it's best to have the best specialized model. How to build one? Let's dive into it at today's episode of Agents by the Bay podcast. I'm your host Oleg Doudin. And I have Manolis Koukoumidis of Umi with me today. >> Hi Oleg. Thank you very much for inviting me. Very excited to be at Agents by the Bay
>> Yeah, of course. Let's dive into your background first before we touch on the fine-tuning models. So, let us know where you came from and like what's your technical background is and how you get there. >> Absolutely. So, I'm currently CEO at Umi, Open Universal Machine Intelligence. I'm sure we're going to talk a little bit more about this later. Before that, I was at Google where I was leading science and engineering for all the natural language AI services at Google Cloud. I'm sure you've heard of Gemini
Before that, it was called Palm. I bootstrapped and orchestrated all the very large virtual team. We got to have 100 people that built Palm into its first GA version until it was moved to DeepMind and rebranded as Gemini. Before that, I was at a startup and before that at Ameba, also working on conversational AI, what now we call GenAI. Before that at Microsoft where for the first time in 2015 and 16, I built something like ChatGPT. Even what people call the rag now, I prototyped and built myself back then. If only I had coined the term for that back then. And yeah, before Microsoft, I did a PhD and spent some time at Princeton and MIT
And originally before that in Greece where I did my undergrad. >> Okay, so you mentioned Google. How long did you spend at Google? >> I was almost 4 years. The last two was almost when yeah, just when about ChatGPT was coming up for the last 1 and 1/2 years. >> Exciting times. So, I'm assuming that your Google background and prior to that uh helped you or made you believe that you need custom models. What happened? What did you see that made you think that? That's really interesting. >> Yeah, first of just in terms of a little bit of background, I got into that GenAI wave a lot earlier before most of the industry
Starting in 2015 or 16, uh literally we had an open-ended multi- multimodal chatbot. You could throw any question, we'd respond to you. You know, we even built a prototype of that back then. And that's what gave me the the belief uh 6 years later to say, "Hey, now that what we did in 2015 16 was premature, now's the time for us to do it again." And that's how I got to bootstrap and orchestrate all this week team uh to build gen AI. It's because I had that belief and I pushed to start this effort before even ChatGPT came out. Going to your question about custom models, one thing that was I would say not clear to most people in the very first few months that ChatGPT came out, but it was starting to become clear to me, was that these generic black box models, extremely impressive, but what I was trying I was already starting to see is that some of the most innovative companies they would say, "Hey, this Palm model, this sort of ChatGPT, they're great, but can you give me the model weights? Can I distill that model? Can I fine-tune it to my own needs?" And that's the very first thing I would say, again, starting in the first few months of ChatGPT coming out, that I started realizing that yes, as it happened with many other technologies before, the most innovative companies early on, the other ones to follow, they're going to start asking for more. If they realize that this is a critical technology for them, having a black box behind an API is not going to be good enough for them, and they will sooner or later realize that. Because they're going to say, "Hey, I want better quality, which you can get only with something that's optimized to your use case, or I want lower costs and latency." Again, you can only get those something that's optimized to your use case, not something that is generic and solves for every use case, or I just want better privacy and security, which means you need something that's can be deployed in your own trust infrastructure, or just better control, because again, this is so critical to your success that the only way to be in control of your own destiny is to own it and control it, not to be consuming it and renting it from somebody like OpenAI, Anthropic, or Google
And this is not just unique to gen AI, it's the same story for other technologies. And so, that's why I started building the conviction that history would repeat itself. Enterprise maybe were a little bit shocked in awe about, "Hey, look at this impressive new technology." that they forgot to ask for more. The early innovators, the most innovative ones are starting to ask for more. The rest will follow. The signals again were not very clear. Two years ago, I remember when I was discussing about this with investors, the vast majority of them would doubt it. They would tell me, "Build an app on top of ChatGPT and we'll fund you
Everybody's going to use OpenAI." I was like, "No, you'll see. The future will come. Sooner or later enterprise are going to move to wanting to own open custom models." And that's what we're seeing right now at a fast and growing pace. >> Yeah, so we love to look at the things that dynamic by the way. So, 3 years ago when we were organizing Scale by the Bay 2023, everyone was excited about rag like vector DB, which is to solve the same problem, right? You have your own data set that is specific to your knowledge and area. You use generic chat GPT, whatever model that was popular back then and you add your knowledge by adding rag, right? >> Yeah, yeah. Or whatever. Solution like that
So, why did you see that fine-tuning a model, distilling, whatever is the solution versus having that >> Yeah. >> Yeah, great question. So, I see rag and model training as complementary technologies. Rag is really great when you want to memorize some information like a knowledge base, right? That you and you can retrieve that information, give it into the model so it can form an answer. But if you want to have the model learn a new skill or capability, that can only be done with training. Yeah, with rag as a few-shot learning or whatever it's called, you can approximate that to a little bit, but nowhere close to customizing and training a model for the task. But to put it in a different way, let's say if you want to customize the behavior of your model, right? You can say, "Hey, can I use rag, retrieve example or two and teach it how to do this?" Again, it could work on limited scale or at least you could get you could get some some results, but you're still using a model that is an order or two or more larger than it needs to be. You're using a model that learn knows how to write poems and novels and do all sorts of things that you don't care about just to feed it some rag context or just for that to solve a very specific task that you care about
And that's where the value of model training fine-tuning Sorry, apologies. Comes in. It's that you can train a model that is an expert on the very specific problem that you care about. So, you can end up with a model that is an order or two or even more in some cases smaller, cheaper, faster, and can do at least as good if not better. And that rag cannot fix in any way. >> So, let's get a little bit more real with real examples. I'm coming from real-time streaming tech and rag would be used there because you can react in real-time to things like someone commented on your post and you can see that and via rag at that to in prompt LLM using that very fresh new information. So, fine-tuning is going to take training and etc
going to take some time. And the example, for example, would be if you monitor financial market and see some changes there and need to react them, then rag would be with real-time streaming would be the solution. So, give me like a simple example of fine-tuning where where that's in the industry of companies. >> there there's so many examples. Let's see. Starting with some actual use cases that we're seeing customers already building right now. For example, there is a bank that says, "Hey, I want to build a better compliance to take a better risk evaluation classifier." Given different types of documents, I want to evaluate the risk they contain. I won't go in more details, but you can get an idea
Some classifier that assesses risk in different types of content. They were using a large OpenAI model and what they saw is that they can build a much smaller model that can have the at least the same quality if not better. And again, it would be at a fraction of the size. Other examples, another enterprise build, a natural language to SQL model. That was Yeah, you could use rag to embed the context of the schema of your database, so it knows how to write a write SQL queries. But when they train the model, they managed at the end to create a model that has much better quality at a fraction of the cost. Let's see other examples, companies that wanted In that case, actually, there was almost no way around this. They want to build small models that run on device
That's also very very common use case, where again, there's no way to do this with a generic model, especially if you want to have high quality for your task. Other companies we work with, extract information from different types of documents, medical records, or other types of documents. And this actually a use case where they found that the biggest OpenAI model was just not good enough. And when they trained their own, they got dramatically higher quality at a fraction of the cost. Overall, I would say the rule of thumb is, if you're solving a very specific problem, then you should be using a very specialized tool. If you want something that usually that comes more in consumer AI than AI, a model that can answer any question about anything and do any type of task, then yeah, just go use a generic model then, that's GPT. But if you're solving a very specific problem, and you're trying to solve this with a generic model, it's just a misfit. It just doesn't make sense
You're leaving something on the table, which is quality, performance in terms of cost and latency, not to mention the privacy, security, and control you can have if it's something that you can fully own. >> Thank you for the examples, that makes it clearer for me and for the listeners, I hope, what to consider when it comes to mind to do something. So, this is Agents by the way, podcast, and we try to look at everything through the context of agents. We're looking at agents not as the single tone agent working alone, but rather as the multi-agent distributed system, where multiple agents work on their own tasks, and then orchestrator helps them to produce the final result and everything. So, my question would be, is there a rule of thumb when all agents should use one model, even if it's specialized one, or many little specialized models, or one generic model? >> Yeah, yeah, good question. So, one of the use case I mentioned before is actually what could classify almost as an agent because they were training a model that can extract information from different types of medical documents, and then automate tool calling. So, they can use that extracted information, make a tool call, retrieve more information, surface it to the human. So, it kind of qualifies as an agent in terms of the all the automation, the tool calling, and all of the steps
Now, going back to multi-agents and orchestration, this is actually one of the use case we're working with our current customers, which is how they can build a hybrid architecture of multiple different agents, orchestrators, smaller efficient models that can decide which agent to route the query to, which again, for example, this is a perfect use case for a specialized model. You don't need to be using even the uh uh GPT mini to decide how to classify a user question to route to the right agent. That's something you can do quite often with a much smaller model with dramatically, the same quality if not higher, at a dramatically lower cost, latency, and all of that. So, that's why there's a place also for all these specialized models where it can be both the ones that route and orchestrate to which specific agent it should go, and in many cases, many of those agents, very likely, depending on the use case, should be built off specialized models, not generic ones. Now, the thing I would say a little bit about a multi-agent orchestration, and there's a lot of frameworks out there right now that help you orchestrate multiple agents, is what we've seen quite often enterprises find out is that, because they're using generic models to build this multi-agent architectures, is that, especially the more you start cascading these agents, and one agent depends on the output of the next or orchestrates to the next and things like that, the more failures cascade, which means at the end, because you need multiple steps to be correct so that the end result is correct, outputs cascade. Why? Because each of the individual agents themselves are not as reliable as they should be. And then the question is how do you make them reliable? There's only one way, you build agents with means underlying models that are optimized to the use case and as high quality as possible. >> Well, um >> [laughter] >> I we are at the risk that we have a hammer and everything is a nail
Let's talk about when not when not to use that for for age. Can you come up with an example when you think, "Oh, maybe maybe let's keep the optimized model. Let's do something else." Like when would that be? >> Yeah, yeah, that's that's exactly actually I like that analogy. That's fine. I also often use. With models like that GPT, enterprises were given the biggest sledgehammer you can think of. And then everything to them looked like a nail. You're trying to push a little nail through the wall, a big one, a screw, doesn't matter
You can just swing that big sledgehammer and push it through the wall. That's how irrational, inaccurate, widely inefficient is what currently most enterprises do. They're not using the right tool for the job. The precision tool, the efficient tool, the tool that is optimized for the task, right? Is what we're discussing here applies not just to AI but to anything you can think of. How even a handyman works. So, you should use the right tool for the job. There's a reason why handyman has multiple different tools. Because he knows that for him to be efficient he needs to use the right tool
And it's the same thing actually we're suggesting to any enterprise. You should use the right tool for the job, not the most powerful one that would be not accurate enough and widely inefficient. And the only way to do it is to build a specialized one and by the way that's the problem we solve. We make it very easy for people to do. Now, the only case where I would say, "Hey, you should use the sledgehammer or the very big powerful model." Is again, usually I see that the only compelling use case is mostly consumer AI. You applications like ChatGPT where users may go with any random question they may have. And then you want to have a single model that can understand everything about everything and do anything. Then yeah, you need the generic model
In other use cases, I would imagine the vast majority of use cases an enterprise have because they're highly specialized, you need a specialized tool. Otherwise, you're leaving a lot of quality, performance on the table, and likely you're not going to be doing as well as your competitors, and you're going to be the one that's getting left behind. >> I'm sorry to push it again. Uh so, um >> Yeah, go for it. >> People going to be skeptical, because first, you need to learn how to train, evaluate, etc. You might not have big enough data set to train your model. Then, you can overtrain, too. >> Yeah, yeah, yeah, yeah
>> Let's Let's talk about each of this uh issues. >> That's a That's an excellent point. You went straight to the core of the problem, to the core of the reason about why what's happening in the industry right now is highly [snorts] irrational. It's because enterprises had no other choice. This is okay. I'm only I only have a sledgehammer. For me to go build my precision tool, the right hammer for the job, or whatever tool looks like, it's too hard. And we we know that very well, because we've heard that from very small tech startups all the way to some of the biggest organizations in the world
Say top 10, 20 banks in the world, and they all say the same thing. We understand the benefit of having the right tool for the job, the right model. We want all of these things, but it's too hard. It will take us months per use case, per specialized model that we build. In some cases, we're even worried that we not have the right expertise, or we not have enough. You know, we have so many use cases. What are we going to do? Have spend months and months for use case after use case to do this? And that's why we built Umi as a platform to help make that process dramatically faster and more efficient. It's a platform that does the end-to-end process of AI model development, evaluation, data curation, training, and repeat
Why it's end-to-end is because the job of an AI engineer involves all these six steps, and unless you tackle all the steps, then uh forgive me, you cannot build data in a vacuum. If you can't train and evaluate to see what was the impact of the data you just built, you just just doesn't make sense. So, you need to tackle this as an end-to-end job, and the only way to alleviate that friction that enterprises had is to tackle it head-on. And that's how now we can reduce the effort that took them months down to hours. And actually it's hours of world time if it's in terms of human effort, it could be as little as even minutes. >> Can we talk a little I have software engineer background. I'm not AI engineer and it would be very beneficial to me to listen to like all the steps that are involved in um like say I opened UMI and now I have this idea that I have great data sets about whatever like my I don't know let's let's talk about the Airbus by conference. I have my attendees and I want to see like which which based on the profile whether it's a good candidate to attend our conference whether they learn something or not and I have all this data that would help me to decide as a person
So now I want to train the model. So what do I do? Which model do I start with like open source which one why? Let's talk about it. >> Yeah, see this is all the same questions that even people who have some AI training that they are wondering is okay this feels too chaotic for me. I don't know where to start. That's literally things we have heard. But typically if an engineer was to say hey I want to build a model that's part of some agent or what what whatever architecture you're building. You can pick an arbitrary use case. Let's say for example I want to build a model that summarize news articles in bullet point format
That's actually an example I was using with a news reporter recently. The first step of the engineer would do is okay I want to see what is my current baseline. Which means I want to go test chat GPT, Claude, Gemini or Quen, Deep Seek, open models as well to see how do they compare on this task. Which one is better, which one is worse and why. This means you need to evaluate them. To evaluate them you need two things. One is you need to know what is the right metrics. What is the things on which you evaluate
For example for a summarization model you say hey I care about how accurate the summaries are, how comprehensive meaning they have the key points, how fluent they are, their faithfulness, meaning there's no hallucinations. So, there's all these different axes or metrics that you care about. And then once you know what are those that you care about, you need to implement their business logic to evaluate a model. Here's the response or an agent, you know, I can use them interchangeably. Here's the input, here's the output. Let me evaluate how well it did. And then you need also test set, which means you because you have the business logic, but you need a specific data set on which to test. So, what we do is that we automate all of that
For example, we need will tell you, based on this task definition, you can say, "I want to build a model that summarizes emails in bullet point format." Say, okay, here's the right metrics I would suggest to you. You can give it feedback say, "Yes, I want to add, I want to remove." But even if you have never built a model, you don't know what are the right metrics. It will suggest for you what are the right metrics to use. Then even if you don't have a test set, it will automatically suggest for you, I can help build for you a test set. Here's how I'm going to build it. It will show you the recipe. It's the equivalent of seeing the code in cloud code or, you know, cursor. And you can change it if you want before it executes and creates for you the test
And you And then we'll go what the AI engineer will do is go and evaluate the GPT-5, the Gemini, the Quen. And you can do the same thing in Umi. You can The agent will suggest you what models to evaluate and compare, or you can say, "Hey, okay, besides GPT-5 and Quen, I want also to evaluate side by side Gemini." Just natural language. It will automate for you all these evaluations and to compare them side by side. Then the next step an engineer would do is say, "Okay, I did evaluations. GPT-5, let's say it's 85% and the Quen is 80%. What is happening exactly? Where is the model failing?" For example, for the Quen that is, let's say, a little bit lower quality than GPT-5 out of the box. So, you have to look through hundreds or thousands of cases to understand what is What are the patterns? What are the ways in which your model is failing? Again, Umi automates this
We call them failure modes. It will tell you, "Here's the five or 10 most common ways in which your model is failing." Summarize them for you even. You can inspect the examples for each use case. And then that helps you understand how to better mental takes about what what's happening. Then the next step in engine it would take us like say okay, the coin was 80% compared to 0.85. Let me create the right training set to try to improve coin on my use case. Omi automates steps for you. It'll automatically build for you a comprehensive recipe for how to synthesize a training set even while taking into account the failure modes, which is what you would expect from an AI engineer would do
Which is okay. How can create a comprehensive training set to uphold quality fix those quality problems. It does all of that automatically for you. Then having the training set created, what an AI engineer would do is say okay, let me now go train a model. I need to pick the right hyper parameters, the right training algorithm, full fine-tuning parameter fine-tuning, distillation on policy distillation or some other approach. Omi automatically suggests that for you. Say here's the right model family, size, hyper parameters, and algorithm to use. If you want, you can change the recommendation
You have full control. But again, even if you don't know, it will make the right suggestion for you. Then after you train the model, it's automatically evaluated, compared side by side, and see how it improved compared to coin or even compared to the simplified model. So again, all the steps of an engineer, evaluate, understand what's happening in evaluations and dive deep, synthesize the right data to improve quality, train with the right model family, size, architecture, I'm sorry, algorithm, hyper parameters, evaluate. All of that process it will automate for you. You can go through all the steps that would take an engineer weeks and months literally in as low as a few minutes of your time. It may take hours of wall time because training synthesis evaluation takes time, but in terms of human effort, it could take as little as minutes. >> Yeah, I see that you say that you lower cost by using Omi and you just explained how that can take like minutes, hours, I don't know, maybe days or something of people's work, but the fine-tuning is still going to take time
So I wonder how you measure the all the costs and latency and everything for your customers. Like how does it look like now and what improvements they see like in what timeline they see that etc. >> Yeah, so the improvements typically can vary a lot based on use case. There's cases where you may train an open model and it approximates but not exactly beats or at least not significantly the quality of the best frontier model. But it can have dramatically lower cost like 10 times 100 times lower. At the same time we have seen use cases where you train the much smaller models 10 times cheaper, faster to run as well. That exceeds the quality of the GPT-5 or the vanilla generic model. So that varies greatly on the use case but we've seen all across the board
Typically again the results can be very competitive if not better at the fraction of the cost and the size. And we when we work with different customers we can suggest or actually something we're working on to provide more I think there's a blog coming out to that we can provide more empirical information about hey when you switch from GPT-5 to that model here's the rough savings in terms of cost or latency that you can see. Again typically they can be 10x depending on the size of the model you choose but we plan to to build a blog that's going to make it even more empirical and easy for somebody to yeah quickly understand what's going to what they're going to get. >> A lot of our listeners are open source enthusiasts and they might be hobbies so I'm I'm not sure like they're going to use Lumi as an enterprise with use that but I still want to bring a little bit of value to them too and just ask your opinion on different open source models which are great for which tasks and for different from the fine-tuning and >> Yeah. >> standpoint like what's the >> Yeah that's a good question and actually this applies both to as I said you know AI enthusiasts or even AI engineers at enterprises. If you ask them okay which model should I use for this task?" They may be able to give you some suggestions, but nobody may always remember or stay up-to-date as new models get released almost week by week and now, which one is the best for the task. Two things I would suggest here. One is, that's why we try to make it very easy to be based on the task to suggest you, here's the right model family or even size
If you're building a classifier for this use case, yeah, likely you don't need a model that is more than 4 billion, for example. Or if you want something that runs on device, maybe you should go for a sub-billion model. So, it makes all these recommendations for you. The thing I would say is, quite often the best thing is to experiment. You may say, "Okay, I'm not going to try perhaps some old Llama model, but between the latest say Qwen 3.6 or the Gemma 4, it may be worthwhile to try both of them." Because often sometimes, you know, even our intuition or whatever other things may fail us. So, that's why sometimes it's good to test them. And that's why we make it very easy. So, you can compare in the same consistent way side by side
You can even say to the agent, "I want to evaluate Qwen 3.6 and Gemma 4 on this task." And it will automate everything for you. So, it's often got to find that's the best way to, if you want to really make sure that you're getting the best performance. >> Well, thank you so much for this overview of fine-tuning as a task, like why anyone would need that and how to use Umi for to solve that. I want to move to the lightning questions. I ask question and you have 30 seconds to answer. Let's Let's keep it really brief. So, most overhyped thing in agents. >> The thing I would say is, I I keep hearing quite often about, you know, all the multi-agent architectures and orchestration that many people are trying to build and quite often then they say, "Okay, but they failed and didn't quite deliver the value." The thing I would say is, multi-agent orchestration makes a lot of sense in many use cases, but the best way to have a robust orchestration of agents is to make sure that the quality of individual agent is as good as it can be
So, focus more on the quality of individual agent, less about multi-agents and orchestration. So, optimize your agents for the task. I would say that's that's uh >> Okay. Uh most underhyped thing in fine-tuning? >> I would say that most enterprises still haven't realized the benefits they can get. And that's partially sometimes because some of the model providers, I won't name the specific one, quite often they try to brainwash people and say, "Hey, you don't need to fine-tune any open model. Just use our big model. Prompt it. If you prompt it, you don't get good results, yeah, you can't get anything better." I've heard that one name them, but I've heard one of the free model providers say this to me in one of the AWS re:Invent events, which is far from true
So, it has been I'll say underhyped or most enterprises don't realize the benefits they can get. Which should be, again, obvious if you use the right tool for the job. If you build a tool, if you build a model that is optimized for your use case. Quality improvements, dramatic dramatic cost, latency improvements. >> Well, they're interested in having customers, so they have to say that, I guess. >> Yes. Yeah, yeah. [laughter] >> Yeah
Well, uh best first model to fine-tune? >> Uh the model you're using in production likely right now or the next one you want to put in production. As long as it's a specialized use case, and especially now that it literally can take you up to minutes of your own time or hours to build it and evaluate it and compare it against GPT-5, there is no excuse to not give it a try. The The upsides in terms of the savings, the quality improvements are is just so high. It just does not make sense not to give it a try. >> Okay. And one future prediction. >> I think most agents are Humans who perform different jobs, again, are specialized. Most of most enterprises are going to realize that, hey, for me, when I have a job and ask a human to do it, I try to get somebody who's an expert on the job
The same thing should be true for agents. >> Thank you so much. Any closing statement you want to add? >> Perhaps the only thing I would just emphasize is any The thing at least I would say that I see, I think one of the biggest trends that are going to be happening in enterprise AI, not consumer AI, in enterprise AI, in late 2026 but even 2027, is going to be the transition from generic intelligence, generic agents that enterprises rent to specialized intelligence, specialized agents that they own. >> Thank you so much. It was nice to have you today at Asia Summit Debate. >> Likewise, thank you very much for having me, Arli.