Build Better AI with First-Party Data and Fine-Tuning | Paul Yang, AI By the Bay 2025
Recording: Build Better AI with First-Party Data and Fine-Tuning | Paul Yang, AI By the Bay 2025
Hi everyone. Um just out of interest, how many people have fine-tuned a model or like have interest in fine-tuning models? Okay, so most people. Um I can keep things a little more concrete than um because I can, you know, give this talk a little bit higher level when the audience is a little bit higher level, but seems like people are in it here. Um like but as a motivating uh question like what is this article of clothing? This is the fashion emnest data set. Uh, if you've spent time in like traditional ML, this is a pretty hello world data set where you're just trying to guess is this a sneaker, is this a shoe? And if you ask something like GPT4 mini, it gets 81% correct, which is pretty incredible because it was not really trained on the task, but it's also really incredible in the sense that like an XG boost model where you just take um that image um and you just like plug it into a stats 101 architecture model and you get 90%. And if you train a reasonable reasonably small ResNet, uh you get about 95% accuracy. And really like the motivating point here is that models have always been adapted to data. Um there's this concept like the concept of pre-training a model for arbitrary tasks is very good
But in practice, the world of machine learning, the world is data science was always about mobilizing your first party data so that you get better results. Um and it is important here because the data that goes into the language models are not necessarily the data that you have at your organization. Uh so for instance for rag um rag use cases you know were hotter and more popular and we've discovered they're not everything but you know it was trained uh the first models that are in like this transformers category were you know BERT models that Google trained to be able to retrieve websites correctly. And so as a result when you use embedding models you get 95% accuracy retrieving web docs but uh for this one uh law statute benchmark on hugging face that I found is about 49% accurate uh for retrieving this kind of law legal document and uh you know voyage AI sold for $220 million to MongoDB to solve this problem of adapting embedding models to domains right so there's value here and uh it's pretty clear that there's a huge gap in the accuracy when you use the model for what it was trained on versus when you use it like kind of off spec. The other thing that you get beyond accuracy is that you can get better latency, right? You can take the same size model and make it more adapted to your task or you can take a much smaller model and adapt it to a task and perform better. So, uh it's really hard to read the small text, but basically what it says on the left is the probability of making a fix and on the right hand side it is the speed. And this is data bricks trained a tiny model that has you know slightly better accuracy than uh GPT40 which is the teal bar but it has almost twice the speed and so you get these cost and latency benefits from uh mobilizing your first party data to do it because it's probably true that opus 4.1 or Gemini 3 can do your task but these are like trillion parameter models and mixtures of experts so they're not all active whatever but um you know they're huge models and you can just take a tiny model and for this quick fix use case all you want is a tiny model to make tiny edits to your SQL query and sending that off to like opus 4.1 and paying the token cost and the roundtrip cost of latency is just not worth it. Um, so you know, this slide just briefly summarizes what I was saying
You know, domain specific knowledge, task specific success, cheaper and faster. Um, which are all good things to have. Um, and and I think like part of the problem is that you have this very marketing centric ecosystem happening right now where it's like don't train models. It's so hard. You need to hire someone who's paid $und00 million like those meta offers in order to train models. And this is a little bit of like a cynical like self argrandizing. Oh, we're building such great things. But it also scares people off from like doing things
Um, you know, like I think the analogy that I would give is like either like imagine walking to the chief risk officer of a bank and being like stop training risk models. Like just give us your risk team, fire the entire risk team and we will be the only people from outside your organization that trains a model to like predict risk correctly. The other thing is like imagine if like Facebook just advertised through like Google ads, right? Meta is like a trillion dollar company printing money on its ads, but at the outset they could have just used like put some Google ads on the corner like you get on like small websites and but instead they collected that data and mobilized the data to create like a really good ads network. So ML has always been about winning with data and so um the last motivating thought is that every model has been fine-tuned and um you know I think this is now obvious to people but you have a pre-training step right where you kind of take the entirety of the internet and everything you can scrape and you do this massive pre-training self-supervised almost um to kind of uh just like learn the generic patterns of images, text, whatever, right? you have um these models like Dinov3 that were released by meta that were just self-supervised learning over images. So you have these pre-training step and then you have some post-training and increasingly you see that the post-training is adapting the models to increasingly complex tasks. So if you think about like why cursor or lovable is so good you go back uh about a year GPD4 comes out and on med things like bench which is a coding task benchmark you get something like 30 something% accuracy on the state-of-the-art a year ago deep seed comes out and it did significant post training over like verifiable like coding tasks and it moves that benchmark accuracy from like 35% to 60%. And then you take that same technique of extending the inference time compute and you uh doing some post- training with reinforcement learning and you kind of see these like frontier labs like open eye and claude uh kind of climb their way up into like the 70s um and I I don't know what the current like frontier but it's like mid70s right so through post-raining by adapting it to a coding task you got basically more than double the accuracy before you did the specific post-training task. And it allows for applications that build on that model like cursor and lovable to suddenly do magic
Whereas if with 30% accuracy they simply could not. So for me a lot of the tailwinds of like why did cursor grow so quickly was actually because of a post-training um exercise that was um a research exercise unlocked by deepseek that then like percolated through the ecosystem. Um and so you have all the data right like coding is a generic task that exists out there in the world if half like so many of the talks at this conference and in general it's correct that um you need to collect evals and you there's all these tools to help you collect evals. This feedback data is just incredibly valuable training data that um you know you should be using to mobilize for uh model fine-tuning. Um, you can do things like super uh SFT, um, soft prompt tuning. Um, I I believe there's going to be a talk on that later. Um, which is not the same thing as just like prompt engineering, right? Like you you're actually doing a little bit of training um on the prompt. Um, Laura for like uh kind of parameter efficient tuning, right? Like you're multiplying a bunch of matrices together
What if you could just add one more matrix that you multiply at the end? Um, RL as I mentioned, right? It's about generating a evaluation environment that kind of grades outputs. Uh you have an inference service that produces outputs and then you have a trainer. So then your uh inference service produces some results that passes it into your evaluation service which gives it rewards and then feeds that back into your training service, tunes the model which then updates the inference service in the loop. Um and then you know um I think the last thing is like this is maybe a hot take. Prompt engineering doesn't scale, right? Like you get you get to a certain level of accuracy and then your prompt is huge and then you like change something about your prompt and then suddenly like everything falls apart. A new model comes out and everything falls apart. It's just not a like very deterministic or like reasonable way uh to do engineering, right? And I think it's like fine for like prototyping, but you do reach a threshold at which like you would want an objective way to like move up the march up the gradient like you there's no like gradient that is like that you can move along for prompt engineering. It's really like spray and prey, right? And eventually it like gets good enough and your system might be sufficiently satisfied by like a pre-trained model in your prompt engineering
But like when you're doing training for instance on like reinforcement learning or you're taking the user thumbs up thumbs down from interacting with your chatbot that like directly marches you up the gradient of improving your system. Um and like there's this fear that like models will erase my hard work. Um and this has maybe been true, right? Like if you spend all this time as cursor building like a system on GP40 and then accuracy triples, then you're you're golden, right? Like u you you didn't need to do all of that work. Um, so yeah, like I think you want to build a data system and this is generically true for all machine learning and data science, right? You start with the off-the-shelf models and then you build some robust evaluations on that. I'm not saying that you shouldn't build robust evaluations and then you take that evaluation gauntlet and you can turn that into an offline data set where every couple weeks or every month or every couple months uh you take that offline training uh you take that offline data and you do one big research loop and try and improve your model and then eventually like you build your user base your YC company you scale a bunch and you have all these users you can take your online data and then just like propagate that on the fly back into your system automatically. And that starts to feel a lot more like um if you think about the traditional world of ML when you're like scrolling Instagram and you're scrolling those reels or Tik Tok, Tik Tok is retraining the recommener system every 15 minutes based on your online activity. Um and like you whatever you can debate me on whether that's offline or online training, but like the way that Tik Tok is really good is that literally your user activity propagates back into their recommener. Um and like under the hood like now like descending level like tuning is just like pietorrch training right so like there's a lot of expertise already out there there's a lot of code samples um you can just use prepackaged things like the hugging face trainer or you can use vanilla pietorch or you can use any number of frameworks and it mostly looks the same as like training any other model that has existed in the ecosystem but they feel different because the weights are suddenly higher
I think you have a lot of teams who spend um their time training models maybe single GPU on a notebook and that feels okay. Um when the model weights grow by order of magnitude or two orders of magnitude then suddenly you have a problem where your like infrastructure starts to become a little bit more tricky and I'll get to that in a minute. And then sudden there's also this point where like a lot of teams don't have a target metric. Like if your goal is like oh my VP told me I needed to deploy a chatbot this quarter. what is this chatbot going to do? I don't know, right? Then then you don't really have like a good like way to like improve your system. Um, and so go do that first. But then like once you solve the infra problem and the data problem and the target metric problem, you can do a Laura training for a hundred bucks, which for most people even as individuals is like not crazy. Um, the the point about the ML platform is I think more salient to me and it's like what we work on right at Runhouse
build a open source library to bring ML workloads to Kubernetes. So spend a lot of time with ML platform teams and spend a lot of time in the infra layer. Um but like most teams don't have a proper ML platform that allows them to launch and experiment with multi-GPU multi-node um like setups. Um if you're notebook ccentric, you're somewhat limited in scale. If you're working on a dev box, you're somewhat limited in scale. And you also don't have really good reproducibility when you use notebooks because JSON files are not code. Um, uh, maybe also opinionated. An okay temporary solution is that you can shove your workload into whatever fits into a single node
And like I view it as both like a positive pattern and a little bit of an antiattern that things like Olama or like um uh Axelottle have so much popularity because that is about like the fact that your platform cannot support arbitrary methods and therefore you must take very specific methods and like kind of like shoehorn it into like a library framework that like allows you to work in a limited domain rather than like maybe over the longer term wanting to like take your platform and evolve it into something that allows you to kind of train more generically. Um, and even for the teams that have some kind of ML platform, the state-of-the-art is like development through deployment. And this is um maybe a little bit subtle for people who don't really spend time with like Kubernetes or redeploying training jobs. But if you think about traditional software development, you can test locally. Like if I'm building a website, I can look, I have an image, I change the alignment, I can see what that looks like locally before I deploy it to my web uh Kubernetes and I scale it up to thousand replicas and I serve the entire world. Um, in ML, you don't have a GPU attached to your local machine. So if you're not sshed into the head of a a VM, if you're not sshed into like a VM and we're not like working directly on like a local environment with a GPU, uh you have to deploy your application in order to see what a print statement looks like. Uh, so you add a print statement to your training
Then you tear down your existing training. Then you repackage the Docker image, push it to your container registry, bring up new pods, load in the data, load in the model checkpoints, bring the checkpoints into GPU memory, and then the 15 minutes have passed just for you to hit like a print statement to see and then reloop through that 15 minutes. And so that creates really bad uh development velocity and drives people back into using like notebooks which has really good development velocity but doesn't necessarily um allow for like reproducibility and like recurring training. Um and then like I think this like all percolates down to a concept and you know people love like giving terms to things. It's like there's experiment escape velocity. I think a lot of teams don't do training because they look at the complexity of the task ahead and about like how much work it's going to be and say this doesn't have the velocity to escape like our planning session right and so it gets stuck as something that like people want to do and like conceptually want to do but you know even when you see an open source repo you're like ah well but like we just can't run this repo on our own data and so then it gets stuck in planning and like training is always like four months away for a lot teams. Um, but like the model that I think like is really interesting to think about on the infra side of the world is Snowflake. So before Snowflake, you had SQL queries that were all running single box
You would like SSH into a server like the SQL server or Oracle server if you're unfortunate. Um, and then you would like run your queries um, and it would work. And then if you had more data because you ran out of disk or your queries were running too slowly, you would email your de DevOps team and they would schedule some time and they would move you onto a bigger box. Um, but then Hadoop came and you had distributed infrastructure uh for like running queries, but Hadoop was hard. You had like separate people doing Hadoop queries, managing separate infrastructure. And so what why Snowflake was really good for me at least was that you could just take anybody two weeks out of school with an arbitrary SQL query and it could be 10 rows of data or it can be 10 billion rows of data and you just execute it from any environment like from your browser, from your local machine, from your Airflow DAG and it would just dispatch that query out to Snowflake which then magically distributed it for you and then executed and then returned the result. And you no longer needed to think about uh compute where your query was running, how to structure query. All you had to do was kind of have code and dispatch it into an execution environment
The execution environment is increasingly for us uh observing the ecosystem. Like we rebuilt our entire product around Kubernetes. Like it is increasingly Kubernetes. Kubernetes is proven at scale. It is uh has a really rich open source ecosystem. um it is really the only way to like efficiently do compute and so like like we are building cube torch there are other ways to approach this like obviously I have to bring it back to the product that we build but really what you want is a snowflake- like experience for your distributed training right uh the fact that you have a really large language model that needs 16 GPUs should be somewhat abstract to you and you should be able to just write the Python code for the training and have that execute because in Snowflake if you're running over 10 billion um rows of data you no longer have to do the Hadoopy stuff which is like rewriting your query and conforming it to like a DSL that runs in a very narrow way. So before CubeTrch uh and like our easy interface into Kubernetes for scale uh there's that problem where you have to redeploy for every single test uh change of code. Um you have uh steep learning curves where like you have YAML which is not really code either
um you need to learn about like Docker and cloud services and all these things and then um if you built things in a notebook you would like take that notebook you hand it off to someone and they would take one week to three months to turn it into a production pipeline and that was a bad experience on both sides. Um, other teams like you know sometimes people kind of uh give you a gunpoint and put you at gunpoint and then say like you have to translate it yourself and then other teams like you know have uh different versions of that. But like the whole MLOps thing just really reflects the fact that like you couldn't just work out of a mono repo and just like run code arbitrarily the way that like a data engineer would. Um with CubeTrch all these problems are solved obviously green check green check green check. um and it's a little bit unopinionated. So really like just to give you a little bit of like the APIs that we're building for like making training and batch data processing etc. very easy. You kind of just define your compute in Python
Um it's Pythonic. It's not YAML. Um and then if you want to say I'm going to do distributed training, you just say distribute PyTorch. And if you think about what happens in Kubernetes is that if you have one replica, it's one GPU in the pod. And then you say like I want four replicas and we just wire up the distributed communication on like four pods for you. And then everything just works. You take your trainer class which is locally defined in your model repo or whatever and you just say I want to send it to the GPUs. The term cube torch is really like indicative of what's happening right like if you're using torch uh for GPU training what's happening you're doing model.2CUDA you're actually producing a copy of your model in GPU memory and then when you call model.forward in like your CPU process um it's making that call out to your GPU doing the computation on your GPU and then returning the result to you
But all of that is abstracted from the user, right? And as far as the user concerns uh they have some training code, they have their model and they just have moved it to the GPU. In the same way, you have some local training code, a regular class or function and we move it to cloud compute for you in the same kind of abstract uh zerocost way. Um inference is also important not just because you care about inference but inference is becoming increasingly important for reinforcement learning training loops. And so uh you know here's like just a little bit of a different decorator pattern for um how to do inference. And then when you can uh what you can do then is just call like uh by name the inference service. Um and this works really well to deploy to Kubernetes as well. Um, if you're using a thirdparty inference service, uh, I don't really know how you would do some of the reinforcement learning loops where you would have to do some inference, update the model based on evaluations, and then do the training and then update the inference service, right? Like that loop of RL requires you to be able to deploy both inference and training. Um, and so like zooming back out again on like a few things for you to like think about and u try
Um, you can run continued pre-training by feeding a large number of tokens in your domain into like an embedding model or a language model and try and give it some more domain expertise. Um, you can distill large model behavior into a smaller one. So go out and just do a bunch of inference over your like sample queries on like the biggest model you can find and then take that and just take those tokens and feed that into a really tiny model and hope it learns the behavior of the large model. And you can kind of see this uh in like the distill versions of Deep Sea, Quen, etc. where they train the really large model and then they very quickly uh get the smaller model to perform roughly as well on specific tasks. Um you should create a high quality label data set. You should like sit down with a team and have everyone label a 100 examples. And this will be your like ground truth highly um like highly well validated ground truth data set for um you know model performance but also for uh fine-tuning in the future
Um and every time a new model comes out you can just take this high quality data set and just run it. Um and then finally you should take your user feedback. I already talked about this but you should take your re user feedback and propagate it back. uh character AI when they were just released if you were like generating like I don't know your AI friend or AI girlfriend um you could swipe and they would regenerate the uh you know character for you and that was just propagating directly into their like training loops where if you say I don't like the looks of this image that was generated that's like the biggest signal that like you have done something wrong as a model and that propagates directly into a gradient of improvement for your model. um whatever this is a slide if you're interested but you're like h I don't have any compute we do have a serverless option it's also open source so you can install in your Kubernetes cluster um I think I'm running uh low on time or okay on time okay on time um so then I'll talk about this a little bit more um yeah like cube just deploys a by a Helm installation and a Python client um if you're using Kubernetes that should be relatively familiar to you if you just want to play around with like distributed training but on somebody else's cloud. Uh we do have a serverless option. Um then you can you know it's early for us. We're a small company
Um you know find me and we'll get you set up. Um think about it as just like somebody else's Kubernetes management but just entirely regular and you can use it as is on your own cluster down the line. Um there's no difference in the software installing them either. It's just the management of the cluster. Um and so can take a couple questions after this but um key takeaways always start with valuation. There is no magic. I am not misaligned in any way when I say you should do training with all the people being like oh like evaluations are super important. There's a lot of vendors for that
You should go out and pick one. um maybe some of the friends that are you know sponsoring here but once you have that valuation you must do something with it right like we used to like measure model drift in like traditional data science and ML and it's like it would be insane if you were like oh yeah the data has drifted but like I I know that the data has drifted I'm not going to retrain my model I'm just going to like watch it be misaligned with like the input data um so you have the evaluation that tells you what you want to optimize and then you can uh start mobilizing your data to introduce like the magic into your AI systems, right? Like I said, the magic of like training the model to do coding unlocked like billions of dollars for like cursor and lovable and you are doing something probably very unique and specific, not a generic task. I wouldn't say you should go out and go fine-tune for like coding tasks generically, but you know like what your company does is not going to be tuned for. Um maybe like a side point like I don't know this is the Bay Area maybe know like companies like Merkore or like um these companies that are being paid like hundreds of million dollars. Like I have a friend who used to be a PE and now he's being paid 150 an hour by OpenAI through Merkore to like make financial models. And so the point of that is that like OpenAI doesn't have financial models that it can train the models to be really good at building. And so therefore it goes out and spends $100 million paying people like my friend to like build financial models for them. But if you work at a bank, you have all the financial models in the world
Like you're saving $100 million on like that first party data. And like it's actually really silly to me that like JP Morgan is not you know training its own models to like replace investment bankers rather waiting for openi to train uh you know thirdparty models to replace investment bankers. Um and then finally like your platform should make you not scared to like try things. Um if you waste the whole quarter and like don't produce a result at the end of it that was a bad use of planning. And so like there's a little bit of this like hey like now is the time to finally square the problems with like ML infrastructure and um like and good platform engineering practices because like if you have a good platform and you can just run arbitrary code at any scale then you're uh you're golden to be able to do these experiments. If everything is a slog and it takes a really long time to run anything then yeah it's probably not worth it to do a lot of this model training. Um yeah and that's it for me and uh if anyone has questions we have five minutes I guess >> so hey house itself is basically sort of just a fake kubernetes system that solves all that problem >> yeah you you can think of it like um depending on what you're most familiar with like cubetor is like the open source library that just you can think of it as an interface into kubernetes for developers. Uh you go out to Google cloud and you buy a GKE cluster and you buy some reservation of GPUs and then you go like okay how do I use this Kubernetes cluster now um it's just like a Pythonic interface into Kubernetes where you like take your code as defined by regular functions and classes and you call that API like two and what that will do is deployment to Kubernetes and then uh future deployments as you iterate your code take like one to two seconds to iterate even when you're working at distributed scale
And then the second question on top of that, how much you >> how much do I need to do on the data side? So great, I I I buy I buy your product. What do I actually need to do in terms of collecting that high quality data and how easy is it to do that or are there tools for that? But but that would strike me as the other problem is great. Yeah. But do I really just collect all JP Morgan's models and shove it in there or is there what else needs to happen? >> Yeah, that's a really good question depending on what you do. um some some training is just shoving it all in. And I think increasingly you see like these like RL loops basically like the R the RL loops like are kind of proven to work if you have some ability to measure the success. So like um uh like for code right like the reason why RL works is like you have a model and you generate like so you have some like sample questions right like um you have a hundred uh you have a thousand coding input prompts. So then you have the language model generate the answer for the thousand input prompts and then you have an evaluation sandbox where you take the code it generates and you try and run it
If it hits a bug, you say that's a negative reward. If it hits success and hit produces the output you expect, that's a positive reward. And so then you collect all of these rewards and the outputs of the model. And then you take those rewards and the outputs of the model and you put it into your training service. And then the training service basically says all the things that succeeded we want to like upweight and all the things that failed we want to downweight. And so you kind of have just like these three components which then you can then think about in like neat way like like neatly separate. The training service is entirely regular like PyTorch distributed training. You have some ML engineer who can like do that if you hand them the data
you have an evaluation service which is just to say like for this particular thing do I have a ground truth to compare the uh outputs of the model and so for the JP Morgan financial model case it's like hey build a financial model based on this input data the AI model will build the financial model and then you have a ground truth financial model you like compare the cells with each other and it's like oh it either did the math correctly or it didn't or like it did the cell like calculations or not and then that comparison right based on the number of cells that look the correct um and like other things um then you propagate that into like the rewards. So like I think the RL is like the increasingly the popular one but like what I was saying for like um like for instance uh like the domain expertise embedding model right like something that you could just start with is just like shoving up like millions if you're in like pharma or something you can just take like an NIH database of all the PDFs of all the drug applications that have ever been filed and you just like shove it through the embedding model and then the embedding model will just get better at like determining the resolution of like things in that domain. Um, and like I can't guarantee it without having run the particular experiment, but it's like generically true that like the original like embedding models were trained to like retrieve websites and they just shoved all the websites through it and then it got really good at retrieving websites and you can shove other information into it and get better at retrieving that information. So um there is heterogeneity in like what you should actually do but like there exists probably some like if you google like how to do your thing like there probably is someone who is doing it in open source >> a couple >> couple questions I see what you uh just provide helmchart and uh python >> pipi uh libraries >> so anybody can inst everything. >> Uh, is it open source or >> open source? Yeah. >> Uh, do you have GitHub? Uh, >> sorry. >> It's it's open source library. >> It's open source
It's on GitHub. You can >> And can you give me uh why for example it's better than hugging face accelerate? >> So, >> can you compile them? >> Um, hugging face would be like I think at the program level, right? um hugging face like if you're using a hugging face accelerate with hugging face trainer um that's really good because it like helps abstract the training for you but then like you want to run uh the hugging face uh APIs over 16 GPUs how do you take your code which is the hugging face Python code how do you run that over 16 GPUs like the standard would have been actually like I think dockerizing your application pushing it up and like having like either 16 or two pods depending on how your like GPUs are structured and then making calls against the entry point on both of those pods right or you're you can use something like cubeflow pietorch jobs um which makes identical calls the whole principle function like the whole way that pietorch distributed training runs is that for every replica uh you make an identical call to it with like something like torch run and then um it will run all the replicas but the actual underlying application and the abstraction whether you're using lightning or hugging face trainer or whatever that's entirely your choice. The problem is like how do you get that to run on 16 GPUs is like what we're trying to solve. Like the interface into compute um is what we're trying to solve. Um but we don't actually care about the underlying method, right? Like you can use hugging face accelerate with hugging face trainer um which is all abstractions over pietorch. Um that we don't really care about. That would all live within your like my fine-tuner class in this code example, right? So you would implement like a train method on this class and that train method would essentially use hugging face to do the training, right? So that's at the code level and then we do the packaging and deployment. >> Okay, thank you
>> All right, that's your time. >> Um I can hang out in the hallway and if you have another question, >> I believe food has uh begun to arrive. So Oh no, we have it until 30. That's it.