LLM Avalanche: Nikunj Bajaj: The $360000 Question:Understanding the LLM Economics
Recording: LLM Avalanche: Nikunj Bajaj: The $360000 Question:Understanding the LLM Economics
thank you all right this presentation is going to be about not llms not math or maybe some math and a lot of Economics okay well we're talking about a 360 000 question do I have your attention as I mentioned some math some economics what are we doing so we all love Wikipedia in this case we are going to use a simple task take the entire Wikipedia and reduce it to half its size okay that's the problem we are going to be solving in this presentation but the important aspect that we are going to focus on is is it expensive how expensive is it what are the options that we have okay so take the Wikipedia reduce it to happen size that's all and we're gonna throw a bunch of llms at this problem and by the way please this is a very dummy task don't try this at home it's expensive it's pointless it's totally useless right only only for cost purposes okay so our favorite favorite llm jpt4 guess how much does it cost to take Wikipedia and reduce it to half its size using gpt4 that's the 360 000 question it costs three hundred and sixty thousand dollars to take Wikipedia and reduce it to half its size using gpd4 how do we do this well your gpd4 charges you based on the number of tokens that you send in your prompt and the number of tokens it sends in a response right thirty dollars for a million tokens sixty dollars for a million tokens do some math I might have made some mistakes in there but almost almost sure that I haven't made any mistake that comes out to 360 000 okay I'll let the slide be for like 10 seconds for people to do the math themselves okay move on so we have gpt3 we have a bunch of instruction tuned models with openai and if you use DaVinci which is the highest end instruction tune model that is available to you to fine tune that cost is half 180 000 okay and if you go to a much smaller model that is query okay query would cost you eighteen thousand dollar to do the same task all right now just to put it in perspective query is somewhere between 7 and 13 billion in parameters different estimates say different things okay so we can pick up an open source equivalent of this model all right I'm not picking an open source equivalent of gpd4 or Da Vinci but we can take an open source model equivalent of query and let's do the math side by side okay if you were to do the same task take Wikipedia reduce it to half its size using one of the self-hosted 7 billion model that cost is two thousand one hundred dollars okay so we have seen 360 180 18 000 and 2100 dollars the economics are looking good so far in favor of Open Source okay now let's let's make the math more complicated let's throw fine tuning so we have the model we have our own data set we want to fine tune the model right how does that changes the cost equation number one gpd4 you can't fine tune right now so it's saved but you can fine tune Da Vinci the Vinci is good if you were to fine tune DaVinci with Wikipedia and slash it to half its size the cost is 1.26 million dollars 1.26 million dollar to take DaVinci fine tune it and slash Wikipedia to half its size and how do we do this because the cost of DaVinci increases six-fold six times when you have fine-tuned the model okay so and obviously there is cost of fine tuning itself so that that number is 1.26 million dollars and QD which is 10x cheaper than Da Vinci the same cost for the fine-tuned query model would be 126 thousand dollars all right now let's do the equivalent cost of a self-hosted fine-tuned model the same model that I showed in the previous slide turns out that even if you fine tune these costs don't change actually all right so 350 and 1750 which is twenty one hundred dollars that we saw does not change the only thing that changes is you add some cost of fine tuning itself and your total cost is three thousand five hundred dollars okay so we have 1.26 million dollar with fine-tuned da Vinci and three point five thousand dollar with fine tuned small model OKAY equivalent query 126 thousand dollars so I think that's that's kind of the math that I wanted to show in this presentation to you all that if you're looking at open AI models and you wanted to work with you actually wanted to fine tune these models there's 7x more expensive right whereas if you take up open source model and you wanted to fine tune them they are a lot less expensive okay and our belief is that the way large language models are going to evolve is you cannot choose one or the other you really need some of the very large open AI type of models to do your generic tasks but at some point once you start putting these models to production cost will become a concern and at that point the way the world is going to evolve is these open source models are going to reside together with these large commercial models you throw the small tasks at the open source models and the large tasks and the open AI the more complicated task at openai models and that's how you balance out your costs that's a problem that true Foundry is solving so we are basically helping you when you are using your openai models to reduce your cost and when you want to use the open source model to help you host them okay now one thing I want to call out here so remember when open AI we were using open AI most of the cost was coming based on the number of tokens right everything is based on the number of tokens you send out but imagine when you are sending out a Wikipedia text to open AI right it has all these words like in off the all these things get counted towards your tokens now intuitively large language models are actually pretty good if you did not have some of those grammatically correct sentences so you could technically reduce the number of tokens you're sending and actually save your direct Open Air cost by 30 we have done that experiment it does not work every time but it works like 85 percent of the times okay so that's one way we can help you reduce the cost and the second is when you want to use your open source models true Foundry is building a platform where you get your model catalog of a bunch of Open Source models and drop in replacement apis for open AI so switching cost from openai to open source is very high you have to change one word in your code literally one word okay that's it if you want to learn more please sign up and I would love to get in touch with you all thank you