SBTB 2023: Chris Matteson, Cost and Performance Optimization of LLM Inferencing.

SBTB 2023: Chris Matteson, Cost and Performance Optimization of LLM Inferencing.

Recording: SBTB 2023: Chris Matteson, Cost and Performance Optimization of LLM Inferencing.

[Music] uh thank you all for coming I'm happy to uh be invited here by scale Bay to uh give this talk uh so first off who am I uh I run sales Solutions engineering training Partnerships Etc at a company named firon we'll talk more about what firon does uh slight at the end but I want want to talk today um about this white paper I wrote background previously as an early employee at puppet and hash Corp and C called Prisma uh I'm currently doing an Executive MBA at UC Berkeley I don't know if that matters in any possible way but everyone in my core tells every possible person they're doing it so I guess I might as well to uh so I wrote this white paper uh I'll put the QR code up at the end too uh this came out in September really thinking about the cost and per performance optimization on llm inferencing and let's kind of dive into the why of this paper and go through this so as I think everybody who's not living under a rock knows gpus are really really expensive and llms tend to take a lot of GPU memory and they require the most expensive ones you know good luck getting a spot instance pretty much AWS and Microsoft has said that's just not happening anytime in the near future so you're paying full Freight for those INF those uh uh gpus if you want them uh tools like Sky pilot can help you go find the cheapest GPU today uh but you still are going to go out there and go find it somewhere and then obviously we've all heard of open AI chat gbt uh and there's a bunch of other uh competitive offerings that charge models on a per token instance uh this can often be much cheaper if you're not going to fully utilize your GPU a lot of what we're going to talk today is about the challenges of fully utilizing your GPU and But ultimately like those problems that you would have doing that are the same problems that the vendors you're buying from like open AI are having those challenges of how do we deal with scale on things that are user impacting so going to go through that and figure out how that implies to your cost and how you can control that so really the big thing to think about is like what are the variables that I care about the most right do I care about the time at total time it takes to complete the inference do I care about the startup the cold start time for the inference do I care about accuracy am I trying to reduce cost like what are all of those pieces that you're trying to control for in your particular use case because the architecture choices you're going to make are going to have impacts on that and so if your workload doesn't need can run at any time right like I think it was um Microsoft somebody just launched a an inferencing service where they're like hey we'll do it in the next day for you uh if for you have an instance where I'm like I don't really care when this gets done as long as it gets done in a day you know there's competitive offerings for that and that's really where it's what's your workload right and we see a lot of these things like batch job and training that can happen at the cheapest possible times that we can go we don't really care if it when it happens we don't care if it happens quickly we don't care if there's a delay we can do it all at once hold it steady state that's great and then we have other uh issues and there's a lot of the uh Enterprises out there today that are building you know pretty much everyone has an AI play today trying to think about what is the way that we can use large language mod mod and AI inferencing at our company and a lot of those are trying to build things that are user facing so one example company I'm working with today is trying to do basically natural language SQL uh generation where they have uh the user they have a big database with all the information about the user's uh systems and they're like hey I want people to be able who are not SQL admins ask questions of this database uh so they want to do it in natural language and so they're trying to use AI models to generate SQL from that but because the user's doing it right you can't wait a day for that response to come back you need to do this in real time and when you have that real time implication your architectural choices start changing and you have to start making decisions around how do I balance the cost with the um the speed that I need this to occur in uh and so again open AI uh token based things are often one of the cheapest ways to do this and they give you the ability to scale from I'm not paying anything because I'm not doing any inferences to I can do a bunch of inferences and I'll pay for those and then I'll come back down of course that comes at the con the cost of flexibility right well that didn't work immediately for me like let me go get my own model so this was slide and I I've seen this a buns where people like they built their thing around open ai's API they gave it a question it gave them a like minute answer like just enough to be like useful for an internal demo but not product worthy uh and the first thing that people think of is like I need my own model or I need to go do some custom Training uh and they kind of go down this very large like the very heavy route of doing either lower weights or fully fine-tuning a model uh and that puts them into the place where I'm like I need to then host it because I don't have options otherwise uh or the options I do have are relatively limited and more expensive so this one of the things that I I like to push a lot here is like the first thing you do before you go down that route is really think about all the things you can do for prompt engineering there's a lot that you can achieve with chbt and the other token based models that are not just hey I gave it a prompt it didn't understand the prompt I give up now right temperature is a big piece to this right like I that really it varies the likelihood of what word it's going to pick next right we don't always want to go at a temperature zero we're going to pick the most likely word every time because we end up with very Bland responses that are often quite repetitive right and we have other variables like top P top K we have a penalty for repetition all of these things we can control and then we see increasingly like the way you ask a question really re like is imp impacts the answer you get out of that and so doing things that don't really make sense to tell a computer can actually be helpful because of the way it gleaned all of the data so there's a study that came out saying like asking the AI model model to take things step by step often like gets more clear results there's another article I read the other day where absolutely just cursing at the thing actually got even better results you know it's kind of weird the the things that fall out of this black box of large language modeling but you know I I urge you to like before you go and say I'm going to go invest in fine tuning make sure you've done your due diligence to go through prompt engineering because that often will give you more flexibility if you can do this on standard models without going and do fine tuning and all of that effort so okay I did The Prompt engineering but I want to host my own model because whatever reason uh and for many people that may just be I don't trust open AI with my data right I'm like I needed this be internal uh I want to use a model that's different than that uh you can there's a thousand different reasons but once you've gone past that point where you've ruled out these token based systems uh then you're thinking about like okay well I need to host this or I need to find a vendor who can host this for me and I kind of get into the situation now thinking about well what does that look like so particularly for workloads that are based on user input we tend to see this kind of thing we have different times where we have less workload and we have different times where we have more workload and the easiest thing to do is to go buy enough gpus in order to satisfy our Peak demand but if we do that you know we're going to spend most of our time in this area of where we're way underutilized and we're paying for resources we don't need right and I see there's tons of this going on right now uh and I I think honestly the csps have created some of their own problems and that if you're a startup AWS Microsoft will'll give you basically like a $100,000 of credits and right now I've seen a I've talked to a bunch of AI startup companies that are like AWS gave us $100,000 and so we went and rented that $32 an hour system and we're running that uh but we don't care because it's not our money and now there's a GPU shortage because we're way under utilizing what's out there so we think about like the number of resources we have well I can start thinking about how do I scale my workload up and down to actually use that appropriately and you know each of the solutions out there like ec2 I can do autoscaling kubernetes I can doot horizontal autoscaling there's lots of different ways to sort of solve this challenge in order to use the resources more effectively and try to bring some of that down and there's a paper written by uh a person AWS who was uh instrumental building Lambda I talked through a bunch of different uh pieces here one of the things he talked about was Little's law right and it's the the amount of work then get through a system right so the throughput of the system plus you know the time it takes to go do that uh and equals like your work in progress it equals your q and at some point you think about it as like a a teller at like a a store if there's one teller and I can process one person every two minutes and if more than two people or more than one person come into the store every 2 minutes eventually you're going to start building a Queue at some point you're going to reject people from that queue because the que is going to become so long that people are like no I'm not going into that store there's 80 people deep uh it's not worth my time and we have capacity constraints as well it's like a lot of stores like 80 people not might not even fit if we've got 80 people in there and there's one teller there's problems and we have ways that we can go like increase throughput right we think of scaling I can do things that are quick like I've already got another person employee in the store bring them them in and have them work a second register I have things that are longer like I can go hire more people I can open another store I can get a bigger store right there's a lot of these controls we have and we have very similar controls in it and like how we're going to scale our cloud services one of the differences though is we generally don't want to leave things in queue uh and so we see this graph on the bottom right where we have you know the number of clients in the system versus our utilization and that's really where we start hitting here above 80 90% utilization uh the the time it takes to process this starts to Skyrocket because we're used to having no q and being able to solve everything right away and then we get a queue and suddenly things go up very quickly but we can't process more than what the system's capable of so this is where we ultimately are trying to provide scaling to prevent ourselves from really Crossing that critical point point where the function starts skyrocketing the latency and so like when do I need a scale right well the time to add your capacity times the divid by like the maximum change that we think this is going to happen you know and for this we're looking at like gigabyte seconds of GPU that I'm using uh you know that's the amount of spare capacity I need the point where I need to scale so if I decide like hey my system can take a thousand and the maximum you know it takes me 10 seconds to go change it and I um the maximum change here is going to be 200 and 10 seconds then I I basically need 800 GB you know in order to solve that uh of course we have to have to add in the fact that auto scaling uh we tend to put delays in there because of how Auto scaling tends to be expensive it goes up and down if we just momentarily peaked above our limit we don't want to spend seconds or minutes spinning up more infrastructure so usually we're going to delay after we hit that limit make sure that we're really over the limit and then we're going to scale and so ultimately we see something like this uh this one references containers but we do the same thing with virtual machines uh Etc where we're always over-provisioned because the moment that we hit our threshold we're going to have to start scaling spinning up the next box because we never want to hit a situation where my workload has exceeded my capacity because then we see what we saw uh in that graph before our meantime to deal with those client skyrockets and then ultimately we start rejecting so we think about this like in terms of L inferencing right like you can handle this and build this yourself and you have to think about this if you're scaling up and down I don't want to be over provisioned the whole time but even if you're using a token based solution or somebody else is hosting this for you uh you're still going to have these s like they're going to have these challenges and they're going to pass those challenges on to you right so if I think about you know a vendor that say may provide me uh serverless functions or serverless AI inferencing in a container uh you know solutions that are out there today take something about 90 seconds to start up uh so I have to think about that in terms of delay because of that huge performance penalty in the front end usually you keep the thing on for like at least 15 minutes because I don't necessarily want to pay that penalty again well how does this actually impact the cost that I spend right and I think it's important that people think about your costs like what they are charging you for your infrastructure is inclusive of the cost that they have and if they are wildly over-provisioned the whole time then they're going to have to they're going to have more overhead and they're going to have to charge you more and most of these companies right out there are not actually using their own data centers they're renting that from AWS or Microsoft or somebody else so they're again passing that infrastructure along to you unfortunately doing it yourself doesn't necessarily save you much uh cuz usually the overhead of owning your own infrastructure and handling all these pieces tends to be higher for you and a big piece of that is because of multi- tendency uh there's a concept of peak to or average to Peak or mean to Peak and that if I have one workload uh it's going to tend to have its own you know there'll be the high the high point of the day the low point of the day and we'll often see some consistent Trends around that right so we see this with uh Nvidia has this problem with uh they have a bunch of gpus that they use for gaming uh and about 5:00 P PM East Coast time these things start going up in usage at about 3:00 am West Coast time they're they're down uh but from 3:00 a.m. West Coast until 5:00 p.m. the next day they're doing basically nothing because everyone went to sleep and then went to work uh but it's in the evenings where they're peing right versus we'd expect a lot of workloads that are uh driven by a company we're going to see the opposite right we're going to see a lot of utilization during the day and because these Peaks and Val IES right are relatively consistent it makes it very difficult for us to do appropriate capacity uh and it means we have to usually scale faster but if I have multiple workloads and in this instance we see a multi tendency of two versus a multi- tendency of four our mean becomes much closer to our Peak because ultimately these different workloads overlap in different ways the speed at which it needs a scale spreads out and then we're able to offer this as a vendor more affordable than you would be able to even do yourself uh and so that's I think with this it's still useful to go have the vendor do it because of this and we won't have this internally unless you're a large company with lots of internal customers um so when we really look at the like how do we reduce our costs here right we want to look at V paying attention to how our vendors are spending money and how our vendor cost structure is and so if the vendors are using Technologies like containers which really limit them and the number of containers they can have attached to GPU how much they can divide up that GPU how fast things can deploy and how long they have to keep it on right like even if they're only charging at me for when they use it they have all of those costs and they're going to need to bake those costs in to what they charge me so if you can find vendors that can look at that and a new an Innovative way and reduce their cost then you can come up with more innovative solutions so what is the net of all of this right what's the cheapest inference and this is probably not the answer you thought I was going to say but it's the one you don't do and this is one of the really again critical things as I talked about prompt engineering before like don't jump immediately to uh training your own model until you really exhausted with prompt engineering it's the same way is don't think that like yes this model is now consistently solving my problem every single time we've hit Victory because like if most companies your workload probably is doing almost The Identical thing over and over over and over why would you have the most expensive resource do that uh there's a really cool paper I liked called The Frugal GPT paper uh and basically they did uh several tactics um one of them was vect Vector similarity search we already use this a lot from Bings right so like the augmented uh questioning where I'm going to say let me do a vector similarity search find the right piece of data at all my data and send that along and then AI 10s to turn a better answer but you can also cach the responses and the question that come back from when you asked AI or when you asked you did the inferencing and you can store those and do the same thing so I would say hey somebody asked another question let me before I send this out to open AI or my internal system or whatever that is let me look internally like have I've been asked this basically the exact same question before yeah let's return the cash response there's no need to generate a new one uh likewise there's a concept of cascading where there's many very there's much cheaper models uh that are much smaller than something like a Chad GPT 3.5 uh that often times will return a uh perfectly fine answer and so by using like the confidence metric that comes back out of this I can say like let's go hit the cheap model first see if we get a high confidence level and if we do let's return that answer and only if we have low confidence levels then let's move up to the more expensive models these kind of things can avoid a lot of the inferencing so you look at how I could build my system and I'm like okay the cheapest I could possibly get to is what op AI is going to charge me at like 35 cents or something per token well that's not really true like the cheapest that you can get is to make sure that not all of your requests actually go to the really expensive big model and a lot of their quests never get there so how do I put this all together well ideally I want a solution where I have a you know I either in myself or using a vendor which is going to drive down those overhead head cost that we talked about and is going to give me the ability to have tooling that's going to make it really easy for me to support multiple models and to do Vector similarity searching by having a vsss embedded enabled database and you see right now like it's really dependent on you as the end user to go wire all this stuff up yourself right and you I was at an AI conference a couple months ago uh in the in the city and like the there was I think six vendors there that had ve like vector enabled databases I'm like cool like that's a problem we have today but like it's weird to me that like hey I is open AI or any any scale any of the people are going to host something for you is going to say you totally should do Vector search you should be using that I'm not going to help you in that in any way right and you need to find a new vendor and wire all this stuff up yourself so finding a solution well you can absolutely do that but find a solution where you got this all stored together can make this much easier so that brings me back to uh firon the company I'm with uh we released a AI service back in September right we are a serverless functions company we're in the business of time slicing CPUs and we found a way to make it really really really efficient and fast to time slice CPUs to provide serverless capability and we realize that we can Tim slice gpus in a very similar way and provide a more effective better utilization of the very expensive resources and then do that in ways where we don't waste tons of time starting up and loading like an 11 gig model into uh a container and into the GPU memory like let's cach that in the memory let's find a way to securely sandbox this with web assembly and then throw the stuff in a queue and very quickly within milliseconds start start handling your inferencing uh and so we think of ourselves as like the next wave of cloud compute right we're not a replacement for containers same way as containers w replacement for virtual machines but a lot of what we're doing is trying to minimize what you as a developer need to think about you shouldn't have to sit there and think about hey I'm going to go build uh I need to think about the infrastructure I need to think about uh where this is all going to land I need to think about scale you we really allow you to just focus on write your application accept a request in get a response out you need a KV we've got that for you you need a database we have that for you Vector search enabled you need AI inferencing we've got that for you you just say inference and it's there and so this is really where we get at the end right is like we think about how I can optimally use these resources we make sing like similar single requests and then I can much better Target uh the workload that I have because I don't have to over provision so there's a few things for call to action here and then I'm going to show something really quickly demo wise but we've got um our quick start if you guys want to get learning with it again another link to uh our AI the white paper that we've got and then IDC released a report on us uh recently that goes through both our serverless functions and our AI inferencing so before we are done I want to show this is llm explorer. furman. apppp you can all browse to this if you want uh and this is basically just a simple UI on top so this is a seress function that is built on top of our AI inferencing uh and so you can create a bunch of different applications and this is a lot for making it easier to do that prompt engineering and understand how to do this better so here uh I basically said hey this is the system prompt I want to do sentiment analysis I've given it a few shot examples of what's going on and then we control the tokens the temperature repeat penalty Etc make that really easy and I can go here and say like was this talk great oh is it not showing that and how do I system settings displays arrange where's Mirror Mirror extend to Daddy Mac I don't think that's what I want all right you know what I'm just drag it over here see it now now I can't see it but so I can't see anything on this thing but now you go there you're welcome to play around with it we've got ones for sentiment analysis asking questions you generally helping uh you know people if you want to have a like let me have a a helpful chat bot Etc uh identity recognition like identifying within a a paragraph what are the actors in that so there's a bunch of examples this is just one for sentiment analy but again let you play around with this really really easy and top of code llama llama 2 and uh see what's going on so that's my chat I happen to answer any questions anyone [Applause] ask