DevReal: Optimizing LLMs for Cost-Efficient Deployment with vLLM - Michael Goin
Recording: DevReal: Optimizing LLMs for Cost-Efficient Deployment with vLLM - Michael Goin
okay hi everyone I'm Michael Goen I'm an engineer Le at neural magic uh recently acquired by red hat but it's not important we're still doing the same open source work I'm a maintainer on the VM project and uh thanks to Charles you'll understand a bit more uh of this presentation as I blast you through a lot of the inference and model optimizations that uh you should be considering once you start deploying out Lambs at scale so first just about neurom magic uh we believe the future of AI is open uh we're truly on a mission to bring more open source llms and the power of VM to to more people uh through open source um that's the quick Spiel if you don't know what VM is it's basically the standard in open source model serving it's a fast easy to use open source inference server for hosting language models of various types um the reason why people love it um is that it supports all the key models uh we actually work closely with the hugging face team and there's now going to be a Transformers backend where you can run pretty much any model from Transformers even when we don't have specific optimizations for it um it it hosts a large variety of inference optimizations because it's sort of a platform and a base Fork uh for a a lot of research that gets folded into the project over time and of course it gets great performance on Nvidia gpus uh but because of its uh you know flexible architecture in Python and pytorch while still being very performant uh it can easily support other Hardware backends like AMD gpus Google tpus AWS INF frenia Intel gpus and gouty and CPUs um and it and like I mentioned it has a lot of infuence features it doesn't just support text large language models it has multimodal support embeddings reward modeling classification scoring reranking um there's many quantization uh Styles which I I'll cover a bit here uh and inference optimizations that I'll also cover uh and of course many features that are useful for building applications uh like you get from the you know close Source apis like tool and function calling and structured outputs uh thanks to outlines and uh of course parallelism tensor parallelism pipeline parallelism uh and more to come so that you can run models across multiple gpus within nodes or in between nodes and BLM as I mentioned has some pretty easy to use interfaces you use it by pip installing it or using the docker images um it gives you a python offline batching interface uh and an open AI compatible server that can get you started and and move moved away from the close source apis and start you know experimenting with hosting your own large language models so some of the optimizations we'll cover today are are model optimizations which are where we're going to cover you know how to make the model forward pass fast you know each one of those tokens or pre-fills we uh we generate like Charles's covered and some maybe the more interesting things uh and where a lot of research is uh are systems optimization so making everything around the model forward pass faster or more efficient so such that with each model forward pass you're getting more work done and you're keeping your gpus more towards that 100% utilization that you're chasing so first model optimizations first I'll cover numerics uh so as you probably kind of know llms are a series of Matrix multiplications with parameters each parameter ends up being a value represented by some number of bits uh usually you know around 16 bits is what we tend to use for be uh brain float 16 or or floating Point 16 and those bits are generally uh you know used for for two things you know representing the sort of dynamic range you know what are the maximum or minimum representable values in magnitude and precision how close can we walk around zero to talk about 0.1 or 0.1.1 and those things and why we talk about quantization you know quantization is really a thing with the goal of of re of reducing the amount of bits we need so we can get more efficiency take up less space in memory or potentially use more efficient compute units um and you can apply this on the weights and activations mostly you know for this consideration we're just going to consider the layers where we have all the parameters uh generally we talk about them as the linear layers in the models so first there's weight only quantization which Charles already covered where we're just going to quantize the weights uh and leave the activations at their original Precision the main purpose of this is reducing the memory requirements so you can fit your model onto the gpus that you have available and it also you know allows you to reduce some data movement at the expense for a little bit of extra compute uh that you use to upconvert the weights for each forward pass as Charles also rightfully mentioned you can overlap these things uh due to the many uh compute units available in gpus and you know the Cuda cores we have to do some lightweight math are uh different than the tensor cores that we use to do the really big Matrix multiplications and you know this is probably the quantization that you know if you're in the open source Community gptq awq bits and btes ggf um and these are the things that make the most sense when you're starting on consumer gpus or trying to get llm fitting on your laptop and you're just a single user doing batch size one decode inference now another part of quantization is activation quantization uh so this is where we're quantizing both the weights and activations um and this is really important because weight only quantization is not really sufficient once you get to a certain amount of load which you are when you're serving llms and trying to make money um so by quantitizing the weights and activations we can finally use those low Precision tensor cores that are on these data sheets uh every time a new GPU comes out and it says fp8 uh you can't use those unless you quantize the weights and the activations so again to kind of cover like why would you want to quantize the weights and the activations so uh I was really happy when Charles mentioned arithmatic intensity because here I have a a roof line plot uh don't let it scare you too much uh but really it's uh just about talking talking about on on the left side of the roof line plot we have memory bandwidth limited regime where we can't um you know where where we're just uh don't have enough arithmatic intensity we don't have enough compute um so we're limited uh by the bandwidth we have from the GPU uh in in terms of saturating all of the flops that that GPU has um and and you're there when you're in you know low batch decode and this is where memory compression really wins um however once you get to the right side with with a bit more computer intensity where you have large batches of requests that you're processing in parallel or doing really long prefills for long context lengths this is where you get into the compute regime and we can actually use all the flops that are on the GPU um and this is where you know activation quantization is useful and so I have a bit of a simpler graph still complex uh on the right side that has you know uh The Gray Line as an unquantized model the yellow line as a 4 bit weight quantized model and the blue line as an 8bit weight and activation quantized model and the y-axis is time per output token so basically how fast are you getting your streaming tokens and the x-axis is the queries per second or how many requests we're putting the server under load and basically you can see the weight only quantization is fastest actually at one query per second so it it can be faster when you're in a memory bound situation however as you start scaling your queries out it's clear that having the weight and activation quantization gets you better results and the other thing here is we have a time to First token of 5 Seconds as our latency SLA and this is why those other lines stop before the activation quantization line keeps going because we can serve that latency SLA under more load making more money so another type of model optimization is 24 structured sparcity um this is nvidia's uh biggest marketing tool it doubles all of their flops numbers um it's not that it's not as easy to get as quantization but we work on it as as well at neurom Magic and this just offers you a straight doubling of the flops you have available and about a 40% memory savings if two out of every four parameters you have in your weights are zero um basically combining this with activation quantization is the way that you get to the peak flops that are available on your h100s um on the uh right here we have another chart where the black line is a dense unquantized llama then the yellow is an fp8 activation quantized llama and the green is FPA quantization with two4 sparcity um and here you know we just see it's it's a benefit for your your you know time per output token all the way uh because we have better compute uh available and we have lower memory bandwidth requirements um one of the ways we make this really easy for you is we make optimized models all the time in open source so uh if you want to play around with these different formats all these different formats um go to our hugging face repository um where we'll have lots of models and evaluations so you can make informed decisions about the accuracy of these different quantization methods um and also get them to run across VAR various different Hardware platforms by deploying them on VM but you don't just need to get it from us we also help make the llm compressor project uh on on under the VM project um and this is a unified framework for applying quantization sparsity distillation algorithms to your models that you have um just within the hugging face ecosystem so give it a give it a try so next I'll cover some system optimizations for LM inference uh AKA all the things around the model so um one of the key things uh you want to think about here is chunked prefill this is essentially trying to deal with the problem that your prefills and your decodes are so fundamentally different you know if you look at the chart on the left we have basically trying to show the amount of tokens you can get out of a model uh when you're processing prefills uh at different increasing batch sizes you can see the pre-fill graph doesn't really change the amount of tokens you can get out of the model uh so you're already pretty saturated however with decodes you get astronomically more tokens out of the model as you increase the batch size uh of of of decode requests that you have however it's still not anywhere near close to how many tokens per second you're getting from a single prefill so chunk prefill breaks down a single prefill operation into smaller chunks so you can start to schedule those chunks of your prefill alongside your decode batches so you can just get more compute utilization out of the GPU AK just more tokens per second out of the GPU now this is really cool for utilization and things like that but this also has a benefit on actual performance and uh you know latency for the service that you're trying to to to to provide you know there's this uh you know kind of famous paper at this point uh that covers this idea of instead of throughput good put you know we only want throughput when it meets you know our latency slas and uh when it doesn't feel bad for our users so in the case of showing scheduling new prefills for new requests without chunk pre-fill when we need to schedule a new pre-fill the current decodes that we have undergo a delay while we're doing that prefill and uh and as a result the user is going to see you know weird uneven uh uh pauses between the tokens that they're getting and their Pro and their uh whole request is just going to take more end to- end time uh however like with chunk prefill you know we make that pre-fill take a little bit longer with a little bit of inefficiency by splitting it into chunks but we get to keep that consistent time between decodes and uh overall get all the requests done faster so it's really an important optimization uh that's you know enabled by default in BLM another thing I want to cover is automatic prefix caching um so first I'll talk about the reasons why this is useful so you're probably familiar with chat gbt at this point uh many of these models uh and and open source models in production as well have system prompts that help guide them and Prime them for being good assistance for you and each time you send a request of these models um they insert a very large uh system prompt before your request um and and these are identical between each of the the the requests on the model another example is multi-t conversations uh which obviously you know uh you run into when you're having a long- winded chat uh about something with chat GPT um you know once you've once the LM has responded to you and you want to ask another question or ask for further detail on that um you know trivially you're sending back all the data you've already talked about and now your new data um since the LM has already both you know pre-filled your response and decoded its response to you um it should just be able to uh you know take that uh previous conversation from cache and and not need to recompute it so this is really important as you start getting to uh you know to scale and and not wasting pre-fills as we know they're they can be pretty expensive um so automatic prefix caching is pretty uh uh natural inside of VM because of the fundamental algorithm of uh of of how it implements attention with page detention where we don't need contiguous key uh key and value stores for keeping you know the cache of previous conversations we can treat the KV cache uh like operating system virtual memory and uh you know uh uh which which is really useful for allowing us to pre-allocate all of our memory ahead of time and dynamically you know allocate more blocks for for more requests coming in uh and to add prefix caching all you need to do is add a hash to each uh KV block and now as you have your your your uh prompts coming in you know you run you check your hash on that and if you have something that matches and in all of your KV blocks that you have you just pull that in and you don't need to prefill all those tokens so another optimization is speculative decoding this one is particularly useful for reducing the latency of Generation Um so taking advantage of another time where you know during generation we're generally memory bound um and and also there's some tokens that are easier to generate than others you know think about cases where you're doing uh coding or uh you're doing summarization uh or uh you know really just repetitive tasks um we can generally use use small language models that are aligned with the large language models um in order to generate many more easy tokens and speculate on what multiple token outputs could be and simply just run those through the larger larger language model in order to validate if those tokens were right and then we just take the tokens that were right discard the ones that were extra and it's not that much of a penalty because you know we have the compute available the great thing about this is that it's lossless because we always validate the tokens that we generate so here I'll describe how you speculate with a draft model as you kind of saw before um you know usually you take a model within an existing model family so you know if you're running llama 70b grab the Llama 1B or 3B um these are good uh model pairs just in terms of you know instruct models but also if you have you know custom fine tune data or custom data sets you know if generally if you fine-tune a large model and fine-tune a small model you can get pretty good alignment um the challenge is yeah if you don't have a draft model if you're doing a custom model um how do you find that draft model so it does take some work in order to uh you know find a model or create a model if you don't have it available and you know I can talk about the performance benefits a bit here uh you can see you know it gives uh pretty decent speedups uh at low queries per second low requests where we have that extra compute but as you scale up the QPS uh to to eight at the end there you can see it's not really giving any benefit and actually if we expand this chart far out to to 16 or 32 um speculative decoding can actually cause performance issues if you're still running it when you have a large batch because it is adding more compute so it's important that you have sort of dynamic methods for changing how many Target tokens you're going to uh produce for a given request based on your current load another meth that doesn't actually require a model at all is matching NRS in the prompt or otherwise known as prompt lookup decoding um so this is particularly useful in methods like summarization or coding um where uh you know effectively you expect the tokens that you're going to generate exist somewhere in your prompt and just based on trying to match engrs with what's been produced so far you can you know uh uh produce all of these candidate speculative tokens as engrs in your prompt or uh from engrs in your prompt and uh just run them through the model just for uh shits and giggles and if they happen to work then that's a speed up this is something that you can do uh pretty easily uh uh in a dynamic way and actually in VM we hope to enable this by default uh if we can be really efficient with it so um but currently it's opt in so try it out another optimization is disaggregated pre-fill and decode uh so uh you know we talked about chunk prefill and trying to uh make you know prefills hurt your decodes uh less but um but they still hurt your decodes um and and frankly they make your prefill slower as well um so when we're talking about high concurrency High load systems we need to start to consider actually physic separating our prefill versus our decod jobs um and so we don't run into these wasted times if we have separate Runners for prefills and decodes because they're just not interrupting each other so And to clarify this means you know you run a pre you have your requests coming in you run those pre-fills on one GPU and then once that gets finished pre-filling then you transfer those jobs to be scheduled on a separate set of gpus that you have for decoding and here's an example of showing that you know when you collocate when you use one GPU for pre-fill and decode um you uh when you need to hit certain uh uh you know goodp puts some latency targets um you're really limiting yourself in terms of the requests per second that you can process per your GPU obviously you can scale this up you can do three gpus of the collocated but you're going to be stuck at that 1.6 request per second that you can serve from that GPU within your latency SLA however if you disagre and you know on paper it looks like it looks less efficient you're using three gpus two for pre-filling and one for decode however you can serve more requests per GPU because you're specializing these gpus and uh in order to better serve your latency metrics um so this is a really really important optimization that um you know we've started in BLM we have basic support for but this is you know what's used for the really big large production Runners uh like open AI or anthropic or meta uh to say the least um so that's most uh of what I'll talk to you about here uh I'm I'm happy to make you eat your greens here uh but check out the docs on VM for some of the other methods that we and to learn how to actually use them in VM what neural Magic on per quantization inference on the torch compile integration try and R um just overing lowering system overhead in general um and of course bringing all of these you know production metrics production features that we find when we bring Enterprise customers to scale the LM um and hopefully bringing that to all of you and open source and you know Rising tide lip Sal books um um also we're a top contributor to BLM I uh am am the top contributor at neur Magic um and uh and and yeah so we work on all things in forence performance engineering and uh if you're interested in that at all uh we're hiring uh please if you write Cuda uh please join me that that's all really M will bring you m um yes so you can go back to the segregated Prill slot um so you need relatively fast interconnect between the refilling tpus and the decoding tpus cuz you have to migrate if like map the detach from theil onto the decode yeah yeah and roughly like how many bites per token like it's that's going to steale with model size right well yes yeah um it's bites Prok it uh I I mean it's on the order of like a few megabytes if if I'm if I'm thinking right per temp yeah yeah for like for the 8 billion and 70 billion parameter models that people are yes yeah yeah but the key thing is that uh you know I I simplified it here we're really considering single gpus but um the key thing is that uh for particularly for really long prefills it takes a lot more time to do that prefill than to move the KB cache over uh you know uh decent interconnects uh where decent is defined as incin or rocky yeah yeah yeah yeah not NV or yeah I mean not okay not in uh I can I I can show okay yeah we take it yeah yeah we can take it offline but uh the key thing is that like when you specialize sort of pre-fill of decode workers you can actually set up different types of model worker configurations um where you know for instance like you really can scale up your tensor parallelism uh really really hard when you know you have a lot of compute for running uh you know really large prefills uh whereas you can't necessarily do temper parallelism uh where this is basically where your splitting up every layer and requiring secreation on every layer uh but this is the way that you scale to get you know better compute um uh on on small decode batches uh just because if you're running a small decode batch then you're going to run into the commun station overheads and that's really going to hurt you ra if you're running a p worker uh you can really optimiz for that and get your generally get your time to First token down um if you can specialize in these ways uh it's definitely a thing that can bite you in the butt though so uh it has to be built at scale still um it seems that the Tosh compile is is a bottleneck and you we keep trying to optimize it information Mar the question to is for you guys considering I'm using ml to essentially uh fact things like bu and the reason I said this is because there's a paper resarch coming find F improve the performance significantly for you there yeah I mean uh I I think it's really interesting uh you know we we certainly hit higher enough uh good people that that can write TP Colonels so uh and and we actually use torch compile not just for its automatic compilation but for injecting sort of cust Fusion passes um the great thing about torch compile is that it now gives us a graph we can compile and we can go back to sort of traditional compiler basic versus uh you know needing to you know replace just one operation at a time we can we can actually have a graph and and perform passes and and insert custom kernels but we're we're limited on the number of custom kernels we can substitute in because you need to identify them and and have you know complete the the people resources to make those um so yeah you know but unfortunately um because we write so many kernels for everything else like for our linear layers for our attention for our RMS Norms uh for our for Collective communication for tensor parismatch.com so um currently yeah it requires some of this sort of manual Fusion work uh if you want to both have you know optimal kernels and also optimize for small uh yeah do do larger larger scope optimizations like that that you could do T and um so I CU a redhead uh fire your best so what what isect like secur like data poisoning or what the you Happ me making and mer uh yeah um I'll be honest I I am not going to wor about that I I'm going to keep working on BLM and most of us are going to keep working on BLM so I mean you know Red Hat already uses VM for a lot of their products and and their offerings and they just want to make that they want to continue to make that project better um so that's compliment to yeah I mean we got a we got acquired so we can keep work yall let just that's a that's a good yeah so we're going to open source more criminals that's the goal at thank you question please