vLLM Project Update

Bay.Area.AI: vLLM Project Update, Zhuohan Li, Woosuk Kwon

Recording: Bay.Area.AI: vLLM Project Update, Zhuohan Li, Woosuk Kwon

uh we'll give you an update about V so we'll start with dra okay yeah um hello everyone uh I'm Johan and I'm a pH student at UC berky and I work on VM full-time and today I'm happy to share about some recent updates on the V open source project uh I would like to know here that the updates I'm sharing today are a joint effort from from many people in the b in Berkeley and many updates are actually from the larger open source Community outside of Berkeley and many of whom are also here at the meet up today yeah and yeah the topic so first I will do a brief recap and do some bragging about vrm and then yeah and then wuk will come up to do some like uh do to cap to do say something about what we did in the past before the last meet up and this Meetup and also then and after that Simon and Li will come up to talk about our development road map okay so yeah let's get started so arguably uh we are in the era of a a are show have shown like remarkable capabilities in a variety of tasks from chatting to content creation to programming and some of these applications such as like chat gbt and GitHub co-pilot are already changing our lives and not surprisingly a key part of these applications is systems that serve the LM us using gpus and yeah however we noticed that serving LM is surprisingly slow and expensive and for example for systems before vrm a single GPU manage it can only manage to serve less than one request per second for moderate size models and inputs uh this means that uh you will not you need a ton of gpus if you actually want to build a production scale services using LMS and indeed the high cost of serving LMS have been a major challenge for many in the field and even like hyperscalers like Microsoft struggle to struggle for securing enough gpus for theirm power services and all these challenges motivates the vrm project and we started the vrm project to build the fastest and easiest to use open source a INF engine and back then as a time we started the project so all open source Solutions were slow or very difficult to use so we decided to make one by ourselves and yeah as Anthony covered just now so while we are building up VM from scratch we realized that inefficient memory management was a key performance bottom neck of serving and to address this issue we came up with page attention a new attention algorithm that operates on blocks of KV cache and Page attention enabled highly flexible memory management and we could manage the KV cach like the virtual memory in operating systems and the achieving efficient and elegant space sharing between multiple requests and we've observed huge performance improvements and open- Source VM to share the results and fortunately we get a lot of tractions from the beginning and VM has continued to grow with the help of community and today vrm is the most popular open source project in the serving area and yeah vrm has a very nice user and apis at first it has the LM LM class which is a python interface for offline batch inference and we can simply create an a object and call generate to efficiently run inference on a batch of requests and you can also start vrm as a full function open API compatible armm server with a single command and an open AI client can query the server by simply replacing the URL and VM has received uh contributions from more than 190 contributors from all over the world and thanks to uh all of the contributors and yeah and from its release uh VM has been adopted by many open source projects including many Sky projects and also many companies and uh most noticeably I think VM has become the default serving engine of Microsoft H AI model catalog and also has been featured at AMD Mi 300X launching event to show its inference performance uh yeah by the way like not an investment advice but after this I started to buy am the stocks yeah and I was pretty happy in the past few weeks and and not that happy this week yeah yeah and yeah and yeah yeah that's all for my bragging and next I will have W talk about actual [Applause] out okay uh hi everyone I'm usok I'm a co-maintainer of Realm with I'm maintaining it with Johan Simon and Lily and I'm going to briefly briefly talk about the major updates in the project since the last meet up we made pretty significant progress on the project we've continuously added new models we optimized latency and serut of VM uh at with different optimizations optimizations at different levels and we added new features including AMD GPU backend and and multi serving uh finally we implemented uh cicd pipeline running on gpus good thanks to the help of the community VM currently supports more than 20 different model architectures covering most of the popular ATMs in the world most noticeably we have been collaborating with open source AI companies like Nal Ai and deeps AI they're officially contributing their models to VM VM started to support uh the Mi model from day one and now also supports deepsee model with an optimized fusede colel okay on the performance side we have implemented a series of optimizations including a new version of page attention colel uh Cuda graph distributed runtime optimization and faster allu Kels next I will briefly go over these items these these optimization techniques okay uh our previous implementation of page attention explo the GPU pism by signing one asend for a sequence however this sometimes led to a significant load imbalance uh just like the lo lo imbalance between SMS just like the figure on the left hand side in the new connal page attention V2 we partition each sequence into smaller pieces and distribute them uh to different different asms uh this is similar to flash decoding uh if you're familiar with it this optimization leads to like up to 7x conal speed up especially for like long sequences and like 15% and TN latency reduction uh for a particular case in on h100 uh and with more and more optimizations added for gpus the time VM spent on CPU becomes more and more prominent and becomes the the Major Performance bottleneck in our profiling we see that uh in in our profile we said that the python or pych overhead takes up to 50% of overall latency a point that gpus execution speed is faster than the speed uh CPU launching the next conel okay to address this we introduce Cuda graph into VM Cuda graph records the the Cuda conel launched during during the model execution and constructs a graph that can replay the conel replay the recorded conel and this way fent bypasses Python and py py runtime in model execution with Kur graph we achieved up to 90 up to 41% latency reduction for small models like Microsoft Pi 2 okay another important optimization was made on our distributed onetime our previous system architecture involved a lot of serialization overhead in sending and receiving the inputs and output uh between the scheduler and the workers with our new design we completely completely removed the calization overhead uh by converting the python objects into GPU tensors and using Niel to broadcast them this turned out to be very critical for high throughputs uh we observed up to 70% serp increase for Lama 1 70 billion model on 88100 gpus and uh our most recent optimization is the introduction of custom o use Connel basically ni uses a ring algorithm for all use which partitions the tensor into smaller pieces and send them through multiple steps uh while it performs pretty well for large tensors the algorithm is not suitable for uh small tensors uh which is the case for the inputs in the decoding phase for small tensors like all to all style like one shot allu performs better than the wiing allu in this algorithm each GPU directly patches the tensor in other gpus uh all in parallel and in a in a single step uh in our Benchmark results the optimization leads to up to 7 up to 2.6x conal level speed up and 20% N to1 speed up for small batch sizes okay putting these together we've continuously optimized the throughput and latency of VM over the several months in the particular set of a Lama 70 billion and 4 a100 gpus Willams reput has increased by roughly 50% and its latency has decreased by roughly 40% and we are also actively optimizing the mixol Moe model for example the recent uh fuse conel improved the performance quite a lot we working on like further optimizations so please stay tuned okay on the on the feature side we've added the support for AMD gpus multi roll serving quation and preface caching let me go over very briefly okay starting from uh December 2023 we have it we have added amdgpu support for all models in BLM the support was contributed by the startup embedded llm and also AMD we're actively collaborating with both teams to optimize vm's performance on AMD gpus especially uh Mi 300X and in this event we have hi here could you raise your hand okay uh did Disc okay okay yeah actually yeah he's there but yeah you can catch up him if you're interested in this MD gpus like okay uh and secondly Laura is an efficient way to find tune at lamps for different tasks in v l we integrated the techniques in ASA and pun paper uh so you can actually load multiple Lowa adapters on the fly during the inference and efficiently batch the batch together the recast to different R adapters uh into the same input batch uh this work is contributed by uh any scale with the support from es team and yeah quantization is the feature that has the biggest Community ask actually we for this uh for the several months we have refactored VM to support different Quant methods for all models and through many Community contributions V now supports most popular quation methods including gpdq awq squeeze llm and fp8 KV cache okay lastly we added like uh basic support for prefix caching which allows you to cach Common prefixes to different recast with that you new recast can directly use the cash prefix without recomputation for example uh the example on the screen like the recast can use the the cast system prompt you know the the Yellow Part uh and the model only needs to compute the short user prompt at at run time and then you can generate can you can immediately start gener uh start to generate the outputs and uh for now this we we have a basic preface catching support and we are working on we're working on automatic preface caching which basically uh VM automatically uh finds opportunity to uh reuse the reuse the cach of the previous recast and and and make it transparent to the to the user end points due to the time constraint there are like many other improvements that I can I can't go through one by one uh we just thank for thank the community for these improvements and one last thing I'd like to highlight is our new cicd pipeline now we have a nice cicd running on gpus to test and Benchmark every PR we believe this will increase the reliability of VM okay and I will hand over to Simon so now I'll talk a little bit about the development role map we're not going to go into detail due to time constraint but feel free to talk to any of us about any specific point of this so let's start with three months ago in our first ever meet up also in the city we this is exactly the same slide we put up about what we're going to work out about a quarter ago and then uh we're happy to say we work on we're able to basically check a majority of the boxes and all the big boxes as well as add a lot more like distribut rown optimization prefix caching to what we're originally planning to do now this is three months ago to today let me give you an overview of the team's plan along with our collaborators for what we're going to focus on the next few months and we're going to break it down into again basic improvements to the system as well as new features so for improvements our Focus are for example Performance is definitely our top priority and here in particular we're going to be focusing on performance optimization for h100 GPU which previously due to Resource access were not able to access and tune V on those Hardware as there are more and more h100 believe there are going to be more and more important for serving as well and as mixo and deep seek um Moe come up and we're going to be able to perform more optimization there we're also thinking about scheduling optimization in particular how we're handling prefill and decode so there are a method like chunk prefill as well as prefill disaggregation both of these have two or even more papers backing behind them uh improving the stuple and lency of the overall system when you have a lot of requests running at the same time so we'll look into this with our collaborators and see how they can transparently improve the performance we're also going to look at how do we support uh more method of quantization in particular activation quantization to improve the performance as well as allow you to squeeze more model in particular improve the throughputs lastly modularity is a lot of uh sort of common ask from both the research Community as well as uh industry Community focusing on making scheduler and memory manager which is a core of V more extensible so you can plug in U mechanisms that will better sual workload and last but not least on the Improvement side as Anthony talked about we are eager to uh bring torch compile to VM so that to sort of take advantage of the optimization made in the broad pyos ecosystem now on two features um on sort of new support one of we believe one of V strengths is ability to run L on a lot of Hardware in particular we continue to support as continue to support MD we are adding support from ad bestest INF frenia and I believe there's a big team here from ad best focusing on this we are working with Google TPU as well as two different team from Intel Intel GDI Habana tensor accelerator as well as team working on Intel both the Intel data center GPU as well as CPUs so stay tuned and S the support there and across the stack we seeing competitive performance on different workloads uh compared to Nvidia gpus now um we're also uh thinking about adding support for multimodal uh uh this includes models like lava and as you can see from gb4 Vision uh it's a very useful model and we believe Mo adding support for image embeddings will help your production workflow as well we're going to support specular decoding this is a method that uh sort of improves the latency uh without harming the accuracy for your inference work workload and in particular we're building a framework in collaboration with anale ands uh to support a lot of different kinds of speculative decoding algorithms so lady later will be talk a little bit more as well as um usok mentioned we're adding automatic preface caching so you don't need to tell us here is my system prompt here's how long it is rather V will be automatically figure out uh the ReUse of for example both the system prompt your Longs for the few shots prompt documents as well as rag documents that appears together and such that will improve better sharing as well as improve application latency finally this is a project I'm personally very passionate about is adding structured output support in collaboration with the best sort of Library out there like outlines and asan where you are able to constraint and enforce the model to Output like proper Json according to Json schema regular expression as well as any context free uh s any grammar for example for python code or SQL such that you'll always get legitimate output all right to just dive a little bit deeper with specular decoding L will'll talk a little bit more um hi I'm Lily so um so for speculative decoding I will say since a very early stage of vrm we see lots of PRS about spec going but we haven't got this feature yet so I will say this is one of the um most important features at least like uh from my site and currently we have um lots of contribution from the community like um it's a joint effort from any skill like AWS and some like independent contributors and the goal here is to have a general design to support different flavors of speculative decoding uh yes so speculative decoding is a way to reduce the latency so you will have draft like we will support both like using draft models or without draft models like Medusa ego Etc um and we do have a plan so we we plan to have uh use n PRS um to make this feature in VM So currently we already uh merg two of them um so I will say we still have like um seven PRS including both um change the change the design and also do some optimization but we do need help um from the contributors right like we need the mqa kernels for speculative coding to acheve give a theoretical speed up and and we need efficient sampling kernels because back decoding do more sampling and we also need some like a verification kernel like how do you do like tree verification so if you are interested let me know and let's let's make it work yeah um and lastly uh we really um shout out like call for contributors reviewers um and adopters so we are currently growing our reviewers and committers um base so please please help us review the pr so there are 100 PRS over there so just pick anyone like you are interested and start review it okay um and and our road map uh comes from you the vrm community so don't hesitate to reach out if you are uh very eager for any features talk to us tell us your uh your needs your requirements your use case your workloads and if we are team think it's reasonable um and let's sit together and figure out how to support it uh and also we we we have released our road map um in this issue please take a look um and let us know which one like you like um and lastly VM uh is an open source project maintained by UC Berkeley helped by the organization adopting vrm so we are not VC funded and we really need help so in any ways right you can write code or reveal the code you can also give us the like the machine right because like people will complain the performance on h100 is not good then give us h100 right like give us a machine so we like we can we can fix it together right so any Cloud credits any hardware um or even cash like whatever okay so we need help we are we are students from Berkeley um yeah so talk to us if you can help yeah I think that's that's it um yes this is our G up and anything thank you yeah so um so we'll now do a global Q&A in which case uh if you have question that you think will benefit others as well uh in particular with clarification as well as sort of uh basically asking about things discussed on the slides uh please ask them now if you have question about sort of more of settings related to your own organizations and since you want to discuss in depth in detail find us right after this cool well sir yes C thank for like raing I'm really F but as a want to like truction we also care about how they go was Avil so I'm curious like every time you release a new B what kind of pass you yeah please repeat the question for record cool all right so the question is about sort of the stability and what how to can we ensure V stays as a uh sort of production grade software so first of all uh I guess there's two parts here one is about sort of the um backward and forward compatibility where here we ensure public apis are sort of Backward Compatible but also the main way you should interact with VM is through the openi S the protocol uh compatible server which will make sure it is compatible with your existing workflow on the production testing side we run I would call pretty extensive testing but definitely not there yet uh due to Resource constraint in particular so make sure we run the test uh both for the CI we make sure we measure the correctness of the models compare against hen pH implementation we scale between sort of one GPU as well to eight to T to test like T parallel we test on different kinds of Nvidia gpus and uh sort of our collaborators from AMD help us test it on AMD but I would say we have enough confidence uh that we have for example like the both the performance are there but we need more help more resource to test down wider set of ranges and that's indeed one of our Focus if you see the road map issue the first item is more benchmarks and more testings cool great questionability something right yeah thanks to our community uh actually H so uh the question is about observability so I would say thanks to our community in particular folks here we have a lot of sort of uh new features added in 3.0 about uh uh points 3.0 about sort of premisus adding premisus metrics adding better loging support that longer term you should be able that'll help you monitor uh V in production cool any question about the features or clarification but what do you consider your clest compe cool so um in interestingly I don't think we really have a competitor as a software where for example for each where our our definition again here we're trying to build the fastest us use inference and surve engine this means Alternatives includes tensorrt for example from Nvidia and TGI from hugging face but from our point of view all of these are great projects that we're not in front in particular for example fore and Uh custom use were near and law front andt and the competitor and we continuous Benchmark against them yeah yeah I think yeah also want to add here I think yeah we also mentioned this in our first meet up is because we are open source project we are very happy to see any kind of new improvements and your new advancements from other open source projects and we are happy to learn from them and yeah study the improvements yeah yes any any uh plans for like support for M3 chips uh so the question is about supporting for the M3 chips uh we debated about this uh this is something um we can potentially discuss more uh right now vom focus a lot on sort of the server inference but we um if there's interest and if folks are interested potentially it would be great for both sort of local inference as well as um sort of local develop Vel M experience yeah I think yeah I want to add here I think that's doable and especially I think the way we are trying to support more Hardwares is to trying to find a good abstraction for these Hardwares and how to make it make VM easier to add a new hardware I think yeah after these kind adding other Hardware it should be also be beneficial for adding like you know like M3 yeah yeah uh I I think it's important to add support for the architecture is similar to the GH 200 chips from video there first the unified memory access yeah yeah so for the G to other chips you can access the CPU and run the to jel SC it m in the long run it might it's might end up being the thing where and same memory y so it could be useful to have some sort of a unified memory yeah that's a very good point with unified memory it can potentially helps improve our sort of performance and architecture yeah by the way I just forgot I to repeat the question so the question just now is about like a support for Mac M3 yeah yeah yeah yes so you mentioned you know if companies can help you so the alliance goal is to Res for on your road map do you have asks right do you specify what you need we can popularize it within the alliance I mean in the video is not a member yet so get some AMD some AMD some stuff with talk right so please please provide a document for what you need and we will share it with the members and see what yeah yeah yeah yeah I think the so just to repeat for the video it's like a comment about our road map to direct ask for what what we need I think yeah right now we have a we decide to publish our road map quarterly so we actually just published our like Q 2024 q1 road map on GitHub yesterday so yeah please take a look and we will add more information over there yes um so I'm from the cloud native ecosystem uh the home for kubernetes and there in the project we are doing our best to support different kinds of uh resources through the thing called Dynamic resource allocation H and I'm curious about how if those efforts are useful to you as you are um building the llm looking at div diversification of Hardware what you would need from the cloud ecos system if anything at all to support your fragant yeah good question so the question is about sort of uh what how can uh the cloud native World CES uh work together uh and to help vom in sort of the technical aspect and I would say uh I'm very familiar with the cloud native world I would say V right now focus on sort of running inside a single container single PO on a single GPU but uh down the road and I believe many of you who are already using VM are doing is to run it inside a custom manager and for that uh we do need to see good Integrations in fact just today I've seen somebody from kerve which is a cncf project adding integration between vom and uh sort of K serve project thank you great question yeah there someone in the back any of you yeah yeah yeah wondering how I yes I saw something output for the I just wondering how that TI structure support that's question for you yes so the question is about uh the structure output and that uh I guess you are probably thinking about the technical part or yes okay so the technical part about structure output we're utilizing basically how does structure output work is that a during each decoding path after the model has generate uh proposed a basically a token where you can sort of constraint how like how does it which token basically out of the entire token space you'll produce a distribution of these are most likely tokens and structure output essentially is to constrain the sample saying like Okay you are only allowed to sample from these legal sort of token and so such that we ensure the grammatically is correct if you are this is this is like following approach like outlines and guidance and others many have done that Jason formers and many have done that yeah so I don't know how many Comm do have Inc more also yeah I think definitely we are trying to add oh yeah repeat the question so the question is about whether we would like to add more commers and and in the future and what's is the standard I think the answer is yes absolutely so we definitely want to add more comuters and also uh what's that called yeah review yeah a review essentially is a triag sign in the in the inab yes yes yes I think we'll definitely add it I think it's uh more right now we okay because it's still only like four of us and probably at with Anthony Anthony from an we have many anthonies yeah and and and yeah yeah yeah and also yeah and we have some Tri from uh China basically they are all like most most mostly like you know top vrm contributors so we will reach out to them and we will we can make if we can make sure okay they understand VM codebase well and they can make good PS and we will just promote them it's a pretty natural process there is not a very hard criteria right now but yeah but I think as we grow we will try to make this more and more clear and Mak it more and more you know like organic process to add more people here to into the project I think we don't have any wars here so we would like to welcome all kind of contributions yeah yes one last question um you already mentioned that the SJ L project contributed uh multi low support so it looks like you guys already like finding each other how do you characterize relationship between the VM and SG L projects from your point of view so the question is about how do you characterize the relationship between SG L and uh and and vom so let me start with like we're actually from the same lab so and we have the same advisor so it's literally as the same group and like for example as a like sit right next to me in the lab and so um the project essentially will have a um a sort of a separation where estr focus on the grammar the structured input and output and optimization in the sort of the the the LM application and they have this code design approach where you have to speak their grammar to get the benefits of their optimized run time inside the run time they actually runs on top of vom because V offer such a great benefit of for example we work across different Hardware we have different optimization we have high stut low lency and works on in like distributor setting so we actually have a very clear layering here to and and in and then have basically if you raise a block they can mention here are the list of specific optimization can do if you adopt our grammar essentially that goes a long way saying that where V is where trying to build this sort of common backround for optimiz element inference and S is pushing the frontier of can you do with this kind of L protocol application yeah cool all right so one one last question yeah yeah uh I can bring up like an alternative but a couple of uh feature requests are ask here I I know are available in the lot of project here's how you guys think about the ls you ecosystem I know it's like yeah yes but all great so that you yeah yeah I think uh so the question is about how do you compare VM with like project like llama CPP I think at beginning we have a very very different goals so llama CPP is TR to enable you to run on your laptop and but we are more like you know throughput oriented we are trying to focus more on the on the server side and more on the on the basically High where you like Enterprise want to serve the that side of the work but I think we are learning from each other I believe Lama CP is also trying to add things like page attention and continuous batching and also on our side we are I think we are looking at for example the quantization quantized kernels very good quantized kernels from Lama CBP I think yeah in general that's uh that's what we are thinking but yeah and and I think one thing I want to mention add here is basically uh in terms of model so Lama CVP is like you know completely C prop implemented model but right now we would like to stick with pytorch especially with torch compile I think yeah when I think at the very beginning we saw the like you know overhead of pyto and we we saw about switching to C++ but L basically when we are talking to the users and like basically through time we find out many people actually like P toch because they have their own models and they want to iterate faster so we would like to uh keep this benefit and stick with py toch and make sure use techniques like torch compiled K things like that to bridge the gap between Python and C++