DIY LLMs with Modal

DevReal: ​DIY LLMs with Modal, Charles Frye

Recording: DevReal: ​DIY LLMs with Modal, Charles Frye

yeah thanks uh Lexi for organizing this event and uh for yeah sharing the overall picture we're now going to go out uh we're now going to zoom really into the scanning electron microscope level from the from the galactic uh telescope level um so I'm gonna like sort of share some of the things that you need to know about the hardware that LS run on about gpus in order to like run and use them effectively you're probably not going to end up like writing this software yourself but even if you're using a nice uh Library like VM uh to to run things there's going to be Flags there's going to be configuration and you need to sort of like understand the like nature of this Hardware in order to understand like which flags are important or like why is this why is this a good idea and why is this other thing probably just like LinkedIn slop uh kind of idea and um like I guess many people work with databases but very few people could like cat up a binary tree with concurrent accesses but lots of people can like know how to create an index know like what queries it'll speed up like oh it's probably good to have an index on this thing that I'm hitting all the time so that's kind of the like level that I want to help people get to um and uh so in order to like uh like first we need to understand like what actually is happening when you run an L right like most people are have consumed these things via apis VIA the Open Eye API anthropic whatever um and so they don't haven't had to think about what happens when you uh when you actually run this thing and for one last bit of like highle context like if we're going to like move most AI workloads out of these like large providers and into an open source AI stack we're all going to need to learn a lot more about how to use this hardware and how to run these workflows ourselves um okay so roughly speaking what happens when you run a language model every time that you generate a token from a language model you have to take each weight in the model where we've like heard about like oh it's a 70 billion parameter model it's a 673 billion parameter model um so that number that's the number of floating Point numbers that we're going to have to load in um and do a computation on um so there's there's other numbers but this this is like napkin math level right so that we can have like a high high um high level view so we need to load all those numbers in uh we need to take them from like the area I've indicated in green on the GPU the like memory of the GPU we need to move it into the part whereare computation happens do something with it move it and then move it move something else in um and so basically like what this kind of looks like for let's say you take a a short prompt is guava fruit you're passing it to uh an 8 billion parameter model with one byte weights you're going to need to move 8 GB in uh in that like when the prompt comes in then do 64 billion floating Point operations more or less um and then uh to that'll generate your first token yes um and then you need move 8 gigabytes again do about 16 uh uh billion floating Point operations then like that'll generate your next token then move 8 billion numbers again um so it's this like it's this very very different workload from a lot of things people are used to doing it's very different from like a typical computer program which is like jumping around grabbing little bites from memory doing like one quick thing with them then grabbing some other bites from somewhere else um it's uh it's very different from the way databases work which are like yeah grabbing those specific uh specific chunks to grab a row from a table or a chunk of columns um it's it's a little bit more like a big sequential scan of a database we got to like just run through every single row there's no way to be smart you just got to be fast and uh you know that's that's what that's the workload that we are now trying to like accelerate and run a lot of so like one feature is that it's like relatively simple and that's the only reason why it can also be like this big right like you aren't moving 8 billion bytes around in like a few milliseconds in most other workloads um yeah it's uh it's like you know it's like kind of scary to think about the like size of these numbers here um like a like CH like GPT for's training run probably moved like a mole of floating Point numbers around like one floating Point number for each water molecule in like a glass of water um that's like a lot all right so like for that workload like why gpus why this particular Hardware um I want to give you the like high level intuition for this um it's gpus are designed for that workload run simple calculations to high throughput started as doing that for graphical frames uh for video games but uh it's now this particular workload um how do they do this they get high throughput via two mechanisms parallelism and concurrency uh the classic tricks in in uh computer science um one of the most important features about the like actually just the like raw silicon is that the compute part is way faster than the memory Parts the part moving btes around is relatively slow compared to the part that actually calculates stuff which is weird because humans think like the calculations is being the important part and like just storing information feels easy um and this actually leads to a number of very counterintuitive optimizations um not least of which is inference time compute scaling which I'll briefly talk about um and then lastly I'll probably skip this for reasons of time but there's like a particular stack for programming gpus um uh the Cuda stack which I'll give like a high Lev uh summary of time permitting all right so gpus run simple calculations at High throughput um CPUs are this like very complex type of processor they're on the left this is like a toy diagram of the actual die where calculations are happening most of it is actually most of the space on a CPU is actually not for calculation that's the alus in Green in the top right uh that is actually a smaller fraction of the Silicon area compared to things that are about control flow and like actually even like predicting what the next thing your program is going to do so you can start doing it before You' even asked and then also caching like storing information like what are the recent instructions what are some like recent things you've accessed because you don't know like writing your program you don't know what things you're going to ask next because maybe it's data very data dependent control flow which query came in um like and like which branch was followed so you need like you need a lot of space there um and there's like complex logic around it um and that's the like CPU design is oriented to that uh on principles of like spatial and temporal locality gpus are oriented around the principle of like number goes bur uh the like vast majority of the Silicon area is devoted to arithmetic logic unit equivalent something that does math and logic um and that's uh that's on the right and like they're they're dumb they're not good at this like control flow if you start branching they like grind to a halt um so that's like it's like the area on a chip is like very tightly optimized and if you can find a part of any system system whether it's like a technical one or a social one where like things change really slowly and there's a lot of like competition and bottlenecking like that tells you the nature of this system uh and for gpus it's like we will do we aren't going to be smart we're just going to be fast um uh and so that is like the the CPU is like one uh horse-sized duck uh and the GPU is the is the 100 duck siiz horses um in the uh the famous like joke or meme um so the like the fun fact about is like gpus have won at this like really important workload by just uh you know scaling out uh on on Tiny things okay but that's like there's that's that's enough to get like okay I understand why um you would need like specialized hardware for this but like how when you're actually like programming gpus or or configuring programs that run on gpus you actually need to understand a a little bit of like how does that how does that how can you actually make use of all of that stuff um so the number one thing is like you want to be using all of the all of the logic units on a GPU you want to be using all of the cores all of the time um and so that like you're you're executing things in parallel which means like at any given moment many things can make progress at the exact same time not like one makes progress and you switch to another and then you switch to another that's like very much what the operating system of like a and and like the CPU are are designed for so that you can like be on slack and answer emails and be on a video call all at the same time um gpus are not for that kind of multitasking they're single taskers um but like like many little copies of the same task so like for numbers on an on a each clock cycle on an h100 16,000 threads can all make progress at the exact same time right 16,000 which is like uh I I want to say the number for like even the biggest like thread Ripper type CPUs like an order of magnitude lower um maybe even more um yeah the numbers are actually at the link there two orders of magnitude thank you um yeah um I got in a fight with a CPU guy about like what the number anyway I be a CPU guy okay so two okay I'll I'll I'll Point him to you next time yeah um yeah so this is one reason why it's like large language models why have we moved in the direction of this scale it's not just like a skill issue we can't make the small ones good it's not entirely that it's also that when you have really big models you can run a lot more parallel threads because what you're saying with a big model is you're saying like take this operation and like do it many times like a billion parameter model means like 10,000 element vectors or 50,000 element vectors all over the place and you can imagine basically like one thread per array elements like Loosely what what goes on inside these GPU programs um so that is that like mostly that's like Cuda kernel Engineers problem to figure out like how to how to do that the like and that's like relatively intuitive to people the part that's most unintuitive is like is concurrency um like being able to operate on multiple things concurrently like which is not necessarily the exact same task but like a you know another subtask um is another part of how gpus get high throughput so on each clock cycle 16,000 threads can make can all simultaneously make progress but also on like you know one of them might need to like read something from memory and that's like eons you know whole micros seconds uh and and so that would you know that would like kind of torpedo your ability to use all the all the Silicon but uh at each clock cycle you're not forced to run the same 16,000 threads there's up to 250,000 can run like can be selected from to be part of that um part of the lucky 16,000 and so whenever like some particular Hardware or thread is like idling or stalled other threads can make progress um and so that's like there's there are equivalence of this kind of concurrency in CPUs instruction level parallelism in CPUs um but what this what this means is like you might think like oh if I add more stuff like if I try and do more things that's going to slow me down but if like the additional things that you want to do like make use of a different part of the GPU um they can be like scheduled concurrently and they'll both make progress so long as they you know don't like directly depend on each other so like in this case like we're seeing a couple different like streams of work in those different colors going into thread schedulers on the GPU and then they're getting like interleaved so that like while one sub the subtraction is waiting for something to be loaded the addition actually like keeps going it finishes and then now the subtraction is finished so we can move on to the next Edition in the blue uh blue stream of work at the top um so like to the goal of understand of like looking at the at the hardware closely is to understand like why why are certain things like easy to do available like and performant so the the clearest example of this I think is this is why everybody's low rank adapters or luras um in this particular way uh uh rather than the like old school like um you know raise your hand if you ever did a like uh trained a new head on like a convet or a Bert model uh like f yeah okay yeah Stacy worked with me at weights and biases many many fine tunes back then so it's a quential adapter is like okay somebody trained the super smart model let me rip the like B uh end of it off and put my own end instead uh and the problem with a sequential adapter is that you take the output of of one step and then you transform it becomes the input to the next step and that like that kind of sequential stuff can't run concurrently you can't do both at the same time because you depend like the thing that does the like base language model calculations now has to like then hand it off to your like corporate Laura that puts it in your own like personal brand um but luras are parallel adapters so the the sequential adapter is drawn here on the right side of the slide um and you would like kind of Stack that at the end of this this um this layer here so I guess it's yeah it's like if you see there's like a dashed adapter written under the block that says above the block that says attention that's like a sequential adapter but a a Laura uh sits alongside the attention matrices so like at the same time as you're calculating the other the like main attention Matrix that meta or or um deep seek trained for you like you're also at the same time running the thing that that adjusts it to your particular workloads to your particular environment and they can these can be scheduled and execute the exact same time um because they're just like two completely independent streams of work um and this so this makes much better like this is much more mechanical sympathy with the actual Hardware um and so these are like way easier to to to run and make performant um I won't go into too much detail on this because I bet Michael's going to talk about this um but it's like kind of surprising if you look at the way people do quantization there's like I'm going to load a really small number like a really low Precision floating Point number and then I'm I'm going to like bring it into the GPU and then I'm going to like scale it up uh and then I'm going to like do operations on it in high precision and then maybe even like cast it back down at the end um and I was like kind of surprised I was like wait isn't this like isn't this going to lead to like extra work that should show up as as additional latency in at least like some case turns out you can actually do these things concurrently so you can do these like um uh you can do the like scaling up of the um like scaling up from low Precision to high Precision at the same time in a different set of registers uh as you're doing the um the like actual uh uh yeah the yeah so yeah so like literally it's like some stuff runs in these things called the tensor cores and that's the actual Matrix multiplications but everything like they're chugging along but everything else is actually sitting idle and the GPU can like if you just give it that extra work to do it'll do it and and it won't slow down at all um this like kind of counterintuitive um like because it it feels like they you're putting in more work but that's like the thing with concurrency you do um you make use of idle things um and so it's free um it just costs you know really great Engineers many hours of pain um which is free um so uh so then like the the other super counterintuitive thing about this is that the part that does these computations is much much faster than the part that loads things for memory so and this is this is like even though people have already kind of like pushed this subsystem as hard as they could so um like if you've ever built a computer uh you know that you can like you can't download Ram but you can like pop Ram out of a computer and pop new Ram in like to to upgrade it and you can't do that with an h100 you can't do it with with uh like contemporary MacBooks either cuz they have this high bandwidth memory so this is like actually like soldered onto the same chip um it's a it's like a different silicon unit so they aren't like fully integrated with each other but it's as close as you can get without like just making it the the like direct memory of the compute um and that achieves like that that's like what where people have been pushed to try and like have the memory keep up with a compute but even then is like continued to fall behind as people have pushed the like through of the uh silicon up uh and remember like our bird's eye view of what we're doing with the language models we got a low data in memory like a couple like we have to load 8 billion numbers into memory of like from the the like high bandwidth memory into the compute do a couple things with it and and maybe write stuff back um and so this is the reason why there's a bunch of sort of like a bunch of the optimizations that and tools that you find out there are trying to like help solve this problem so so um like and this is particularly bad for like small batch LM inference like you've got like one or two queries coming in at a time like you got yeah you are you are moving 8 GB and people want the like the next word in like a millisecond 10 milliseconds um and you've got to be like moving like you got to move stuff in and back out like o like every single time a token is generated um and so the the like this is like this is a very important fact about like um GPU programs in general the arithmetic intensity of the GPU program how many how many mathematical operations do you do for each memory load um and small batch LM inference is like uh really far away from that um so this is uh this is one reason why torch. compile the new like compiler feature that that has been in in pytorch 2 is so important like if you write a normal like if you're like writing an actual like neural network program yourself you'll say like I want to do this operation then I want to do this operation then I want to do another operation the way that torch works is it's going to like launch all those as separate things that happen in the GPU um and so in each in each case this is like much smaller than the like an entire token this is more like multiply two numbers together add some add some numbers to them this is like a a more granular level um those operations like in regular torch would require like go in to the compute unit and then come back out go into the compute unit and come back out for all these like intermediate tensors like the key value cache or even like things that don't even have names cuz they just symbols in an algebraic equation and torch compile one of the things that it does is it takes a look at that and says oh wait a second I've been like you took this thing out you did no manipulations to it like you just put it back in and ran another program on it let's just like compose all those together it's called operator fusion um and torch compile but does like a bunch of smart tricks to solve this problem it's also part of the uh like Cuda graphs another optimization that you'll see it's like also trying to avoid these uh like this kind of like overhead and some round tripping at least with like virtual tensors and cud graphs which I'm not sure people use as much virtual tensors it reduces CPU overhead okay yeah yeah yeah that so the other thing with cogaps is the reducing CPU overhead from Colonel watch that different thing but the ver yeah that's underutilized feature IMO but yeah okay so then um this also why so I mentioned that like ization you you'll see these qu quantization things that like all we do is quantize the weights um we don't um we don't quantize the activation so like and again like intuition is like oh I'm doing a bunch of math like if if I have low Precision storage but High Precision math like why is it any faster to have these like low Precision weights um so this is the like uh High Precision weights are there on the left as like a big block of numbers low Precision weights are on on the right as that like skinny block of of numbers that you know needs to be blown back out to the larger size um and it because this uh like because memory is the bottleneck on many of these workloads it actually like like trying to speed up the the uh actual calculations wouldn't show up as any additional speed up uh because actually the like they're actually sitting idle most of the time waiting for something to be loaded um and so like these like you know weight only quantizations will like frequently give you all the speed up that you could expect to see um great um and then finally uh inference time compute scaling is also like uh such a powerful idea in part because it it like leans into like doing more operations for each memory load so this is a graph showing the victory of 100 D horses over one uh um uh large duck so the large duck here is GPT 40 it's trying to write python programs and the y- AIS is how frequently it fails to write a correct Python program um so you the the uh dash line is gp4 O's performance you ask it one time uh write a program uh and uh does it fail so it fails about 10% of the time on this particular data set um then if you take uh like small language model like llama 318b or smaller language model like llama 318b you run it one time and it does way worse than um than gbd4 but you can like blow that up and run it a 100 times and uh and you'll see like if you take all those and you check which of these python programs passes the tests and you take any if any of them pass the test you win um which is you know how software development works right you just have to right you just to write programs until the test pass yeah um which is to say like in software development you know whether you've gotten the right answer if you have good tests um and so you can just so you can just run this until you pass uh until you pass the test and you actually do better than calling the big language model once and the counterintuitive thing is that it can in fact be cheaper to run a small model many many times because you can run them in parallel um so like a single memory load uh now becomes like a 100 tokens output instead of one like token of output and so it's like it it increases the compute intensity of the workload like per query and that's actually kind of a better fit for the hardware um so yeah um so this is like this actually makes it it also like because you can kind of trade off more flexibly like how much you uh how much compute you apply to a given problem it also makes it like easier to make economical use of the hardware because you can say like oh not many users right now everybody every user gets a whole GPU to themselves oh like big Crush okay well like model gets Dumber for a bit and every gets worse outcomes but uh you know it's easier like you know our infrastructure can handle it instead of going down um yeah so I think uh running out of time so kind of this the stuff about um the like Nvidia stack and the uh compute for it or the software for it I'll kind of skip over um like hit me up afterwards uh for more details on this I guess I would just say um like very high level language and the part that's actually much more slower change changing and where the like uh bottleneck is is in the that Cuda gets compiled to this intermediate representation called pts um and that's the target of the Triton compiler uh that now I produce which is now how people are starting to like people are using that targeting this Intermediate Language and that's allowing people to like Explore More of the design space for writing programs for concurrent and parallel systems um so like both gpus and other ones that try and like achieve High throughput um the like couple of like core intuitions about gpus they run simple calculations at high throughput nothing fancy um they do the like the fancy part is not the calculations you're running but how you get to that super high throughput which is like H being able to do a lot of work at once and also being able to have lots of different type uh types of work to do uh so you can jump between it very um effectively this leads to counter optimizations um including like the you know luras instead of sequential adapters um finally like because comp is so much faster than memory um like things that seem like they should like linearly increase your costs may like not show up as increased cost or increased latency at all um and so like there there's like opportunities for for scaling up um okay so uh a lot that's like a super fast uh run through of a bunch of deep stuff uh wrote up a bunch of this in this GPU glossery um so that's available at moto.com gpug glossery it's like all of these things like tightly Interlink so you can go from like reading about the software uh to reading about the hardware details to then oh what's the like you know um what's the abstraction that Maps those two together um so definitely check that out um I guess I can say this um we like right now this is just like a thing on our website but we are uh like people got really excited about it people on uh Hacker News on Twitter and elsewhere so we're actually going to make this open source so that other people can like contribute to it um and uh build uh you know build on it and bring in other you know knowledge that we don't have um so excited to like sort of help spread this knowledge sort of like commoditize this knowledge the more people can get out there and run their own llms and uh um and make effective use of this Hardware add it to the general sort of software Engineers toolkit the way that we have networks and databases we'll have GP accelerated neural networks as something we can just whip out um so uh part of making all that happen is you need to get a hold of this Hardware need to be able to like uh uh this uh it's hard to get a hold of it's like painful to administer this is like one of the problems that we solve at modal it's company I uh work uh work at we um make it easy for you to run code in the cloud where there are these like beefy gpus we make it easier for you to run things concurrently in parallel not just on highly concurrent and parallel Hardware like gpus but on many copies of them asynchronously communicating with each other um and we do this like with a like basically built a custom container engine from the ground up designed for data intensive workloads to make that possible um and to make building those systems and developing those those like uh high performance systems as easy as developing something locally on your laptop um and that's uh you know not just the like GPU part but also like um like s like you know little sandbox units of code uh serving things over the web um and uh and deploying them so you can uh do it all on modal uh happy to talk about it more afterwards if people are interested um yeah come check it out thank [Applause] you Bel if you're my question budget or uh we have I think one or two very quick questions bring M yeah unless the recording mic will project a little uh so when when you're doing something like serving an llm um trying to understand performance bottlenecks how do you think about profiling that other than um double the batch size see if it's faster double it you know yeah yeah yeah yeah so definitely always double the batch size until it order like until it out of memories and then and then pull it back um uh that's the that's number one tip yeah so the um the like first page of the Cuda best practices guide is just like a really long like please look at profiles please look at traces it's very very hard to just like look at code and into it whether it's going to be faster or not I think this is generally true writing super high performance stuff you got a benchmark um but it's like especially true with this Hardware which is very like counterintuitive so um uh torch has a profiler that I like a lot it's like it's more like torch chips a bunch of like Integrations and and uh uh on top of that um on top of the like core thing offered by Nvidia they have like their profiler that can do both stuff on the CPU and the gpus you can make sure that the CPU is not slowing down the work on the GPU uh and getting in its way which means like you have easier problem like you just have to do normal software engineering um versus like being able to look at it and see like where am I spending time like like you can figure a lot of these counterintuitive things are much easier to figure out by like staring at a sequence of events and being like Oh wait a lot of time is being spent here not a lot of time is being spent here um so yeah strong recommendation for using the profilers uh Nvidia Insight is pretty good um but you can also just like use the torch profiler and Export something and look at it in profetto um but yeah you just like aggressively look at the look at the traces first profile is very statistical very like hard to parse until you have a good mental model um so the Trac like look at traces every time it's a new workload look at Trace um and then once you once you rock the workload well enough like graduate to looking at profiles to make those decisions and um yeah uh and compare like compare them empirically yeah I can for cool uh what's your perspective about scaling law yeah um so yeah scaling laws are important part of what has made all this like um actually engineerable right like so this is this is really this is a scaling law that I showed with this this is an inference time scaling law it says that like you should expect this amount of like a multiplicative decrement in your error rate for a multiplicative increase in the amount of compute and so this allows you to actually engineer and say like oh well if I like I can look at this and say like oh if I were to take it to 10 looks like this thing hasn't started bending yet it's going to probably still be a straight line across another order of magnitude so it maybe go down from like down closer to 10us 2 so down closer to like 1% error rate rather than seven um and so this like with inference Time stuff now you can actually see these and you can like um you can generate them for own problems uh they're also used at train time to like to like drisk running a big training job and so they like help like you you don't get that many even if you raise $500 billion for or allegedly raise $500 billion um like you can't just like go willy-nilly running you know entire data center multi- dat Center scale training runs like you want to be sure it's going to work before you do it and so the scaling laws you like the the magic is that you can do a bunch of smaller scale stuff and then project out to a much larger scale confidently and so like this is like if you were training your own models um like you should absolutely be doing this um and it's like yeah a little bit embarrassing for the ml field that it took so long to like like come across this idea um only with like open Ai and and deep mind um like in the year or two before uh uh gp3 um when it's like a it's like a fundamental engineering to for for building uh ml models and why will you Pi the uh just these two so just for comparison in The Benchmark like other open source model or like cutting face dipstick you you name it and then would they perform just like a or is it some expectations yeah I think the surprising thing about inference time compute scaling is how robust of a phenomenon it is like this is I saw paper that that like had this result on a different data set with different models the first different data set and different model that I tried it on showed the exact same phenomenon um and so there's like a universality to these scaling laws that gets a lot of physicists very excited um but like uh you can generally expect that like this you'll observe this phenomenon um and that also makes it powerful because it's a very general engineering tool okay than thank you thank let's than [Applause]