Charles Frye, Parallel Processors: Past & Future Connections Between LLMs and OS Kernels.

Charles Frye, Parallel Processors: Past & Future Connections Between LLMs and OS Kernels.

Recording: Charles Frye, Parallel Processors: Past & Future Connections Between LLMs and OS Kernels.

[Music] looking forward to presenting this talk parallel processors um this talk is a bit different from a lot of the other talks like the talk that you just sh saw from Stefan if you were here um this is uh a vision talk uh and a talk about what you know what could be done um and it's also a talk about an analogy uh the analogy the emerging analogy between large language models and a layer much lower than them in the stack the metal um this intersection of processors uh kernels and user applications and there are three basic goals that I have in this talk uh one is to draw attention to some of this work that I really like at the intersection of large large language models and systems uh the second is to inspire some of you to do some future work there especially on uh efficiency and on agency to problems where we need a lot of improvement to deliver on the promise of large language models uh and then finally to explain just uh what the heck karpathy has been tweeting about recently um if you walk away and can understand his recent tweets then I've done my job here um also as a sentimentalist I want to dedicate this talk to my great uncle Bill who worked on uh the semiconductor process at Intel in the 70s um and to my sibling Teddy uh who shipped their first web game app to external users this week and uh everybody who's worked on computers in between all right so what's the tldr what we're going to cover here so there's been a lot of excitement about Transformer based large language models in the last year and change uh a lot of people have been claiming that they're software 3.0 a new type of computer Uh I that can be made a little bit more rigorous they are differentiable natural language processors we'll talk about what that means as many have pointed out llns are in their demo era right now we can produce demo Weare very effectively at a rate of like five Banger tweets a day um and a lot of people have complained about that uh but this was also how personal computers got started so we'll talk a bit about that and the connections there uh We've also seen a wave of efficiency improvements to large language models in the last year ported basically directly from the layers closer to the metal like paged memory uh we'll talk about those there's a hope that we can actually Port more sort of conceptual things from the operating system kernel layer in particular to make something like an llm operating system and I'll uh put a little bit more meat on the bones of that idea um I think we're not quite there yet we're missing a few pretty critical things uh that I hope we can work on in the coming year uh on large language models to deliver on that promise all right so now if you're the impatient type you can just leave now and you've got the you got the gist so you have maybe seen the claim that large language models allow us to have a new kind of computer well what what kind of computer differentiable language processors what does that mean the computers that we have now are based off of central processing units that perform logical operations on address based memory Transformers are very different they perform vector-based operations on content based memory uh this also means that Transformers and the programs they execute are not Turing machines they're a thing called a rasp machine uh which did not exist before Transformers were invented uh and CPUs cre created a particular collection of user interfaces that we're now all very familiar with terminal user interfaces and graphical user interfaces these Transformer based models uh unlock perhaps natural user interfaces a holy grail of human computer interaction so double clicking on each of those points in turn uh CPUs perform logical operations on a dress based memory so this is these diagrams are from nand to Tetris which builds up a basic computer and operating system from The Logical gates up to the game of Tetris uh and the introduction begins saying that every digital device is made from the same building blocks Elementary logic gates these Elementary logic gates take in zeros and ones and output zeros and ones like this Adder gate uh shown here you put in some zeros and ones and you get out zeros and ones we're maybe used to thinking of zeros and ones as trues and falses in Boolean operations but you can Implement addition and and things like floating Point operations using only these truth values U the memory in our computers is also based on the same basic principle of these logic gates uh so this is this is what all of the applications that we have talked about at this conference in the end compile compil down into these Boolean operations I just want to quickly draw your attention on this slide to the tiny fire emojji next to a link I put those next to resources that I think are particularly useful and interesting uh so that I don't have to keep saying it every single time so watch for the fire emojis on a slide that you like so Transformers are a new computer are they doing bullan logical operations if you want to get technical yes because when you go down they are implemented on the computers that we already have um but the computers that we already have are also implemented on the basis of the differential equations that describe the evolution of electromagnetic field so so you know they're not digital and binary either so let's follow those principles of abstraction and say what is the natural way to express the kind of operations that Transformers do uh so Transformers do Vector operations on content based memory so the diagram you see on the left of the slide there is the classic uh Transformer architecture diagram and in the middle of the slide is a slight reorientation of that diagram from the folks at anthropic so you normally when you think about the operations of the Transformer layer you think oh I do some attention thing and then I do some feed forward neural network thing and then I create the outputs but very critically about Transformers is this residual connection that goes along the side here and these boxes represent the different operations but this one in the middle this add one is a very simple operation ignore the normalization factors that's a gradient flow thing you know they could be removed um and in fact recent work has removed them in in many cases so focusing on this we're just adding something back in and what this means that when when we do our operations we preserve the information we don't destroy it to create our outputs uh so we can reorient ourselves drawing this line here of all of the information passing from the input to the Transformer all the way to its final outputs and that's following this line here but just sort of Shifting everything on its side and following a similar algebraic transformation that allows us to reorient how we're thinking about the keys and queries and how we're thinking about the uh MLP layer uh we realized that what's what's kind of really going on inside this architecture is that inputs come in something is read out of them so we extract information from a high dimensional Vector down into a low dimensional vector and that information is transformed in some way and then written back out to the same place that it came from so this looks a lot more like reading from a memory and writing to it uh so read from memory transform write to it just like we had in the um more familiar uh logical operation case but now our fundamental operation is not some truthtable describable thing it is a uh much more Vibes based content based um operation on that memory uh so there's uh the details of how exactly that works are still being worked out by the folks that an open open Ai and elsewhere who are working on uh decompiling these Transformers uh but a really useful mental model comes from the world of theoretical Neuroscience where content-based memories have been used quite frequently you can imagine like if you put in an incomplete uh something like this incomplete picture of Homer Simpson and you compare it to complete pictures of Simpson's characters the thing that's most similar to it is this completed picture of Homer Simpson and so that's content based memory I don't have to have a pointer to Homer I can have something that vaguely reminds me of Homer and then I can get Homer back so this is the core internal of language models it what's it's what makes them so powerful so able to transform our documentation from python docs to JS docs or to answer in the style of a pirate and to operate these analogies um it's also what makes them so frustrating because you can't tell them to look up information at a specific spot so this is the this is the core new computer that we have it's this uh high-dimensional Vector based content addressable memory machine we have a wonderful for those who celebrate model for uh CPUs in the turing machine uh which has a tape of symbols it reads from that from that uh tape of symbols it makes a decision about where to move on the tape and then what to write in the new position uh so this model actually predates the original the like creation of computers uh as a model of computation uh so do we have something similar for Transformers what I described as a very mechanistic thing about the actual guts do we have a nice algebraic model for them uh the answer is yes kind of uh it postdates the invention of the Transformer unlike turing's machine uh the trans each Transformer is what's called a restricted access sequence program a program that transforms one sequence into another sequence but but has restrictions on the manner in which you can access that information so this is probably not the time and the place to describe a novel model of computation uh so I strongly recommend Sasha Rush's tutorial on this raspy uh that goes through how to do this in Python um in like write a Python program that can be compiled down to Transformer weights that do things like count open and close parentheses and things like that classic CS stuff uh the tldr is that sequences are transformed into other sequences via VIA element wise operations like is this equal to the letter L so I can take hello and turn it into a sequence of zeros and ones that way uh you can also select and Aggregate and this is kind of the equivalent of the attention component of a transformer you can say I want to look for all the tokens that match this current token so all the letters that are the same as this one uh so all the letters that are the same as L and that will so you can select with Boolean predicates over the sequence um and then you can aggregate via sum so that restricts your ability to access the sequence you can't do whatever arbitrary operation you want you have specific access patterns that you have to follow but by combining these operations of elementwise plus Boolean predicates you get out uh like pretty complicated behavior um so it's very different from the way we think about programming computers and the way we do computer science for doing you know Big O notation for the behavior of machines um but it provides some of that similar ability to like write very um like yeah mathematical or controlled code and then you know execute it so this doesn't give you gp4 um but it does give you a model for the kind of thing that gp4 is so great work from Deep Mind by the way primarily setting this up so CPUs this different type of processor created a bunch of user interfaces that we know and maybe love like the terminal user interface um and the graphical user interface uh that allow us to interface with the hardware and and get our uh get our stuff done uh so what kind of interface can these new processors make I think this question's a little bit unanswered yet right now the interfaces that they're making are mostly just chat Bots um but that's kind of lame and maybe just an artifact of History so with a new computer and a new interface perhaps that inter interface is the newie the natural user interface this is an idea that goes back to the very foundations of human computer interaction which is that the ideal user interface is something that just like directly plugs into our intuitions about an object or about a system and all actions as a user to interact with that system are exactly the ones the system expects uh so this is a pure purely a demo but from the folks at new computer uh who this demo shows like what would it be like to have like an online shopping experience that's integrated with my home with a magazine on my uh on my desk where I could just tap a picture and say like send this dress here and maybe this that's a cute dress here to that friend I was going to do Halloween with this year so that's the kind of interface that if you're interacting with another human you can that can get turned into actions in the world in a pretty reliable manner uh but we don't have that with computers we have to make sure to match people's uh schemas and uh and click very precisely in their interfaces so there's wonderful talks from new computer and from Adept on this idea um that of what these interfaces might be like where we've sort of rethought from the ground up now that we have something that's less like metal and more um uh softer or or Vibes based uh all right so that's this kind of highle vision of what this new type of computer is as a new type of computer there are a ton of analogies that can be drawn uh historical analogies technical analogies uh with the computers that we have already uh so uh I'm going to talk through this historical analogy for a specific thing which is the demo era of llms that we live in 2023 was essentially the year of the demo um I Am Not Innocent of launching a a very rickety LM demo onto the internet for likes um and a lot of people have complained about this but one of the first demos of the first PCS was like actually the grossest hack imaginable and we had in the mother of all demos the mother of all demo Weare for almost like 60 years uh so there is some utility to this demo era um even if it's maybe a bit embarrassing uh liftoff for our for personal computers was achieved with office software and we're just starting to see that kind of liftoff in um productivity software with large language models so perhaps Peak demo year of um for llms was hustle GPT which was somebody who was just prompting gp4 to make a business for it um so I gave gbd4 a budget of $100 told to make as much money as possible and then I'm just doing whatever it tells me to do uh so this was like a viral Twitter thread that was like very fun to follow along um where this uh yeah this guy basically tried to run a small business that was oriented around like Health exercise things like that um went out and like you know paid influencers and like bought ads and stuff um so how did it end it failed um so there was like some cool stuff that happened along the way it started kind of well but then it just like crashed and burned lost all the money um let heu without sin of losing investors all their money cast the first stone but uh this is not great um you don't love to see these like very non-robust demos uh but the first demo of the alter personal computer um was similarly rickety at The Homebrew Co Computer Club uh uh Steve dopier used the fact that it was not electromagnetically shielded shielded to program in songs by controlling the frequency at which the electronic components switched and then like just put a AM radio next to it and then like you could hear um I forget the song I didn't recognize it some song from the 70s uh and like this dude uh named Bill Gates was like this is a super cool demo and he went on to do some important work in personal computers uh so you know in the you future Bill Gates is perhaps sitting in a lang chain webinar watching a watching a hack demo um so we're in this like Homebrew cognition Club era right now we should expect these kind of riety demos even an entire year in to this process um another famous demo is the mother of all demos which in 1968 showed off two people collaborating using computers with they were both pointing to the same file they were both interacting and editing it at the same time they could see each other's faces they could speak to each other and collaboratively edit it's 2018 and this kind of works in visual studio uh like 50 uh 50 years later um 5 years after the release of video vs code live share I still don't think they have like a good working video integration into that which means the mother all demos is still a bit demoy to this day but that demo inspired work for the last 50 years on improving and making usable these Computing systems um what actually made them usable on a shorter time scale was office software so viakal and lootus 123 with the spreadsheet uh Word Perfect with document editing these things like cause Mass adoption once people could actually make their make themselves more productive with them and with co-pilot in particular and Chad gbt as well perhaps we're starting to see some of the first steps in that direction uh the first steps of of actually making people more productive uh so pivoting there's also the opportunity to use this analogy of Transformer LMS as a new computer to take the things that we have used to make computers more efficient since you know the the first computers were built and just Port them over to this new type of computer so there's been a number of wheel reinventions in the last year that are very exciting so the VM inference server ports the idea of paged memory and this doesn't only make the thing a lot more efficient but the manner in which it makes it more efficient unlocks actually smarter and better models like Monte Carlo tree search based models speculative decoding ports the idea of speculative execution uh over to language models and improves their efficiency and efficiency isn't necessarily everybody's favorite thing but it's allowing us to sort of cram more cognition into integrated circuits which is in the end one of the most critical things to keep in mind as this field is developing so the concept of paged memory is introduced for a number of reasons well one of the key ones is that it reduces the fragmentation of our memory so without paged memory every process or task is allocated some big block of addresses or it's allocated address by address which which is very painful um in the case where it's allocated in large blocks then if I have say three processes that each use up 150 memory addresses and I have 550 addresses in my memory I can clearly fit those three potatoes into my three potato sack uh but only if I place them very carefully if I misplace them then I end up without the block to allocate to the to the third process if I instead break the memory up into a bunch of little chunks Alias Pages then I can give them out sort of piece by piece and ensure that with much higher probability I can fill up the majority of my memory and avoid being unable to allocate to uh when I when it is in principle possible so that kind of fragmentation of memory is referred to as external fragmentation so fragmentation outside of a process outside of a unit uh there's also internal fragmentation which is not depicted here but you can imagine if I give out 150 addresses to a program that might be like the amount it needs eventually but it might only need that when it's doing like one task for a brief moment that's very data intensive and then it doesn't need it anymore so you'd like to be able to pull pages in and out of a program as well and that is called internal fragmentation when you aren't using all of the addresses um so VM uses pages to speed up inference of Transformers by paging this key memory structure for Transformers called the KV cache the key value cache and it's the basic idea of paged memory but applied to kind of like at the token level of saying I'm going to store a bunch of information about an individual token logically at the level of my like of a certain level of my Cuda stack I'm able to just think of it as like a line of tokens one after another or a line of representations of tokens one after another uh but then physically there's like a there's like the same abstraction layer the same IND Direction like a page description table that says oh no actually if you want to find that it's actually here um it's not you know they aren't one after another they're actually split up and this like substantially improves the memory usage and so the throughput uh due to the memory boundedness of inference uh of serving Transformers uh and so very large benefits both for external fragmentation and for internal fragmentation uh so avoiding needing to allocate a bunch of of memory for a sequence that doesn't need it and avoiding um like having gaps between our um our sequences so that's that's very cool because of how much faster it can make your uh language model inference but I think the more cool thing about it is that this it's handling that caching and memory allocation better is enough to actually make models smarter right now when you draw samples from a language model what basically everybody does is they calculate probabilities for next token and they draw one at random from that and that's basically the most naive way to sample from a probabilistic model uh of sequences just like token by token it's called greedy decoding and we all know greedy algorithms are bad uh and one of the reasons why we don't do anything more sophisticated is because it requires very thoughtful caching in order to be able to handle like things like try like going forward in a sequence for a while and then deciding it was a bad idea and going backwards the way a chess engine might try out a play uh determine it was a bad idea and then roll back up and try out a different play so for both parallel sampling which is like just generate 100 responses to the user instead of just one um and for like the kind that I just described where you move forward and perhaps move back the you the results in the VM paper with page attention they report huge memory savings especially in the beam search case and this will make models smarter for the same cost uh and so there's a lot of opportunity there um it will also probably make make them more deterministic which is something people who are more opsy might care about um there's other ways to speed up LMS by porting from the mdle layer speculative execution is a famous Infamous way to speed up processors by when you hit an instruction that will cause you to branch and go down different code paths you can just kind of like guess which one you think is going to happen next and then just start doing that in hopes that by the time you figure out what the result of that Branch instruction was that maybe triggered a cache miss or something um you are already if you were right then you're already like pretty far forward and you have mitigated a bottleneck the same idea has been directly ported to language models with speculative decoding where you use a small model to generate a bunch of potential next tokens and then you pass those tokens into the large model and in fact the small model could be just like aex uh can be like a deterministic rule uh you pass that into the large language model this collection of of tokens and it scores them in parallel Transformers are very fast when you give them a bunch of tokens that have already been made but they're very slow when they need to make new tokens themselves uh this is like because they were designed to be easy to train not easy to run an inference um so the at in the end the small model was like missed out and said this the quick brown fox jumps into the something rather than jumps over a like lazy brown dog uh but the large model got that right in the and so we can keep Fox jump because that's what the large model thought was a good thing to say um but we and we can throw out into the so we generated two tokens and the cost of this is roughly the same as one single forward pass um just due to the ability to parallelize the work uh and so you've gotten two tokens for the price of one uh and so you can actually pretty frequently realize this like 2 to 3x improvement with a if you use especially like a stack of draft models a regx draft dra in for a Bert that drafts for a llama for example uh and the the inspiration for this came directly from this uh speculative execution idea so those are both efficiency gains and I think a lot of people especially the people who are more researchy more excited about things like new computers are like less interested in those efficiency gains but I think they matter they're incredibly important um for the same reason that efficiency gains in the construction of integrated circuits are important you don't necessarily need to be able to know how to use extreme UV lithography yourself uh in order to recognize the benefits that it brings so in 1965 Gordon Moore noticed that the cost of integrated circuits was going down and the density was going up so he looked at a couple of graphs and did like a kind of viby like how you know where is this price going to go um and we can see the same thing occurring with language model capabilities so in the interest of time I won't go in detail into this chart but suffice it to say that for like certain capabilities you can expect the price to exponentially decrease and for um a certain price you can expect the capabilities to rapidly increase this is a critical long-term Trend so plan for it all right so the more speculative thing past this these efficiency improvements ported from the metal layer is can we also Port conceptual ideas from the metal layer and end up with an llm operating system operating systems are there to operate the hardware of the machine the sensors the internal components and effectors mgpt is a pattern that moves towards the llos by adding interrupts and virtual memory and maybe if we take those things we go multimodal we add some peripherals in a network like we're there at the L OS um the at the operating system the primary thing that it does according to tanon Balman Bose is that it takes the ugly interface required to operate hardware and turns it into a beautiful interface that an application program can use it also handles the map between the like realtime world that Hardware must live in and the clock-based world that processors live in mgbt does something similar for llms uh so it adds the ability it it places the model in a position of waiting for interrupts waiting for information from the outside world it also virtualizes the context it makes it possible for the language model to change around what information is in its own prompt so that it can and store that information elsewhere and retrieve it later uh so that gives us the um like both a sort of like local like Ram feeling kind of memory and then also an external store that's a bit more like a file system um it's able to store a lot more information uh so Andre kppi and others have sort of been like oh what if we kind of combine these patterns together if we can you know if we can get information that's stored for a long time that's kind of like a file system um if we have models that know how to browse the internet that's kind of like our ethernet connection our Network stack um if we have multimodal models then we have a connection to peripheral devices to the screen to sounds and then if we have code generation and code interpreters then we can interface with a classical computer and so we have something that looks maybe like this diagram from tanon bamman Bose but extended up a layer where now on top of the beautiful interface of the system calls uh we have a new layer the llm that creates a natural interface that application that happens at the application layer down to the rest of the system where humans can say things like um please order me that dress that I mentioned uh an hour ago uh so are we you is that it is the diagram are we ready to ship um raising 10 million at A1 billion valuation maybe um but we're missing a few things um we're missing isolation uh from llms uh we're missing open architectures and Open Standards in an important way um and work needs to be done on both of these things uh the trustless coordination of resources the ability to run multiple programs on one machine or for multiple users to run on one machine is enabled by the isolation and security features from the hardware all the way up uh LS resist this with Fierce uh Fierce fiercely uh gpts leak their internal content and their prompts um I got a model to leak its prompt by telling it I was needed to its prompt to help orphans find jobs um and if you use browse mode and you are tricked into going to the wrong website you can uh like uh leak information and um uh that is not great um so more work needs to be done on that security layer and Simon Willison has been kind of like raising the alarm about that and collecting those examples um open architectures and standards mostly one in the PC World the things like the Xerox star from xer Rock Park uh beautiful machine but you couldn't write your own programs for it um whereas the IBM PC and its clone ability uh ended up creating a very dominant system even on the basis of kind of like a weak chip uh so can a similar process happen here um I think it's possible but it requires coordination and thought around incentives so the open source initiative um which sort of helped crystallize this definition of Open Source in the late 90s is a attempting to do the same for AI we need this badly uh SAA nadela just called a model an open source release earlier today even though its model does not fit the open source initiative's definition or mine of openness um there's also the AI engineer Foundation which is starting to work on getting people together on protocols and standards uh so that we can have a shared way of interoperability between our models um so stable versions of this are going to require thought around incentives for that kind of collaboration uh just as there were economic models and incentive models for open source that have enabled it to be successful so to recap we have a new kind of computer hopefully I've convinced you with the Transformer llm based on the differentiable natural language processor um it's still in its demo era but that's where we all start um we've also seen this wave of efficiency improvements to language models that's been ported from the the metal layer of classical computers um maybe we're starting to Port Concepts into an llm operating system that will enable an natural interface um but we're missing a few things so let's go and work on it um I'm in the interest of time I'll skip my plug for computational cognitive science ask me about it if you want later um there's good stuff there uh all right thank you for listening and I will see you at the Q&A [Applause] session