Transcript: DevReal: Michael Goin Interview

hello everybody I'm Alexi krau the founder and organizer of beera AI the longest running deepest technical biggest AI metup in the world running in the Bay Area continuously since 2015 live on location AWS ji LOF and here with us we have Michael going who is the engineering lead at neural magic uh and uh they are the maintainers of VM uh welcome and please tell us what VM is VM is the is the most used open-source inference server uh around the world it's uh loved for its you know uh support of many models inference optimizations uh quantization methods uh and of course constant research that gets cont contributed into it and uh why is it so widely used and why is it so Loft like it has a fiber Comm on it yeah I mean we have you know sort of many companies and many academic institutions that build on to it uh you know it's it's a really good thing when you see uh you know dozens of papers every year based off of VM or comparing to VM uh you know uh against some new method that they came off of uh which is usually made as a fork a VM with some new idea put in and so this is allowed for a lot of Rapid development and Rapid integration because these you know promising research results can be quickly validated reproduced and integrated in um benefiting everybody and you're gu based out of Boston uh yes yeah we're originally an MIT startup MIT start I wanted to ask you this uh so MIT and Sil Valley do you see any cultural differences is being kind of a little bit removed from Sil and well does it help you to focus on science I think uh I think yeah there's definitely a lot less talk about tech uh over there and uh I think you know I I haven't been over on the west coast too much but uh I know from our perspective it's made it pretty easy to uh you know hire and find people that are very grounded in uh high performance Computing and uh you know math software libraries uh which has been really good for us as we're really focused on performance um you know these these models are are mostly Matrix multiplications so uh this is really our Focus as a inference performance company is it okay if I reveal the secret of AI to a go ahead so oi is just Matrix multiplications all the way down and matrices need numbers so first we can basically convert reality to numbers put them in matrices and then we run a bunch of multiplications yeah through tensor course through tens right so this is all you need to know about yeah uh and but this is a complex engineering problem right so you mentioned uh some of the challenges what are the kind of hardest problems VM solves actually one of the hardest problems we have is I would say CPU overhead uh particularly because a lot of VM is surprisingly enough written in Python M even though it's you know a very performant Library we choose to continue writing the majority of it uh the vast majority of it in Python and in pytorch to make it approachable and accessible for new people uh of course this has required us to you know take advantage of uh sort of Cutting Edge uh technology you know probably from from pytorch you're familiar with torch compile this has been a great uh you know uh advancement that's allowed us to sort of get back into traditional graph compilation uh and and try to you know uh get some larger graphs uh to to prevent the amount of CPU GPU communication we have to make but also it's required us to make sort of uh you know relatively complex multi-process uh you know python architecture with interprocess communication proper asynchronicity um to keep these you know uh big very fast gpus fed uh you know this is essentially the goal we just want to keep these things at 100% utilization and anything we're not doing on the GPU needs to happen while the GPU is running not while the GP is waiting uh so you know these are some of the biggest concerns uh that that that we have to do to keep the project uh successful as all of the new inference optimizations and methods come in through research and through uh uh you know Enterprise and uh uh I mentioned that your title is engineering lead and so uh building V involves a lot of engineering can you talk a little bit about how this uh discipline of engineering is software engineering uh knowledge is helping you scale V yeah I mean uh I have maybe kind of a vague title uh because I I work on a lot of vague things I think uh to to to be in the in the to be really successful in the field of performance engineering uh I I believe you have to go down to the assembly level the kernel level uh when you really need to make a really really tight Loop uh but then also go all the way up to the the python the machine learning level uh in order to you know understand and connect to the applications that these engines are are ending up just being used for uh you know ultimately the worst thing you can do in performance engineering is get the wrong Benchmark it's very easy to delude yourself that you are performant if you create some micro Benchmark and you're like oh I did a great job I'm at 100% utilization but then you get some real world data and it's uh it's very heterogeneous you have some very uh you know long prompts coming in or or lots of images all at once and uh you have maybe uh you know a bunch of small ones piling up um you really need to be sort of uh antagonistic to yourself and uh play Devil's Advocate to uh you know understand all of the workloads uh and and how you can optimize for them and also what things are worth optimizing for and not because you still want to keep the software uh maintainable and and readable and accessible for new people so uh you know it it really requires I think not being afraid to understand how computers work uh but also not being afraid of the applications that people are building on top of the computers uh which I think has been a a problem since computers have come around makes sense this is great to hear and I come from you know strong typed programming background and at first you know I was a bit skeptical about python succeeding in this space but it's become more performant and what I really like I like types so I like things like pantic which gives a schema and I'm super excited about moo Chris lner language what what are your thoughts about kind of type Machinery help it by be more performant I that's a really good question I mean So within neurom magic and within my background uh we were heavy users of C++ heavy template meta programming modern features you know we we we love that so we did give up a lot of that going to python but within VM everything is typed we use mypi and and linting to to you know verify this on oursel and it makes it much easier to uh you know crawl across the code base um you know I I I think projects like uh like modular or or mojo as you said uh and and Triton which are these uh you know not these domain specific languages um are you know for for efficient code generation are really interesting uh I I mean we use Triton a lot pytorch uses Triton a lot a lot of the ml Community does as well I think this is uh you know unfortunately kind of the best thing we have at the moment an open source uh I uh while I really love the language and I wish I could use it more it's it just it's kind of too high level and doesn't have enough performance to truly saturate the you know h100s h20s the these really uh you know heavy gpus and I I do think something more like like Mojo that has a sort of progressive uh deepening uh where you can get down to actually defining you know the the width of of registers or the layouts of things in a little bit more careful uh uh pathway um is really interesting but one of the things that we're actually one of the biggest users of for our most performing kernels is a library by Nvidia called cutless M uh which is essentially a code generation it's a it's a template meta programming library in C+ Plus but it's a code generation library for Cuda that allows you to Define these very complex asynchronous pipelined uh Matrix multiplications um in a way that uh is is much easier to do manually than than than in Cuda um so uh we we definitely are proponents of code generation we're just not a little sure uh we we we definitely don't want to write everything in cut lless even though it's the it's the best thing we have for the best performance uh but you know Triton and Mojo are maybe a little bit too high level um where we we definitely want to invest more in Middle grounds uh but you know we've uh We've made compilers before uh within our company so you know we we want to encourage other people to work on it sounds great well thank you so much it's a lot of information and uh we're really looking forward to your talk and welcome to the community we hope to hear more from you guys as you develop your then yeah thank you so much it's a pleasure to contributing to open source