DevReal: Zig: Steeve Morin

Well, thank you all for coming. I feel super humble. Um Yeah, so probably So, to be clear, I have zero slide. I'd like it for that to be an exchange, but before that, you know, hopefully I will talk to you about why we built ZML the way we did, you know, why was the first thesis, you know, and how it went, the problems we encountered, etc. But, please, you know, feel free to interrupt me. Uh I, you know, we would like to if you have questions to answer them. Um and we'd love for this to be a conversation. So, the first, you know, problem that like the fundamental, you know, problem that I wanted from a technical perspective

Also, mind you, I've been doing a lot of, you know, meetings, so I'll try and be more technical, but if I'm too technical, please, you know, pull me back. We can go wherever, you know, but I would assume you guys want a bit of technicality. Um so, the idea was to essentially build at the very bottom build the framework DI framework that I wanted to use uh as, you know, next operator of a pretty big, you know, I would say infrastructure uh in terms of size, we had about, you know, 14 million requests per second. We're the second spender on Google Cloud in Europe for like 3 years uh with a discount, right? So, that's counting the discount. Uh 80% Um We had a few good shots. Um and ultimately when it came time for me to not at that company, but, you know, after that company was killed um by Snap you know, and I will go on record. Um Essentially, I decided to you know, product an AI model They don't really They don't really, you know, uh found what I was going like what I wanted in terms of the tools, the uh the observability, the robustness, etc. And it felt really duct-taped

Um And the first, I would say, two problems I encountered was that either I would essentially pay money to open AI or I would pay money to Nvidia. And there was nothing, you know, fundamental, from my opinion that would justify that. There were a lot of circumstantial reasons, but not really fundamental reason. There were other chips. There were, you know, Llama just came out, so that was a year and a half ago with Llama just came out. Um So, essentially eventually, I set out to build the framework I would have want wanted to use and ultimately also the products I would want to use to run these models. So the first thing that I essentially did that I essentially didn't find on the market was pick a side with between inference and training. Uh You could like think of them as one being research, the other being production

Um they have very different incentives and everything I would find essentially was trying to do both, but it ultimately it was just training bent, you know, a bit to do inference. So there wasn't the right tools, there wasn't the right, you know, philosophy. Everything was made for fast iteration, but not for robust, you know, production and high performance and where everything matters. So, the first thing I did was essentially, well inference is a universal problem. No matter how sophisticated you are in that field, you will touch it one way or another if you decide to do uh AI. And not only LLMs, mind you, all of AI. Um And if you actually pick a side, there's a lot of things that change in the way you do things. What you want ultimately is using, you know, if you can, I used to, you know, operate a Python production for 10 years

I have the scars to show for it. Um I'm sure, you know, some of you have too. I love Python, but yeah. Too many sleeps. Too many sleepless nights. Um but yeah, I wanted like to have a language that I could integrate into anything that was fast, that was typed, that was compiled. And also, I wanted my models to be also compiled, and I wanted them to be compiled onto any chip. It didn't really matter whether it was Nvidia, whether it was AMD, whether it was TPU, whether it was whatever uh new chip, you know, AWS and so on

And, you know, choosing picking a side, choosing, you know, essentially allowed me to do away with the bad abstractions. What felt for me at least bad abstractions. So, for instance, one thing you do not want is what's called eager execution, meaning that, you know, every operations runs after the other sequentially. You want to compile model. You want operator fusion. You want, you know, kernel generation and so on. Uh there's all of it. Uh so, ah there we go

All right. And also, I wanted it to be zero compromise. Um if I would, you know, change, you know, from let's say AMD to Nvidia or vice versa rather, uh I didn't want it like to have a first-class experience and a second-class experience. Um so, the cost of the cost of change had to be, you know, not cheap, but actually zero without compromise. Um and the last thing I wanted to do was to solve all the little things, you know, the little pains you get on the way to success, right? From the cross-compiling, to the packaging, to the trimming, to all of little these little, you know, fire ants that bite you and ultimately win, right? So, this is what, you know, I set out to build. That was a year ago, mind you. Um and now a year later, uh we've done it. So, we just open-sourced, you know, the the framework

And I guess the number one question we get asked is why Zig. Um and actually, so at at at the very beginning, the answer was, so from a technical perspective, there were there were not that many languages we could have used. Um it could have been C++ or it could have been Rust. The reason being that integration was a big factor at play. We wanted it to be integratable, cuz it's not about essentially floats over a socket. So, if you want, let's say, you're doing like image processing, you know, you cannot send bitmap over uh the network. It's too uh ineffective. So, it had to be you know, had to be integrated

And also, without I would say compromise, so that ruled out Go because of the scheduler, etc. We didn't go with C++ cuz it's C++ and it's it's a it's a tough sell, you know, in an ecosystem that does Python. Um Rust would have So, it it would have worked, right? We could have uh from a technical perspective, everything would have worked, but at least from what I say product perspective, um C++ didn't make sense, and Rust, the onboarding was way too high essentially. There was a big ask for the people in that community to say, "Hey, you know, you're going to fight the borrow checker uh on your way to um to your deployment." So, this is, you know, I'm trolling a bit, but not that much. Um and enter Zig at that time, which, you know, essentially got critical mass. Uh so, that was the escape hatch. We had to go like syntax uh a go like syntax that is, you know, nicer to use than C, very high performance, and also there's this thing called compile time which essentially allows us to write what feels very high-level code, but it turns out compiles to very low-level, very optimized code. Um some may say that, you know, there was a discussion on Twitter recently that it doesn't really matter because in most of the time is spent on the GPU anyways

So, Python is here to do the glue. This is I don't find that argument to be very true because on, you know, model deployment, you don't have only the model code. You have, you know, the integration code. So, getting the data from outside, you have the normalizing code. Uh and you the model code. And, you know, you have to be fast at, you know, essentially ingesting data normalizing data running it through the model and out. So, these little things add up, right? And they add up very quickly in production, of course. Um so, this is what we set out to build

Uh thankfully, Zig had achieved uh critical mass last year thanks to the release of Bun. Uh so, that was all the signal I needed. Uh and so, we went with it. Uh on the journey, we found so many great things about the language, but um ultimately uh we're very happy about the things we could have done. We we're we are doing uh especially on the model code uh thanks to the language. The last piece of the puzzle was how to essentially glo- glue all this together. Um from I would say, you know, the little things as in, you know, I'm on my Mac, how do I generate, let's say, a Docker image uh for my you know, Intel Linux, you know, with CUDA inside and so on. So, all these little things that creep up essentially

Uh and this is, you know, why we went of the build system of the whole thing to be based on. Um, so this is how we do the, you know, the the runtime sandboxing. This is how we achieve the zero, I would say, zero uh, dollar code uh, chip change, if you will. Um, and so we're very happy with our choices. And I guess probably the last part is achieving essentially zero compromise performance on chips. Uh, some we'll go a bit technical on that, but the idea is that the way you achieve that is by essentially finding the right balance between expressivity, expressibility, whatever I don't know what what what would you say. Um, and uh, interpretation. So, what I mean by that is that you want to have a way to to talk to these chips that is low level enough for you to express what you want, but high level enough for the uh, the compiler or the chip, whatever it is, to essentially do do the optimization, do what they do best

So, that was the whole case behind the creation of of a framework called MLIR, which doesn't stand for machine learning, it stands for multi-level IR. Um, MLIR is essentially a uh, a bytecode manipulation uh, framework. The important thing is that you have dialects, which essentially express semantic operations, and you have transformation between these dialects. So, you can go lower, you can go higher. It's uh, it's very loose, right? So, some win, some lose. Um, and for that, we use that essentially the way to emit IR to the chips, and the dialect we chose was the one that was used by Google to run their TPUs. Uh, so it was battle-tested in terms of cuz semantics is in this case is all that matters. Um, if you have the wrong semantics, you don't get the performance, you get you get degradation, you get abstraction leakage, you get all these things

So, the semantics are really the core of of of the program programmability of of this. And it turns out that in 2022, all the vendors rallied behind Google, well, not behind but with Google to um essentially around the consortium that's called OpenXLA, uh which has lots of things, but one of them is the interface to standardized interface to a compiler and a standard dialect to talk to this compiler. So, this is what we use, you know, on at the lower level to emit IR for the chip, talk to a compiler, sandbox it, sandbox the runtime, cross-compile it, package it, and then finally run it. And, you know, in the near future, serve it, too. So, that's roughly the gist of the of the whole thing. So, now it's now working, it's open source. You can just go and download it and try it for yourself. I have a few demos, of course

Um but mind you, right now it's it's a framework. Uh you have to write the code in Zig, too. So, we So, we did that by choice. Uh we have tools internally to facilitate the imports of PyTorch or whatever it is or ONNX and so on, but we just want uh the people who are actually crazy enough to, you know, dive in. We do think that at the medium at least in the medium term, it's a lot lot cheaper to actually write the model in Zig or in GML, rather. Cuz we're talking like 200 lines of code. Uh because it's actually a lot shorter than writing PyTorch. There's less, you know, raw lines

There are about half of the lines to write GML code than PyTorch code. So, it's a lot clearer, there's much less maintenance. Um and so, yeah, a few people have already announced they would do it in prod. Can you explain a little more why it would be shorter in GML than PyTorch? Yeah. Right. Yeah, so there's a few reasons for that. One is PyTorch models are essentially executables and they're written like the executable. So, you have very little visibility about how the model is organized

I would I mean, you could do it the right way, but a lot of people don't. Uh so, it literally reads like operations in in line. This is the first thing. The second thing is ZML code is organized by modules. So, you see, you know, so let's take uh let me show you maybe easier. Uh Let's take MNIST, you right? So, if you look at MNIST code, this is the model. This is This is basically, you know, digit recognition model. For those familiar, it's very uh it's a very small model

Um but the way that model you see is comp- comp- like So, there's the bigger model, MNIST. Um and then it's, you know, you have two layers that are essentially submodules. And so, we actually exploded it so that it it reads very verbose. Um and so, this is, you know, the code you write. Uh roughly. So, you essentially say, this is my forward function and I want to compile that or like I want to compile that module and its submodules. You get an executable back and you're running it. Now, the reason it's smaller has to has a lot to do with the IR itself

So, and also Zig. Uh we have this thing called uh axis tagging and the idea is that you give semantic semantics to axes in the tensor. So, say for instance, you have an image. uh you want to have you know access you know access the dimension 0 1 2. Let's say you know eight width channel. Um you would have you know you would say, "Well, my image is like h.w.c." And if you do a transformation on the tensor that tag is following. It's like named tensors in PyTorch. Only we actually bend a bit the um uh the Z compiler to enable it as a at a native without strings essentially

So, you know in that case you could say something like you know input you know with tags and then you can give it I don't know let's say two tags, right? Ooh. Let's say in that the example I said b c. Now so now you know I can do I can do input.dim.h, right? And no matter like if I do transpose, if I do whatever, that h will follow. So, it's kind of it's kind of nice, but a lot of the code when you write a model has to do essentially with putting tensors into the right I would say expected shapes and expected you know format that the operation will run. So, what you find is that the operations you're going to do so let's say a matrix multiplication is actually pushing on you semantics. So, and you actually do a lot of work to adhere to these semantics. Because that IR was essentially battle-tested and you know rallied by all the hardware community there's no implicitness. So, there's no for instance matmul

That matmul you'll see is an emulated one. Uh we have dot. And so the way it works is that that semantic that used to be implicit by the operation is now pushed at the tag level. So, what essentially that creates is that you completely remove all that booking that bookkeeping code. The the reshape, the transpose, and so on. Because you essentially said, "Hey, this is how I want my operation to run uh given these, you know, uh axes." And some operations actually will push you, you know, axes tags, you know, too. So, for instance, if we look at uh SDPA, the real meat is here. This is So, this is a very naive SDPA SDPA implementation, right? But, this is it

And what you see is that we, you know, multiply, well, dot product, you know, Q by with K, and we contract on the HD dimension. So, there used to be a lot of, you know, transpose and, you know, whatever 01120 and so on. It's all gone because we just tell it how to, you know, how to ex- do the operation. So, in short, you know, there's a lot of code that just instantly disappears because of that. And the last thing is because we propagate that to the IR level, not only it's shorter, but it's more performing because there's a lot of operations that just don't have to exist anymore. Hey, but sorry, I was going to give you a mic to record questions. So, if you ask a question, you can speak into the microphone. Should I press any buttons or No, it's not this

It's all good. So, I'm just curious, uh when you say performance, did you actually did the same comparison like neck to neck with the PyTorch implementation in East? And We we don't put out numbers because it's very easy to get shut down from the sky if you make, you know, a tiny claim or whatever. So, I can give you, you know, the numbers offline, I would say, but for now we don't put out numbers. But, it's more performant because in PyTorch you execute operations one by one. So, if you do tran- The famous case is transpose transpose, which is a null operation. So, if you're on eager, you will transpose, materialize, do another transpose, materialize, and, you know, you have to copy, you have to do all these things. So, this is an operation you can remove completely, right? So, what a lot of people do essentially is they they show the like they describe the graph and try and do this optimization at the graph level. Uh and only materialize when essentially when you pull the data

Um but this is, you know, the job of a compiler. So, this is why, you know, ultimately it's it goes to uh the good thing is that it goes to MLIR, through MLIR it goes to the chips compiler, which then runs its own optimizations that are very specific depending on the chip and so on. Um the good thing also is that because of MLIR, there's a varying level of abstraction. So, for instance, we could say to the chip, "Hey, I just want to run attention." and not, you know, a dot product, right? Well, actually two dot products and an add and a softmax. Um and we able to do this and push that at the compiler layer thanks to the IR. So, it is quite a bit uh we haven't spent much time on performance, mind you. Right? You look at that implementation with, you know, probably have a few people faint in the ML community uh cuz it's very naive. But, it it works really well because of all the little things, you know, that work that essentially happened before

Uh yeah. This question is related What do you see the relationship between ZML and other industry initiatives like Triton Lang? Like what? Triton Lang. Oh, uh so Triton is very cool, but there's a pretty big problem we see with with Triton. So for those who don't know, Triton is a way to essentially program Nvidia chips without doing CUDA actually without doing C++. Um it's pretty cool. The the main issue with Triton is that it's very cool as I say as a compiler tool, but if you use it as a library, what happened, which is actually the scenario that happened with, you know, the H100, is that you don't write code for CUDA, you write code for the chip. So you get nice performance. It's very easy, you know, it's Python, etc

Uh but now, you know, then H100 comes out and the performance is terrible and everybody, you know, is freaking out, which is was which was the December 2023 uh uh scenario. Um Now mind you on the CUDA side of uh the compiler infrastructure we use, which is called XLA, there is some Triton. Uh you can use Triton. So it will generate these big kernels. I mean it works. It works pretty well, but what works also really well is cuDNN. So cuDNN. So essentially there's two ways to look at the problem

Either you generate big kernels uh that do the forward pass. In our case, it's like, you know, for each layer there's a kernel, roughly, you know. Um in the case of Llama or a transformer. Um What we found is is that there's actually a slight performance penalty as opposed to just using the raw, you know, cuDNN or RockBLAST kernels. In the first instance, in the case of of uh of AMD, there's a 100 times, you know, performance penalty of using Triton. Because there's a lot of missing things here and there. But if you switch, you know, to RockBLAST kernels, it works really well. Because you also it works really well because you encode the whole graph into essentially the the GPU pipeline

So, there's no overhead in, you know, launching kernels. They just link, you know, between themselves. And we actually see that in profiles. There's like, you know, zero time between kernels. Um And also in the case of Triton, it's sometimes also it's not a silver bullet if you have synchronization points. Uh so, if you do like these big current big kernels, it's it's sometimes not easy by hand, I mean. It's sometimes not easy also to do it very parallel, and you see sometimes a lot of sync threads, which actually kill performance. I had a question, but now I actually have two questions

My my my first question is pretty basic, but how would I test this? Is there an established way to test my model? Is it easy or and is it a strong suit of ZML or is it like in the average? So, it's No, it's actually a lot easier, but not from a from where you would expect. Uh so, this is an example. Actually, let me let me load you the uh that Yeah, MNIST is good good as good as any. So, this is the model. Uh so, you know, you met me or whatever. Um But it's not only about that. It's about how do you load the weights, how do you compile, and all of these things you have to do essentially uh before the model is run, right? You talk to the accelerator, what is the runtime of the accelerator, how do you deploy it, etc. etc

So, from a model perspective, it's pretty simple, right? Uh but and and by the way, mind you it all languages. You know, if you look at Python and PyTorch, yes, it's three lines, but there's like half a day of setting it up of setting it up for it to finally load. So, in our case, there's not. So, this is like a small MNIST, you know. So, this is essentially UI code. Um the important part is that here, you know, we open So, we have a torch loader uh that is entirely written in Zig, no Python. So, we open the Python the the PyTorch checkpoint. Um then we populate the shapes

So, the idea is that when you compile, you do not want at least in production, you do not want I would say invisible JITs. You want to be sure, you know, when is, you know, the time you compile the kernels, while the the the programs, and when is the time you actually run them. So, what you do is like we you open the model, you load the shapes. Then you compile the model, and as the model is compiling, you actually transfer the weights on the GPU, on the accelerator. Once that's done, so this is, you know, what you see here. Um then, you know, you wait on the compiling. And then, what you get at the end is an executable, which you just call. So, this is what happens here

So, here I'm just reading, you know, the sample data. So, sample digits, this is a digit recognition. Uh I just read them, and then read them into a tensor. So, from the GP the CPU memory to uh the accelerator memory, so that you recognize, you know, the shape of the tensor, the data type of the tensor. Uh and then I just call the executable. I get a result back which is this one and off to the races. Now, this is only a short part of the story. You know, the so the model code, the weights, you know, loading code, uh the compiling code, but then there's also, you know, runtime, right? And so this is actually a demo I had for you guys

Um So, the idea is that it's not only about, you know, um uh Is it working? It is. It's not only about, you know, just uh having, you know, the model having it compiled, then it's about how do I run it on Nvidia, and how do I make sure that it runs. My machine doesn't have, let's say, CUDA installed. This is the case of the machine on the left which doesn't have which doesn't even have CUDA, only the drivers. And so, what we'll do essentially this is this is a uh Yes. Actually, let me kill it, and I will come back to it. Can you guys see it? No. I'll make it even bigger

Um So, essentially, what we say is So, this is all from source, mind you. Right? So, I pre-compiled so that I you have have to wait. Um So, what I'm saying is I just want to run Llama 3.3.1, and I want to use the CUDA runtime. That's all you need to do. Everything is done, is taken care of. And so what happens is we download CUDA, we repackage CUDA, we strip CUDA actually by 10 times. Uh and then we make sure that the CUDA version you use, the cuDNN, whatever, is, you know, bound to your model. So, there's no CUDA version mismatch and whatnot

That's all you need to do. You run it. And so, it has found uh Actually, let me Ah, well. Let me, you know, so that uh Well, you see there has found the devices. Then it's compiling. So, this is the IR. So this is, you know, we had we had found uh two devices, uh the CPU device, and then the CUDA device, so it selected that one automatically. Uh it has found two GPUs

Then, you know, we load the model, etc. etc. debug code. Um this is the IR we send to the GPU, and then this is the model executing. So what you see essentially is, you know, we don't we we load the weights, we compile the model. This is the prompt, we can choose whatever. Uh It's It's a llama, right? And there you go. Yeah

So I don't know what it's There we go. Um So this is CUDA, right? Uh so nothing fancy, it's CUDA. You guys know, so let's, you know, ramp things up and go on AMD chips. So again, pre-compiled because I don't want to have you have you guys waiting. Actually, you know what? Let me just copy that as is. And just change by ROCm. So this is AMD's runtime, and we run it. There's no ROCm in the machine

It found the It found the GPU, it compiled, and now it's going to run the model. So there's literally zero code change uh on it This is the same binary. I can also take an arc make an archive out of it, copy it over. Everything is sandboxed. There's no provisioning of any sort. The good thing or at least to me the compelling thing is that you can like what I'm showing you is like an example implementation, but ultimately the assembly to how to do that is all, you know, in the sources. You can say, all right, now how do I turn that into um uh you know So, here I'm running. But, let's say how do I turn that into uh an archive

Uh so, just a llama model without the weights. And this is an example of building an archive, a self-deployable archive that will, you know that you can deploy wherever you want. You copy it, you run it, it works uh all of the time. Uh and from that, you know out of that archive, we can actually build a docker layer. Uh and out of that docker layer, we can build a docker container which we can push and cross-compile and everything. Um so, that's the rock and is a bit heavy, so But, there you go. And that includes the model? Not the weights, but the model, yes. Cuz the weights, you want to keep them separate

They're like 140 GB. So, you may not want to keep that into an archive, right? Um But, the important part is that these little things, you know, they add up. For instance, you might have seen that we're compressing in ZSTD instead of gzip. And this is how we turn, you know, 8 minutes of unpacking CUDA into 4 seconds. Because we use a different algorithm. Because we strip, you know, the the unnecessary weight. And so on and so forth. So, we uh tweaked the compression a bit, so it will take a while

But, you get the point. So, you're optimizing for compacting speed. to Yeah. Yeah, cold start. Like cold cold start. Um So, this is this is what I sent to you. Not a different machine in my cluster. Yeah

Cuz the idea is like let's say you're running in the cloud, you know, you want your auto scaling to be pretty react pretty, you know, fast in case of machine goes down or whatever. And if that number, you know, is like let's say under a minute, then there's a few things you can do. First is like you have less overhead of, you know, over provisioning. But also, if you're like if a machine goes out because it takes a minute for another one to come up, you don't care. So, you you actually can run them on spot instead of reserved, right? Or on demand. Uh because they're cheap to to create. Whereas if you're provisioning, I mean, there's a number that comes out over and over and it's like tens of minutes, you know, 15, 20 minutes. It's like if it takes 15 minutes for a machine to come online, you know, you're not running on spot, right? Because if a machine goes out, it's terrible

And this is the same, you know, um And of course, uh actually Let me you know Uh I'm on my Mac. Uh let's see. Um And I can run that same command. Uh Where is it? Here. So, I want to do Where are we? Here we are. And I want to say Actually, this time I'm going to build with a TPU. And actually, I'm going to create an image. So, I'm running from This is on my Mac, right? This is local

But this is actually going to download everything. It's actually compiling cross compiling to Linux. It download in this case the TPU runtime. It could have been CUDA, whatever, it doesn't matter. And at the end, what I'll get is a Docker image that is ready to to deploy. The OTI and everything. Trimmed, you know, so in the case of CUDA, it's like 500 MB instead of like 6 GB. And it packs into 3 GB instead of 26 GB

So, all of these things compound. Um But mind you, there's nothing I did. I My machine is completely vanilla. There's nothing uh particular. It's completely isolated. And so, this is an ultimately the promise we do is that it's not on also about the model, it's about all the things that add up on the on that journey. Uh and this is how we achieve true, you know, we feel true cross, you know, true agnosticity of the chip. The code, the runtime, the deployment

Hopefully, it will go, you it will do its thing. So, that it's re I did a build, but I didn't do a Linux build. So, now it's doing one. Yes. How would you stack rank the value propositions? How would you stack rank the value propositions, deployability, portability, speed of inference? Like how how would you stack rank your as your design goals? Um optionality of the compute uh being the first one. Um reduction in engineering cost cuz you go from like spending, you know, at least a day to get something running to well, you've seen it. Um So, you know, I may go on a limb and say, you know, profession- productize and professionalize these tools essentially. So, it's it's a compounding thing, right? Um but what we're starting with it's sure is serving uh because this is where, you know, everybody you know, is hurting and everybody wants us

And also, it's the one that is the easiest to abstract. you know, just build an open AI compatible server and off to the races you go. Um but it's going to be built on that. So, we'll have the same, you know, actually, one thing I didn't do that I'll do after when it's compiled is that you can you don't have to choose, you can choose multiple. And the binary will run anyways. It will try and see, oh, I have a Rock-M plugin, I'll try and, you know, load it. Oh, there's no Rock-M GPU, I'll just go to another one. So, all of these things, you know, will be available into the whole serving stack

Right? But it start with the tech, obviously. And of course, you know, on top of that, then, you know, the serving, you know, in general. Hopefully, it will finish. Uh What What do you mean by the serving in general? So, like there's a pretty big gap between an ML framework and a serving, uh let's say, an ML serving server. Um so, this is that gap. Like observability, um deployment, uh loading the weights from cold storage as opposed to loading from disk because it's cheaper, on-prem orchestration, all of these things. But these are like things we used to deal with, you know, as back-end engineers. They're not new

Like we're reusing This is also a big part of, you know, what we're after is like leveraging the 10 years of of back-end engineering that went, you know, And so, yeah, this is a the image. And now I can I could push it actually, maybe. Let's try. And now we'll Now we'll it will push when it's done. Oh, yeah, so it's pushing. There we go. Um And so, yeah, you you go from essentially your code is to a fully self-deployable image that unpacks, that is trimmed, that unpacks very fast. Uh in the case of CUDA actually, you can try and download it

Um in the case of CUDA uh um or ROCm for instance, uh we like we can do a lot of more things such as removing unnecessary byte code, so you can on top of it over at 40% more, you know, size reduction. Uh so, there's a lot of knobs you can you you can tweak. You don't have to tweak them all. They're same defaults. Um and you know, all of this is from my MacBook. Live. I just wanted to make a comment. Mhm

So, you know, the only other variant of the meet up we did on programming language other than Python was Swift TensorFlow launch. Right. So, the day we launched it with uh Google MLIR team uh so, an article I tweeted 5 hours before we're giving MLIR to all of the information. Wow. open sourced it, right? Then and it was about 2019, I think it was before COVID. So, this is kind of So, the next one we do and it uses MLIR now as established way to There we go. Oh, yeah, yeah. So, it's a unit of different workers

So, like uh I think we see real progress here. Oh, yeah, definitely. I mean uh at least, you know, MLIR is is pretty daunting though when you start cuz it's it's it's loosely coupled. Mhm. So, dialects they compete, they rise, they fall. So, it's hard to essentially do your way um into it. Uh this was also, you know, dialects uh it may seem, you know, it's it's it is a very like semantic is how you get performance. Is how you get true you know, necessity

So, you know, there may be there may come another dialect, you know, the ones that you know, essentially is better than them all. Um but for now this one is pretty good. Stable HLR it's called. It's pretty cool because everybody sprout from it, too. So, all vendors essentially take this as input and then will run their own dialects. Uh sometimes in closed source, you know, TPU is one example. It's completely closed source. There's no um like it's it's literally a shared like an SO file you download

And you know, you do it you do it all. I was wondering about the language thing because it's the big bet, right? And as you know, in a completely different space I'm working on a product that also Right. has to do has to convince people to write new code on an on a new set of APIs and language choice is a big issue. I'm wondering um just in case you run into objections to adopting ZML because of Zig Zig. Um do you have any tools in the toolbox to address that? Like I don't know, can can realistically the smartest models ports my PyTorch codes to ZML? Could you in the future have some sort of a multi-language binding situation? I know right now I know the bet is Yeah, yeah. that you won't need it. But you know, just in case. Well, especially the the porting, the transpiling

Is it realistic? Um So, to try it. I want to play with it and maybe I'm not the target audience because I'm not an expert productizing a model. But I like the idea that okay, I can express this code you know, if as I can experiment especially the multi-model use cases you showed like oh, I could I could do like a cool video related or sound related model. Um maybe start with an expert's model and then play around. That's appealing, but at least because I don't know Zig, it's not that I don't want to learn it. It's just that it's one more thing to learn, you know? Anyway, I'm just curious about that topic in general. is that is that is a great question and on you know after which there are actually multiple answers. Because there's like there's lots of actually all valid different answers depending on where you sit

So, a lot of people they just want to run, you know let's say a llama. Then so they don't care about, you know, how it's written. They just want a server. They want to run it. And for them, you know, we have that ease of, you know use that you just saw. Then we have the people who want to run custom, you know, models, right? Custom llama models, fine tunes or whatever, Mistral and so on. They're based on llama architecture. Again, these ones don't need to touch anything with Zig

Then there are the people who have their custom models, but lots of them lots of these models are essentially assemblies of primitives, transformers and so on. So, for them there's not a lot of code to write in Zig, right? And then you have the hardcore people who have their very custom thing who actually, you know, have to write the whole model code in Zig. So, not assemble parts, but actually, you know, like a transformer layer, actually do the, you know, really deep down operations. Um There are a few observations we made along the way where actually when we built the framework. The first one that you know, we actually did build the framework by porting model models over to Zig ML from PyTorch, right? What we found that we used to actually used to have a leaderboard about it um is that 80% of the time and I also I I would answer, you know, also the perception of things, right? But, you know, in fact um 80% of the time was spent trying to understand what the hell, you know, was written in PyTorch code. Um and this is probably, you know, less than what it really was. Um So, that was our first thing. Uh and the second thing, which was came as a surprise, sort of, was that since the launch the Zig situation, so to speak, has been a complete non-issue

So, I have a few explanation I don't know why that is. I have a few explanations. I think one of them is that they were trying to jump the boat onto on some new technology and Rust was maybe a bit too involving. So, maybe they were waiting for an excuse to diving into Zig, but I had so many people to reaching out to me and say, "I guess we're doing Zig now." And I'm like, "Okay." Uh but um so, this is like, you know, what's happened and you know um Also, yes, we have tools to ease that essentially that porting. We haven't released them yet. The reason there's a few reasons for that. The first one being that we didn't want essentially to cater to everybody. We wanted to cater to the people who would essentially involve themselves

Um but also it was it's very easy to fall into a long tail situation. And this is a story we see over and over again, at least in PyTorch, is that it's very hard to get that that I would say that level of of ease of use uh like very, you know, multi-year effort to get that ease of use because the abstractions don't play well by essentially uh doing, let's say, code transformation like tracing. So, what a lot of people do is tracing. Uh so, Apple does this with Caramel. Um Transformers, you know, a few people that do that to try and it's an excellent PyTorch XLA does that, too. The problem is that you end up with a lot of essentially generated code, which is exactly the PyTorch code, mind you, right? But, it's not less clear. It's not much clear. So, it works as a gateway, but I'm trying to optimize for, I would say, in the mid-term is so that we do the 95%

Uh and we strike that balance between saying, "Hey, you know, PyTorch is not the front end of ZML." There's an import that happens. We do the messy stuff, and then, you know, you do you can do the the nice stuff of the polishing. Uh but, you know, but ultimately, yeah, we have found to be a I have a follow-up question, but Sure. Yeah, go ahead. hug the mic. So, see, what I have uh in addition, and I wonder what you think about this. So, uh so, we started the CAI Alliance, which is open source coalition of, you know, IBM, Met, and 100 other companies. And there is a very vibrant community there for industrial AI, right? Which is very different from Silicon Valley

Yeah. Almost unheard of here. Like, because we only do Uber for pets here. Like, we do Yeah. this kind of stuff. Like, an industrial AI people embed AI in machines, right? In turbines and factories and stuff. Right. So, instead of LLMs, they're thinking about something called SSAs, small specialized agents

So, this is going to be custom, you know, models trained on the domain. Yeah. And embedded in machines. And to me, basically, like you did in C and C++, and Zig is a successor Yeah. of this with the model sex. So, would this stack be useful for this kind of applications, which are not connected to the internet, locally all hosted, Yes. right? In machines. Yes

Uh yeah, so, we talked to another industrial where we talked to people who have a uh similar, I would say, requirements. Uh there's a huge, I would say, a blind spot, you know, in the AI market or at least in the AI discourse, which is late which are latency-bound workloads. Um these are like we found it to be like a very, very big, you know, portion of the market that nobody talks about. Uh so it's not about LLMs also, it's like transformers, etc., but bound by latencies, right? So fraud detection, that's all that stuff. Um so we're a good fit for these, you know, use case. Industrial too. Uh the reason being that we have a set essentially it's not so much that we have low latency, is that we have predictable latency. So it doesn't go all over the place, right? Uh so for like I would say mission-critical, fraud detection being, you know, a big use case, ad routing, and so on

Uh because, you know, there's a compiling step and an execution step, there's no there's much less variation in um in uh in latency. And the second, you know, maybe technical answer to that is why also we went with, you know, a language such as Zig, or it could have been C++ or Rust, is that we could integrate it completely into something else. You know, it's not always a serving situation. It could be integrated into something. Uh as long as you have an accelerator, you want that. Uh we don't do ahead of time compilation, we could, but it's a very big mess at the moment, and there's no real value because you'd rather have one deployable artifact that essentially will use whatever is available dynamically to the to the full extent. Could could be the CPU, let's say, instruction set, if you run on CPU. Um so that way we haven't pushed the ahead of time, and I would say small, you know, deployments, you know, mobile and so on

Um but for industrial, of like if you have a CPU, if you have an accelerator, um, yes, absolutely. Can you maybe elaborate a little bit on the pre- predict- sorry. Can you maybe elaborate a little bit on the predictability? Why is it more predictable? I I'm I'm not sure I understand. So, there's a few reasons for that, but I guess the big the bigger one is because when the model runs, only this happens. There's no caching. So, there's no hidden, I would say, compute that goes on. When you When you is execute a call, we talk to the GPU, we execute a precompiled, you know, uh, kernel, well, program, so to speak. And it's the complete, you know, forward pass that is encoded into that program

So, as long as we'd say, uh, and mind you, uh, because the shapes are static by design, you know, you can actually predict how long it's going to take, uh, on the more or less, through, you know, as variability and so on. But, like, if you take, for instance, JAX or, uh, PyTorch, you know, it In the case of PyTorch, it's fully dynamic. So, you don't know, right? If If somebody adds, you know, changes, whatever, you know, input, then, you know, you run a bigger kernel, right? It's fully dynamic. Uh, in JAX or PyTorch today, there's hidden compilation that goes behind the scenes to try and essentially give you that dynamism, which actually doesn't exist at all, because it doesn't even exist in the model itself. Uh, models are trained with static shapes. So, batches can be dynamic, but you want to push that, you know, a layer above, you know, you want to be able to route to whichever, I would say, version of the model you see fit. So, this is why we have predictable latency. And flat, too

Um, now, this is on the model side, but then also there's the, uh, the rest, you know, ingesting the bytes, normalizing the bytes. Um And for that, this is where Zig shines. Uh especially through its allocators pattern. Because we have we can choose I mean, each allocator we want to use on the journey to that. And it's not like it's I mean, we can make that essentially very predictable through especially through our arena allocators. Come again? Because you control Yeah. Yeah, yeah. But the idea is just, you know, it's not to add complexity

This is to me the important part. It's just to ask the good questions at the at the time where it's cheaper where it's the cheapest to answer. So, allocators are a good example. If you know you're going to run an operation that will allocate memory, which you know, creates a uh uh uncertainty, you know, a certainty in execution time, then, you know, it's better to ask me when I'm about to do it to write that code than me, you know, trying to trace everything and finding that it was this end point that was essentially uh exploding in latency. So, by asking the question at the moment in which you write the code and in this case, which allocator would you like to use? Can use the default allocator? You're good. Not It's like the same thing you would get with C++ whatever. Or you can, you know, since you're asking, well, yeah, maybe I can go a bit fancy. And this is how actually you squeeze a lot more performance uh out of everything

And it's it's very, you know, subtle. Uh but it first of all, it adds up uh in ways that, you know, you don't I I I wouldn't have thought. Um but also it's I mean, it's I mean, it feels you very nice because you're in control and there's like much less surprises, especially in a production environment. That's a great segue to my my part two question because it's about embedding what you're explaining why ZML is great for embedding basically cuz it's it's mini it miniaturizes the implementation of the model I guess. There's no hidden runtime and all of that stuff. So it's perfect for embedding in an industrial platform. Connecting that to the language question, it I'm wondering if that also makes it perfect for embedding in any software product. Right? Like could you could you wrap your ZML model Like you talk about the layers of use, right? Maybe I want to modify or customize the model itself then it's Zig

Cuz I'm I'm interacting with the ZML API itself. But if the model is great, I like it as it is can I pop up one layer and could you give me or could a third party in the ZML ecosystem package any ZML model as a Go library or as a Python package? You know, and I just import that into my application. I embed it. And I'm not using the ZML binaries. I'm just using that that model and it has its own API. And I would use that probably until the moment where I think, oh, I would love to customize that model. And and would it be real like could I mix and match several models that way or would it be so slow to do the about the about round trips that it would just not be worth it? No, no, cuz I mean there are round trips but models are big so you amortize it. But I will let you guys in on the on the secret

This demo we did was exactly that what you described. It was a Go server with ZML as a library. Um So for those who don't know that demo is like we got a llama running a small llama model running, you know, with a few layers on an AMD GPU for a few layers on a Nvidia GPU, a few layers on a Google TPU, and all of that over internet. Shard, yeah, pipeline shard. And which model is it? Llama. So, it's like Go llama? It's like what? Go llama. Yeah, sort of. Llama is a Go library

I would use that. But I'm probably not smart enough to use the ML, but I'm smart enough to use Go llama. No, so the reason we didn't push for it is that essentially the FFI you need to write is very dependent on what you want to do. There's like lots of answers on the journey. So, mind you, Zig has perfect C integration in and out. So, by far the best I've seen. Like, it's close to, if not transparent. So, it's very easy to make a C API also

If you use Basil on top, it's like literally zero, you know, the integration is literally there's no downside to it, except for the FFI layer, which isn't clear exactly how you would want that API to look like. The reason being that on the Zig side, there's a lot of sugar we implement to make that easier. So, the like when you do the compiling, for instance, we would spawn that into a background thread, so that you can do other things while the model is compiling. So, for that, we have So, we have some edible, right, mind you. But, we have an asynchronous runtime that we can bind to some other reactor, you know, if you run into Tokyo, then we can hook into Tokyo and use that, you know. What's Sorry? Tokyo is the Rust asynchronous runtime. Or the Go, you know, for that matter, runtime, too. Cuz you don't want to have, you know, three reactors in your process communicating over thread, you know, that that I mean, you it works, but it's suboptimal

This is why we have this, you know, little thing here, async.zig, which is, you know, uh uh stackful uh asynchronous implementation, so that we can have, you know, essentially stack-full coroutines go like uh on top of Zig, right? Uh so, this is how we do async compile, then do some other things, you know, then wait on that, and so on. So, the reason we haven't pushed for the, you know, we'd say the FFI libraries is that during that demo, we found that it was very dependent on what you were after. And it was more like uh it was very much bound to the user code as opposed to ZML code. Can you explain that? So, for instance, you might want to say, "Hey, you know what? I'll do I have two methods. One is prepare, let's say, and the other one is uh run, right? And run takes a slice of bytes or floats, and then it, you know, so it's very high-level at this point, right? But sometimes, you know, you want to have lower abstractions. For instance, if you're sharding this in the demo, we shard the model into multiple places, so what we compile is not the whole forward pass. It's only the transformer pass, right? So, instead of having, you know, one execution per token, we have, you know, I don't know, 40 executions per token. Because at each, you know, each time, you know, the layers have have done, then we take the data, we send it over the internet, put it back, and reinject it

So, you have varying levels, essentially, of of FFI code you want to write. So, and because it wasn't clear as as essentially I mean, we could do an API, but it wouldn't feel really, you know, it wouldn't feel something, I would say, I wouldn't say usable, but we haven't found yet, you know, when is the balance between or like to put out an API. And just to make sure I understand like the to do it that to do it correctly so that I can I can really fully take advantage of it, say in my Go program. There would be some Go specific work you would have to do. Like you would be basically writing a thin Go framework on top of ZML, but then you have to do that. we we we could uh we wouldn't need to cuz the um the I mean if you put that abstraction, you know, very high then, you know, the C code is, you know, it's just taking, you know, arrays of floats and that's it, right? Maybe a shape and that's it. So, it's very static, but if you you know, as you go down you you you get then yeah, you start to want more a bit more infrastructure on the or sugar, right? In the case of that demo, we had quite a bit of sugar and FFI and bidirectional. And the issue with FFI is that if you haven't done it before, it's very easy to shoot yourself in the foot with respect to ownership of data between, you know, languages and so on

So, it works, mind you. But we want to do it like you know, we want to just want to strike the balance if we go multi uh multi-language. But in the meantime, we found that people would just say, "All right, we'll just do Zig now." I'm sorry. I For me, FFI is uh foreign function interface. Yeah. How does that C Go. Yeah, C Go. So, how is that in your four windowed demo? Where does the FFI get into play? Oh, so Cuz this were four machines, right? Yeah, but this is they were running a GRPC server written in Go

Okay. So, and the Go server was calling into ZML. Come again? The FFI was Yeah, yeah, yeah, yeah. Okay. But there was a also a Zig part, which is to expose the Zig API as a C API. So, that part was, you know, the reason we found was that that API that we would expose it was really bound to how we would use it ultimately. So, because we couldn't we were not really sure how to untangle that thing because it was really depending on the use case. Uh we're trying to be generalist about it, right? Uh this is why we pulled back a bit because we found that there would be a balance we would find and we would have to search then find that just, you know, that was self-evident to everybody

Okay, so basically you're saying that um the FFI API you came up with for the demo was a very, you know, unstable API and Yeah, we need work for that demo. Okay. Yes, exactly. Come again? Oh, SWIG as in, you know, the C++ to whatever? Uh I'm not aware of it, but I I think I so I used to be a pretty big user of SWIG. Uh I have a few I have a few patch um it's a you take it takes C++ code and it generates uh FFI code in Go and a few but I used it most notably in Go. Um I would go on a limb and say that so you mean like to generate SWIG code from C++ code? Yeah. All right, yeah. So, there wouldn't be any point in doing that

Cuz C uh Zig's C story is actually, you know, state of the art. Actually it's actually is way higher than everything else. So, it's actually easier to just create the API in Zig that is C compatible than just having being generated. For C. Oh, oh and oh yeah, yeah, yeah. I mean I mean we we could yeah, I mean we could we could. So, we could use SWIG with your generator. Yeah, but you you you might want to have uh uh over that SWIG code

I used to like use Lip Torne that way, which is a C++ library. Uh I would have Go code on top of it. So, you have, you know, so it simplifies the Cgo situation. Uh but there's a cause, there's a lot of heap allocations involved, also. Uh and you know, the setters and the getters and, you know, the uh whatever uh for C++. I would say that it's it wouldn't be really needed. Uh cuz ultimately the value add wouldn't be amazing because it's a C interface. It's very valuable when you like at least for me it used to be very valuable like for C++ when I would instantiate templates and getters and setters and operators and so on

Um uh but in in the in the SWIG interface um but in that case technically you could have, you know, Zig emit, you know, uh a header and then, you know, give that header to SWIG and it would work, right? Yes. Um but it's we found it to be I mean, it could work, but we ultimately for that demo we just had, you know, a header which we see we see imported in uh uh include C, you know, in uh in Go. Uh and it works fine. And then, you know, we added our sugar on top of it. Um it turned out that our sugar was very dependent on our use case, which is why we pulled back from, you know, pushing forward on the FFI side. Since you're um basing everything off XLA with uh Nvidia and AMD and Google TPUs, have you considered um AWS's Neuron hardware? yeah, yeah. We're working on that. It's actually under under development, okay

Uh do you have any timeline associated? Let's talk after. No, they're my friends. They're they're supporting me. Still, I just wanted to say what's next for the project and that'd be a Yeah, so oh yeah. So, we're working on the LM server, obviously. The serving side, more chips. An active area. We're working on the sharding, too

So, there's, you know, code was merged, I think it was last week. To do sharding. So, essentially split them all up across multiple GPUs. The I would say the opinionated thing we we have is that we we think that you you wouldn't want to use enter, you know, machine interconnect in product in in pod. So, our assumption our base assumption is that you have a fast interconnect inside the machine. So, you're running tensor parallelism inside the machine. So, sharding, etc. And slow or as I say, not infiniband interconnect between machines

So, if you're going between machines, then you might want to run pipeline parallelism. So, just get the activations, transfer them, reinject them. This is the demo you see. So, sharding, yeah. Is is the one that is top on this because this is how you run the 70B's and and stuff. But, it's, you know, it's easier said than done to do it in a way that is performant uh uh uh by avoiding copies over and over again. Because when you run the operations shorted on multiple GPUs, there's a lot of crosstalk. Um and there's some tricks people use to essentially make it look like it works

But it's not color It's not, you know, rainbows. I actually thought of a question. So, uh we have today AI coming to town, which is October 23rd, and I happen to be a judge of the hackathon. The hackathon is on uh 19th of October. So, Intel sponsors it and it's a big deal with a lot of people. Mhm. So, let's say I want to suggest some folks to do a project in ZML. Right? What are the resources they can take to do this, you know, one kind of cool demos people do with voice and video, whatever, using Llama and stuff

So, the first thing they might want to do is go to the documentation website. If Chrome is opening. Uh so, this is how you get started. Uh so, we essentially take you from zero to a working, I would say, matrix multiplication on a So, also So, like you have lots of them, etc. Uh but also we have, you know, examples such as, you know, that one. That one is a basic, you know, matrix multiplication benchmark. That tells you how many flops run, right? So, this is a great way to get started um to see essentially because these models, like examples, they essentially show you the framework. They don't show you models

They show you This is the model. It's a dot product. Uh this is the framework, right? Parsing of this of the the the flags, the compiling, and etc. etc. So, the best way to get started essentially is to take that examples folder, copy it, you know, somewhere else, remove uh models you don't want and just, you know, get hacked. Um, get, you know, hacking, right? And let's say you want to use the community to help you build out ZML. What are the low-hanging fruit, uh, things people can do to contribute? Models is Models are cool. Uh, uh, you know, diffusion we have a somewhat unenough effort of So, we used to have about eight models total, uh, as we were working on the framework, we stopped working on them to concentrate on on polishing

So, we just wanted to have Lama running at all times and in NIST, you know, um, so, there's an effort sometimes that Guillaume of our team is live actually live streams on porting over, uh, um, stable diffusion. But, you know, stable diffusion, multimedia models, uh, we actually used to have a demo of, uh, an AI audio codec uh, running inside of FFmpeg through ZML in the same binary. Um, so, I guess more models, you know, existing models, you know, whatever. Uh, and if they see something they don't like, please tell us in in the API. Uh, there are varying levels of abstraction in the framework between, you know, the basic tensor operations to and then to, you know, IO to that sort of stuff. We have a very opinionated also IO model, too. So, yeah. All right

If there is no other questions, let's thank, uh, Thank you guys.