Rust AI & Data Meetup: Oussama Saoudi & Scott Sandre
Recording: Rust AI & Data Meetup: Oussama Saoudi & Scott Sandre
Um so folks, uh my name is Osama. This is Scott. And uh we're going to be um talking about Delta kernel Rust, all the wonderful things we're doing at Databricks on the open source and through Rust. For starting us off, Scott will uh take us away. >> Cool. Uh thanks everyone. Hope everyone can hear me. I'll try and speak close to the mic
Uh who's heard of Delta Lake? Okay, maybe I didn't need this slide, but that's good to hear. Um so, Delta Lake is an open source uh lakehouse metadata storage format. It's optimized for fast writes, and it's essentially a metadata layer that sits next to your parquet data in the cloud. Um it brings a lot of advantages uh to the what was once just a bunch of files in some object store uh written by Spark back in 2015. So, it brings ACID to your tables, multi-version concurrency control, uh efficient DML like merges and updates with deletion vectors, time travel, data skipping on reads, uh schema evolution, um a bunch of really exciting like features. And what's interesting about Delta is it's rapidly evolving. Um we don't have like a, you know, increasing numbered protocol version. Instead, we have very specific features, and you can have a table and specifically support certain features
And this allows us to rapidly evolve and develop the protocol over time. So, especially with like AI workloads, we're able to add new table features that add, you know, very specific support for like particular AI workloads, like being able to uh add a new column to your table uh that includes just the model weights and alter and just update that single column and things like that. So, because of this, you know, previously, there were a lot of different implementations of Delta. There was ones in Trino and Spark and Flink and DuckDB and Delta RS and then of course through at Databricks. And this wasn't a scalable problem. The goal that we want is to have one kernel, one core Delta abstraction library where you can build features in one place and then have every connector take advantage of it. Um you know, we were we were able to achieve this with a Java and Rust kernel and we were really really proud about that until we still had to duplicate every single table feature. And so what Asama will be talking about today is our goal to basically deprecate the Java kernel and replace it with a thin Java API over Rust which will let us actually build every Delta table feature in truly just one location
We're really excited about it. So Kernel is a is a library that provides APIs for your connectors to read and write your tables. So it starts with table APIs where you can you can do your scans and do your writes. And the goal of Delta is to abstract away the Delta protocol. So internal details like replay and data transforms and data skipping expressions is all abstracted away from your engine. Instead of what we do is we give your engine the opportunity to give us a set of utilities. So you can read your JSON, write your Parquet, do your cloud listing and evaluating your expressions in your engine's own native format. And ultimately here the goal is to integrate once into Kernel and then get every upgrade for free
Uh so far Kernel is doing very well. We're very pleased with the feedback we're getting from the community and from uh companies and connectors that are integrating with Kernel. So DuckDB, ClickHouse and a bunch of people are really pleased that they could just take our user guide, honestly throw some agents at it and Kernel integration is is is pretty pretty quick and easy. Um you know, these are a bunch of some really nice logos of who's of who's using it. Um at the end we'll I'll linking our GitHub and our user guide and we hope that maybe some more people can be using uh kernel after this talk. So, why did we choose Rust? Um frankly, we had open source and internal Databricks demand for a Rust Delta abstraction, right? DuckDB really wanted this. Delta-rs, which has over 30 million downloads a month, really wanted one core Delta abstraction. So, we wanted to build it in Rust
Uh we chose that over C and C++. Um you know, a performance, um for ergonomics, um for correctness. Um one interesting note is that compared to Java kernel, which is a very similar project, we do see about a 2x performance um just on no garbage collection, uh bounded heap size, uh native memory size, and things like that. Um Rust also makes the C interop uh very easy, and Rust has allowed us to have bindings for C, C++, Python, and then now the next major initiative is Java. I'll now be handing off to Usama, who's been leading all of the the Java migration of kernel Rust, which is really exciting. >> Yeah, folks. Um all right. So, we started off with Delta kernel Rust
And the obvious first consumer was something like Delta-rs, where they wanted to optimize Delta scans with Datafusion, but it would have been nice to hand off the this Delta protocol specifics to us, to kernel. And so, the very first place that kernel's uh powering Delta is in this open-source, organically grown library called Delta-rs. They use Datafusion, widely used out here as well, and they provide Python bindings. And they use kernel for what's called the metadata phase, where you figure out what files do I need to read for this version of the Delta table. Now, let's take it a step further. This was within the same language. DuckDB, let's take it out to another language, C, C++. So, here we leverage some FFI bound uh bindings, where you know, DuckDB calls into uh Rust
Mind you, DuckDB is in C++. And it works just like that. We handle the memory management really well across the the language boundary. You can do a select star from a Delta table and it just works. And this works nicely because DuckDB is also a fellow native language. You know, they they know how pointers work. They don't have some pesky runtime managing uh data for them, right? I eat the garbage collector. Okay, so we saw success in Rust
We saw success in fellow native languages. We wanted to shoot up for Java, the JVM. Now, the JVM is really tricky to work with. And this is for a couple of reasons. One of them, you you just have to like cross the boundary. And there's a litany of APIs with different support and different performance characteristics. To name a few, there's um foreign function memory and that's stable as of 23 plus. There's JNI
There's JNR. I think there's also JNA. Well, anyway, there's a lot of APIs to choose from and we had to like figure out just like what is the most performant option here. You also have to work with the garbage collector. And we had the option to Hail Mary and let the garbage collector clean up our Rust objects, but for experience at Databricks tells us that's not very wise. And so we leverage Java's language features and like use auto closable. So that memory management is very um almost ownership based, very familiar to Rust programmers. Um now, comes a little bit of a trickier problem
That's memory. So for those of you who are not familiar with like the data space, often times folks running Java um applications in the data sphere disable memory swap. Usually the operating system will will take your memory and happily run it write it to disk so that you can feel as though you have infinite memory, use as much as you want. But in production systems, that'll slow you down, right? Reading memory causing a read to disk. And so folks disable memory swap. Well, okay, if you disable memory swap, and you got this little rust library eating up a bunch of extra memory, you're at the mercy of Linux's OOM killer, out of memory killer. You just used too many uh too much memory, and the whole process gets killed. So, we have to figure out how to bound this uh this memory
And a couple ways that we did this is by just, you know, let's limit the concurrency, not too many concurrent requests, and let's keep the data phase up in Java land and bring it down to to Rust land. We felt that we could do better. And so, stay tuned for that. Um let's see like how this first implementation of a Java Rust interaction uh looks like. You know, Kernel Rust will go ahead and and figure out the the metadata. What files do you need to read? And then the Java program will see these files, and it'll do the big data reads across Parquet. All right, you know, maybe you'll want to read data one and three. Maybe data two has been deleted uh by the the Delta protocol
And this works nicely because much of the data is over here, you know, in this blue data size uh data level as opposed to the metadata. But this metadata can get pretty darn big. It can get real darn big. Big enough to kill your process with uh Linux. Okay, how do we solve this? Um oh, rather. Uh so so we're we're we're seeing that like we we have this like FFI boundary that we're we have to select across many different um Java implementations for FFI. Uh we're trying to achieve this dream of letting an engine sort of do all the work, right? Remember what Scott was saying was that an engine should be able to do all the work. Well, here what I just showed you was Kernel Rust doing quite a bit of the work
Um and of course like we want to allow this like distributability, this parallelism to to to surface up into uh other languages, not just in DuckDB, but also in in Java. Also, there's asynchronous asynchronous. How will that work? All right. Well, let's get back to the drawing board. And this is an observation that will sort of solve a lot of these problems. Okay, and this observation is that metadata is actually data. Right? This is This was brought up earlier by the Lake Sail folks. They were saying, "Well, hey, well, reading a Delta table is really just reading a bunch of JSON files and parquet files and applying predicates over them, doing skipping so that your query is more efficient, and then reading more data
This is great for database engines. But this this is the observation. One, metadata is data. Reading a Delta table is really just a bunch of operations that are data-like data flows. And SQL engines are really good at this. The other side of this is that engines typically know SQL and they execute a hell of fast. Things like Well, ClickHouse and DuckDB. Spark can go out and distribute for really, really large workloads
Lake Sail might be pretty good at doing this as well. And they also have the infrastructure for it. They know how to parallelize SQL. They know how to distribute it. They know how to run this asynchronously in their own language so that threads aren't blocked. And of course, they know how to do optimizations like filter and projection pushdown. And of course, there's fault tolerance. Okay
Well, we're trying to make a an abstraction for all table formats and we want engines to be able to use them. Well, what if we speak their language? What if we guide engines on how to read tables using logical plans, using SQL? Okay, so the if you think about it this way, Kernel's really just a choreographer. Kernel guides the engine. First, do this and then do that. And then you'll find your answer. It's in that data that you just read. Um and so this is sort of like how the contract evolved where we're looking to evolve the contract of kernel. Where kernel's one of the Rust library, you can use it in Spark, in C++, in C, heck, wherever you want and it just guides you
And all the memory, the really heavyweight operations, all the multi-threading and asynchrony gets to stay in your language. And this is really Rust at its peak where you have a really efficient, small library you can use it anywhere. So previously where something was being done in kernel and you have a bunch of expression evaluation that was done on the Rust side, there was a lot of data that you had to pass to Rust. There was a lot of optimizations that you're leaving over to the Rust side where maybe Spark can do this distributed and maybe in parallel. We now make it something like this where kernel just says, "Hey, go ahead. Here are the expressions you need to execute. Just go ahead and do it." All right, read these Parquets, execute expression 1 2 3, and the engine's free to parallelize across CPUs and so on. And if you're curious, this is what the the plan for reading a Delta metadata looks like
It's pretty, you know, it fits on one slide and this is something that just about any engine can can execute. Um and there's a there's a bit of an asterisk to this this future direction and that's at what Delta does has some interesting semantics. Things like applying deletion vectors. Things like taking a relation, right, of like paths, right, like what you're reading from the Delta table is a set of path. You know, go read file one, file two, file three. Well, you need to be able to take those paths and read all of the contents of file 1 2 3. How do you represent that? Well, right now we're looking at a couple sort of like unique operators that just like bound everything that we're trying to represent for Delta so that you hopefully or an engine will hopefully only have to implement a couple sort of custom operators, things that you anyway needed for Delta. Make that part of our our language, and then we can reuse that as much as we want
Um so, yeah, this is like an active field of development. We've prototyped it with DataFusion. We prototyped it with DuckDB. And we're seeing that these are engines that can that can really execute this and do it fast and parallelize for us. So, not only does the Rust ecosystem benefit Rust, but it benefits everyone around us. Um so, yeah, thanks for coming out to our talk. This is the exciting stuff that's going on in Delta Kernel Rust. So, one, if you're an engine, come talk to us
Two, if you are looking to read or write Delta tables, please use Delta Kernel Rust to um go ahead and do just that. You can find our GitHub right there under delta.io/delta-kernel-rust. You can find our user guide right here. delta.io/kernel/rust. You can download us on crates.io/delta-kernel. Uh we have a weekly sync Thursday, 11:30 a.m. Pacific time. And then, if uh you can't quite catch us there, we're always on Slack at me, Scott, or any of the other um Delta Kernel Rust maintainers
Um and if you're really interested in this in this plan stuff and seeing a demo and more details on that, please come check out um a seminar tomorrow for Delta. We're talking about Unity Catalog and this declarative plans business that we're cooking up. Uh you can find that stream on LinkedIn. Thank you all FOR FOR LISTENING. >> HEY, GREAT TALK. UM ARE THE engines usually calling the Delta Kernel RS library, or are is the library calling them? >> Ooh, okay. So, uh remember how I said that thing about asynchrony? Um we can chat afterwards if if you're really interested in this aspect. But, um currently, it's sort of uh uh a connector
Let's say I have like data fusion as a library, and I just want to write read and write Delta tables. I would call into kernel and pass in this dependency injection style. Uh pass in data fusion, and then kernel will turn around and direct data fusion, go do this, go do that, go do the other. And um so that's how it is uh right now. We're potentially looking to evolve that to achieve a synchrony, not only in Rust, but also remember we're trying to be cross-language. So a synchrony across language boundaries is a very tricky problem. Um so but yeah, currently it's just kernel directs the engine, go do this, go do that, go do the other. >> Super interesting
Thank you.