Rust AI & Data Meetup: Shehab Amin Interview
Recording: Rust AI & Data Meetup: Shehab Amin Interview
Hello everybody. I'm Alexi Kraber, the founder and organizer of this new Rust AI and data meetup. We are on location at AWS builder loft and we open it up with a talk by Shihabamin the co-founder and CEO of leg sale and in full disclosure I wear many hats I'm very happy to be the head of assistance at lake sales since May and uh I'm very excited about sales so first question is what is sale the open source project and what is like sale the company behind it >> yeah great question so sale is an open-source distributed multimodal lakehouse comput It unifies data and AI workloads and it's built in Rust as a drop in replacement for Apache Spark. Fundamentally uh you know we stumbled upon a problem where we realize that the legacy engines that run big data can't run AI. So what if we can make the data engine AI capable? What if we can turn existing data pipelines into AI pipelines? So that's the question that sales answers. And then Lake Sale is the best place in the world to run sale in production, >> right? And so uh Spark is not just you know an engine. there is a whole ecosystem around it. So uh can talk a little bit about what uh sale ecosystem implements in addition to spark
>> Yeah. So sale implements the spark connect protocol. So you don't have to change a single line of your spark code except for the host URL and you can uh switch to the sale engine. uh you know under the hood we use Apache Arrow, Apache data fusion and the whole rest ecosystem uh in order to unify batch processing, stream processing, comput inensive AI workloads as well as ad hoc workloads uh with two deployment modes in single node and distributed >> and you also are able to support various formats [snorts] like delta and iceberg >> right so we rebuilt Delta Lake and Apache iceberg completely from scratch and rust and so that's integrated all the way you know from the logical planning to the physical planning. >> Uh I I really like how uh how we talk about engine truth and engine contract. Can you elaborate a little bit on that? >> Yeah, so when we first looked at Lakehouse, you know, uh at the lakehouse ecosystem uh and we started with using a few libraries out there and we didn't fully appreciate how deep lakehouse integrations go. Mhm. >> Uh although the deeper we went in, we realized that lakehouse formats such as Delta Lake and iceberg, they're really engine contracts, right? Uh it's a contract that deeply integrates all the way from you know logical planning to physical execution to you know the catalog layer >> uh you know to part you know partition you know you know file partitions and pruning and all that good stuff
Uh and so we realized in order to have a you know order to have a productionready data engine you have to rebuild the ecosystem as well. And the ecosystem has historically been in you know Java and Scola and so we we put a lot of effort in rebuilding that ecosystem in Rust as well. >> So you know I kind of was on hiatus from this whole thing. I started the very first Spark meetup with mate, invited him to do it at SF Skulla and then did the first Spark meetup and uh kind of went into different startup world direction and I came back. I kind of find this whole like gigantic blobs of Java in all places and I thought they're not really necessary like why do you think they appeared and how can we get rid of them? >> Yeah, you know, Spark uh was created 2009. You know, a programming language like Rust didn't exist in 2009. So in 2009, JVM was really the best thing to use, right? Uh Rust V1 beta came out in 2015, and I would say that Rust wasn't really production ready for enterprises to adopt until maybe, you know, 2022, 2023. And so, you know, there there's a whole there's a whole host of ecosystems and tools and and engines in, you know, on the JVM that we uh you know, that's going to take some time to rebuild
And that's what we've been doing for the last several years. Uh, and it's, you know, it's a really exciting time to be building in in Rust right now. We're seeing so many, you know, so many revolutionary and so many impressive companies out there. By the way, the meet up today is the ground review of the Rust ecosystem in the Bay Area and beyond. So, there will be at least eight talks and at least eight videos coming out from this. So, check it out on our channel and our URL is ras.ai. So uh let's kind of kind of since we talk about history and the arc of software engineering get back a little bit and tell us a bit of your personal story trajectory. How do you get to do this? What kind of made you choose Rust? What kind of made you take on rewrite of uh Spark in Rust before it was an AI thing to do, right? You did it by hand
This will be probably the biggest large system handbuilt before AI. >> Yeah, that's a great question. So me and my co-founder Haron, we spent our entire careers in uh you know big data and AI infrastructure. We've built, you know, real-time AI search systems and real-time personalization systems at scale. And the the interesting thing is when you're building, you know, real-time AI or personalization or embeddings or physical AI or whatever it is or model training, >> you're very often using an engine like Apache Spark to do all your data processing. But the second you want to you know calculate an embedding or or do some feature engineering that that has to live somewhere else >> and but you know fundamentally those are data processing operations and so what kept us up at night is why does that have to live somewhere else why can't we bring that to the data engine >> and after some digging in you know the answer was was pretty simple the JVM doesn't have great interoperability with other programming languages notably Python if you're doing anything in AI you're very likely using Python and So if you want to run Python on Spark, Spark runs that out of process >> and you do a lot of interprocess communication and there's a lot of serialization deserialization. So you can imagine how if you don't have native Python support, you know, you you can't support a ton of things, but also your performance is is not going to be up to par. >> Yes
>> Uh so we rebuilt sale with Python in process and we're able to run Python at its native engine speed. >> Yes. which you know I've realized it's a major advantage and it's very difficult to do on JVM. So I keep crossing the boundaries and it's uh it's wasting it's wasting time and then some like basically the slowest element will slow you down no matter what else you speed up. you can put data fusion underneath JVM still not going to give you much speed up right because you're stuck you're going to be stuck in this uh and talk a little bit about the startup time and single node operation how it's important for the workflows >> yeah so engine starts in under a second so it's great for ad hoc workloads it's uh you know on a on our distributed runtime it's stateless so we have stateless elastic workers that scale to zero and we use object store for everything >> uh so object store for checkpointing for caching uh and for even for shuffles you know we have two shuffle modes uh pipeline shuffle which is um you know streaming memory shuffle and uh blocking shuffle which is object storage based shuffle uh influenced by ray's paper on exo shuffle >> right and uh because and so basically users can just install uh sale as a python package >> right >> right and there is a pispark client which does not require server which is a standard pispark right so this is how I first tried it, I got by Spark client and I got sale and I just launched it, right? And suddenly like the Spark ASKI logo comes up and there is no JVM anywhere. That was the aha moment for me, right? Because when I did this before, right, it's slowly it was 15 years ago slowly the ASKI logo creeped up and there was JVM somewhere. So, so I mean this is amazing. So kind of going to the future
Uh I mean there is a lot of technical details are shared obviously we cannot like fit it all in the interview or even in the talk but you guys can check it out at legl.com and on this blog we'll put all the resources but uh generally speaking uh if you are building an AI company right now right or you're coming into the enterprise and you need something like your lake house is not really ready for the AI why should an enterprise use like sale to upgrade their lakehouse and why Should a startup pick sale and like sale platform to basically the ground truth uh like house? >> Yeah, it's a great question. So, Apache Spark is battle tested. 90% of the Fortune 500 use it. If you're an enterprise, you want to operationalize AI in place. You want to turn your data pipelines into AI pipelines. It's a no-brainer. You don't have to stitch together all these disparate engines and create a Frankenstein architecture to, you know, to just create some, you know, basic AI uh functionality. You can operate initialize your AI in place
If you're a startup uh you know you and you want to think about the future right sale has a really cheap switch from vertical to horizontal scaling right and so you can use single node out of the box with pip install pi sale >> and you can run production AI workloads you know in a handful of seconds >> uh but that same engine is going to support you when you scale and you need that you know that terabyte and pabyte scale and and so you won't have this technical debt right and that's really important for startups too >> yes and the agents can write much faster, >> right? And so we have uh you know we have an MCP server and we also have a oneshot uh execution script. So sales spark run it's it's built for agents and it will uh start up a sales server, execute a a pi a python script >> and then return the results and then shut down the sales server all in one shot. >> Yes. Which is not easy to do with JVM, right? They all say about warm warm time, right? Warm time, >> right? A spark cluster is is known to sometimes take up to four minutes to start up, >> right? So uh and finally since we're here at the first community meet up for this new uh ecosystem uh how would you like uh community to engage? Where do you want community to help and what advantage can community derive from engaging with the open source ecosystem of sale? >> Yeah, great question. Uh so we welcome community contributions. We love when the community comes and you know tells us their problems, creates uh you know feature requests or even contributes. uh we're we're fully in support of that. We're also building out an extension ecosystem
So if you want to extend sale like you've extended Spark in the past, we would love to see uh you know we would love to see that work out in the open source. So that's a great way for the community and engage uh you know beyond coming to the events and and joining our online communities. >> Great. And I just wanted to add that you know as a head of ecosystems I'm very keen on fostering that extension ecosystem. We have a discussion thread in our repository. So go check it out and we're basically uh brainstorming various architectures. One of the partnershots with Sedona as an example you can basically put in uh very interesting kind of extensions for all kinds and we're actively interested in making happen. So if you want to build on sale with sale it's MIT license, right? >> Uh it's patch 2.0
>> It's Apache 2.0. >> So you can do that, right? Like there is no strings attached. You can use it, you can extend it and you can build on it. And we are welcoming everybody to to do that with us. Uh and again all the resources, all the talks we are mentioning here, all the slides will be linked from ras.ai and the sale documentations on lakesale.com. Thank you very much and we hope to see you in the community. Thanks. >> Thank you.