Spark Rebuilt In Rust: Making The Data Engine AI-Native

Shehab Amin of LakeSail presents Sail, a Spark Connect-compatible data engine written entirely in Rust. Sail keeps the familiar Spark SQL and DataFrame interfaces while replacing the JVM-based runtime with an Arrow-native architecture powered by Apache DataFusion, running Python UDFs in-process with shared Arrow memory and integrating natively with Delta Lake, Iceberg, and DuckLake.