The advent of the open data lake

Over the past decade, the big data ecosystem has matured and evolved from a melting pot of competing projects into a composable ecosystem organized around a few open source standards. The components of databases, distributed or not, have been commoditized as individual parts that anyone can compose into use-case specific engines. Define your constraints and build a query engine that solves your problem. It’s been incredible to see the adoption of key components like Parquet, Arrow, Iceberg, and DataFusion. They provide an interoperability layer that enables using data without creating silos and duplication. In this talk I’ll discuss the impact of the cloud and the advent of the Open Data Lake breaking silos to form the foundation of this ecosystem. As compute and storage can be efficiently decoupled, a common storage layer enables a vibrant ecosystem of on-demand tools specialized to specific use cases that avoids vendor lock-in.