The advent of the open data lake

The advent of the open data lake | Julien Le Dem, AI By the Bay 2025

Recording: The advent of the open data lake | Julien Le Dem, AI By the Bay 2025

Welcome everyone. Thank you for coming to this talk. I'm going to talk today about the advent of the open data lake. And so I'm Julian. So I'm going to skip this slide because we just heard a lot about me. I'm going to start this talk about um so so start uh first we start from the origin origin of times which is from my timeline um starting with Hadoop uh back in the early 2010s. Um then we'll talk about how the ecosystem has changed a lot since then and now I'll talk about how we've been moving towards composible data system and how this um ecosystem enabled um better composible systems today. So first when we started at the [clears throat] beginning there was Hadoop right so we started Hadoop was in two PL uh parts one is map produce running on the distributed um file system uh and you have those two abstractions and that's how you build distributed uh processing but of course um the processing layer is aware of how the um storage layer is organized and you write a map bridge job by having a mapper that reads in parallel from a distributed data set, does a shuffle, outputs key values, does a shuffle and outputs the result to reducers that then write back locally

Right? So at the time we had this idea of moving compute to storage because transferring data over the network was expensive. Accessing disk locally was a lot faster. So there was this whole notion of moving compute to storage and data was immutable so that it was easy to maintain replication and so on. And at the time um you could describe Hadoop as a great way for looking a needle in a haststack with no plows, right? The idea being that it's a very uh blunt tool and um [clears throat] and it's kind of not very subtle um it just use a lot of um horsepower to make things happen. And at the time there was this famous post by Michael Stonbreaker which is um during award nom winner a few years back and then really didn't like the idea. The first he said that this is not a novel idea. It's not even implemented efficiently. It's ignoring all what happened in databases for the past 30 years and it's not compatible with any of what we've been working on in the database uh world

and he had a point, right? Let's let's say he didn't really like it. So, databases existed. Uh SQL was a thing, right? Has been a thing for a long time. Uh it's nice. It's declarative. It's a standard. It defines consistency constraints. It's a low schema evolution

There's a lot of good things that uh make SQL a good language to support and make things like that instead of map produce. And the way this works, you first um parse the SQL query. there's a semantic um [clears throat] notion of understanding what's a table, what fields they contain. Um then it goes through an optimizer that's going to decide what's the most efficient execution uh plan for this query and to make it from your data. So the way this looks like right you take the query you parse it you get a s u a syntactic tree you use the catalog to know what column is in what table you throw it to the optimizer that knows a few things about the size of tables or things like that and spits out a plan that the executor is going to be able to produce the output of your query and [clears throat] and so if you compare the two you could think on one And um RGBMS could be complex, somewhat inflexible because they're very vertically integrated stack, right? Like everything has to work together, right? The optimizer has to really know all the details of the execution layer to produce a good plan for it, right? So there's a very vertically integrated somewhat inflexible. You cannot really skip some layers. It's all has to work together. When map produce is a lot simpler, it's very flexible

It's very composible, right? You can do anything. You can just you're just running code inside your cluster. So you can do a lot of things and maybe it's a bit too simple but it's so composable that you can even implement SQL with it right and we did so it's kind of this composible snowplows that you pipe together string together and then you produce the output of a query engine right um so in that period there was a flurry of new um SQL and Hadoop uh ecosystem or things and there a lot a few of them endure and they don't really use Hadoop anymore, but they still live in that same ecosystem when we replace uh HDFS uh with blob storage in the cloud. And we used to build those databases from scratch, right? And so when you think of the initial uh release of Hive, you know, you parse the query, you transform it into a chain of map produce job, store the result in the Hadoop fight system, done, it works, right? And you can actually make relatively easily build something that just works but it's not efficient right like and it takes a lot more work to make it efficient right you need to think of okay how are we building an optimizer how do you implement vectorization how do we build a interchange from this distrib distributed system and how do we build a column store a whole bunch of uh components that you need to make an efficient all app system um that works and that returns query in a reasonable time and you don't have to go get coffee while it's running. So if we think about the optimizer, there's two levels. On one end, there's a rulebased things you always want to do like for example moving a filter before a join in your plan, right? It's always better to do the join on fewer rows and so do the filter before. And there are things that are cost based and depend on context. For example, join ordering, right? Like you which tables are small, which tables are big, how we order the join so that things go faster

The vectorized execution came um initially published in uh the monadb or x100 paper that came out um and talked about in so 2005 uh for this one and explain how you can organize the data in vectors in a columnar representation in memory so that your query engine goes a lot faster and this is based on how CPUs have evolved right over the years CPUs don't execute one instruction at the time anymore, right? Instruction gets staggered. So the CPU starts the next instruction before the previous one is finished and so that it can move faster and can have a faster clock speed. So that we can move a lot faster. However, of course, this has a critical flow that if the next instruction depends on what the the result of the previous instruction, you cannot do that and you have to wait a bunch of cycles. For example, if you have if statement or a for loop that um ends or a virtual method call, right? If you have a virtual function and it depends on the type of the object which function to call. This is those are all data dependent branching that you need to wait for the privacy instruction before executing the next. So to avoid this, the pipeline, the processor um tries to predict what next instruction is going to run and therefore if it's right, it's going to finish the instruction right away and go as fast as it can. If it's wrong, it's going to have to rerun, discard what is done and restart from scratch with the actual instruction it was supposed to run

And so it's no worse than if you had waited for the result, right? So that's the general technique that processors do. Of course, if you implement a database with a if statement in a loop, you are going to have to wait um uh you don't know, right? the prediction is going to be wrong a whole bunch of times because and so therefore you want to organize the data in a way that's going to optimize for that and basically that's what vectorized execution does u trying to optimize for having very predictable execution that does always the same thing and get to the maximum speed of the CPU the other characteristic of modern CPUs that benefit from that is processor cache since accessing main memory is a lot slower than accessing the cache organizing data in vectors also makes more efficient use of the cache because you're more focused on processing that data and going really fast on it. You do a lot less of fetching data because it's less intertween in your in your memory, right? It's more efficient. You get the chunk of data that's only the data you're actually using in your cache. So you can this is the other benefits. The other thing you need is efficient interchange to exchange data um between nodes when you do um distributed execution and you need a columnar storage on disk so that you can take advantage of vectorization from reading all the way to inmemory and do fast uh execution on things and so here we go over the next decade that's what happened in the community right um hive turned from the simple thing that was just parsing SQL and turning that into map produce jobs into an actual database, right? And it added all of those things having a cost optim base optimizer push down columnar storage um all those things that made it efficient and then we're done right things uh are good. So the thing is meanwhile things have evolved the ecosystem has changed right we started from this world where um we move uh processing to the compute uh to get more uh fast access to the data and so on and that's what we're starting from but over time things changed so there were some limitations of Hadoop right like we use immutable data there's no transactions um it's tying storage and compute right you move compute to the storage so the things has to run where uh the data is stored which constrains how many machines can run the actual workload um and it's reliant on discrete steps that are persisted on disk. So it creates a lot of latency in the execution of the query but you know it [clears throat] was great right so you don't actually need transactions is the if data is immutable uh that there was a nice thing turns out I don't know how many people knew in the room transactions are useful like so we found out the hard way and you know who who knew you know who knew this guy knew not only stoneer knew he banged us on the head saying what are you all doing right this makes no sense you need transaction Obviously you need all the things

Um what are we all doing? So the other thing um it's coupling storage and compute. I remember like back in the day um we had um so I was working on the data platform team and there was another team in charge of designing the next Hadoop cluster we're going to deploy to do stuff and he came to us and he says can you tell me the ideal ratio between compute and storage on your machines because I need to know how many discs I put on the machine I need to know how many CPUs I mean and make I put I need to know how much memory we have what's the ideal ratio and we're Uh and that's for the next five years, right? Because we'll need to amortize this cluster. We don't know what we're going to do next week. So I don't know like workloads in six months will look very different. Things change all the time. So the fact that you have these fixed uh clusters with fixed machines that you have to make use of uh is very constraining and in the end we always ended up either not having enough room so you need to delete stuff that was useful or not having enough CPU right and then you're underutilizing your cluster. So it's kind of lots of constraints that were problematic. So those were some of the use cases and things constraints we meanwhile what changed right things have changed networks changed the thing that happened is that networks got faster faster than the disks right so the ratio of the speed of getting the data locally from the disk and the ratio of getting the data through the network changed dramatically right and the computers themselves got more cores more memory more disk more parallelism on a single machine so the need for distributed system was less um necessary and that's why you see today um things like DB that are very optimized for big machines with lots of CPUs lots of RAM then can do a lot on a single machines and don't have to worry about the trade-off of distributed system as much um so those things happen and the other thing that h happened is kind of connected right like the cloud became ob ubiquitous and things became on demand, right? The resources were elastic

So, we're not racking servers anymore that we needed to use. Uh we would, you know, if you need a 100 node cluster to just tomorrow for this one thing you do once a year, you could get it. You pay for this one day of the big cluster and then you give it back and you never have to worry about it. And if you're wrong on the sizing, well, scrap it and get a new one, right? you don't have to worry so much about we bought all the servers, we racked them up in the data center and now we need to make sure we use them properly. The other thing of course that happened is decoupled storage because the network got a lot faster. You had things like blob storage like S3, GCS um you know cloud uh blob storage uh that became fast, cheap and decoupled from compute, right? So you can store all your data, turn off all the compute, and you don't have to worry about it anymore. And then when you need compute, you turn it back on, and the data is still there. It's not tied together, right? You don't have those CPUs sitting around doing nothing while they keep the data in the hard disk, right? So the architecture changed like old school

You have storage and compute tied together. And so when you process something that you would move, we would move the compute to storage. And therefore there was big constraint, right? Like if a lot of the compute needed the same data, well, there would be constraints on what resource they'd be using and otherwise it was making inefficient use of the system. Now, if you decouple the two, everything's so much more elastic and you can make much better use of your resources because you you can scale independently storage and compute and so you can create new resources as you need, return them and things are just easier and that um [clears throat] that's what enable this rise of the new cloud first data warehouses, right? So if you look at BigQuery and Snowflake, they both use that same new um architecture which is all the data we store it in blob storage. We compute decouple storage compute you pay per use. It's not that this cluster you buy or you pay a license per number of CPUs and then you tell your analysts or whoever is using it, hey this is what we have. So either there are things you decide not to do or you wait or you know this kind of constraint resource. Now it's not the case anymore

you need something just for one this one time, you just get it, you pay for it, and you're done. You return it. We're fine. But of course, some of the bat sides, they use proprietary format. So once you put your data into it, right, it's only accessible from inside the database, right? If you want to use different tools, you have creation of silos. Uh it's expensive to import or export your data and um a lot of other constraints of that order. And that created new silos because people didn't use just the warehouse right they wanted to do machine learning they wanted to do analysis they wanted to know data transformation and use different padding to do things so we ended up with this spatable and harass uh it's very similar to Jamak's uh slide on the previous presentation of how people exchange uh data is just like you have a lot of brittle pipelines that pull data in and out of things and you cannot just replicate all the data because that would be too expensive. Um, and so you end up with very brittle uh exchanges data and you never have the right data

It's always in the wrong system and it's kind of very painful to manage and in the end wait a minute everybody's storing data in cloud in a blob storage in a columnar format and it's all the same thing and I'm keep moving it from one to the other which really the underlying architecture it's all the same we they everything has been moved to blob storage in a columner format and you're like Wait, how do we move towards a better way of doing things, right? Like towards composible data systems and avoid all those constraints. So first when you think about oh we've talked about database like creating all app systems and like what the various reasons right like so the first thing to think about is the trade-off right like depending on use cases uh the database you would use would be very different do you actually need distributed systems does it fit on one machines do you want to use duct db or do you want to use trino or what are some of the things you can use right uh does it fit in memory or on disk how big is the data what's the latin of injection you care about. Is this something when you're going to like bulk injection? Maybe it's fine if the data shows up one hour after it's been produced. Or do you need real-time injection that really as soon as the data is available um comes is created, it needs to be available in the database uh to do processing. And some other things can help also with trade-off on how costly these things can be by thinking about do we need approximate data structures or sampling appropriate or we do we need exact results. What are we building? So we can actually nowadays right so we started from this flurry of SQL Hadoop on Hadoop system that were all very vertically integrated like a database right they had the query the optimization the execution everything vertically integrated into a much more composible there's series of components that came out of this and that databases have started reusing these all app databases have started reusing right and you can cut the system into a few layers the First layer is kind of the front end. It's the part that parses the query, optimizes the query and the lower layers there's the execution that actually implements you and you know doing joints uh how to process the data efficiently and finally the last layer is the storage how you keep data at rested in between queries right and there a few components that enable connecting all those things together and you see a few icons on this image and I'm going to go over there from open lineage substrate calite arrow um and of In the storage you have park iceberg as well. So first whenever you do data processing in this modern age you need governance you need lineage

So open lineage is a standard way to expose how you uh transform data where data is coming from um how data quality or compliance uh are involved. Second um the component that emerged as the most reused component in this stack is calite. Um it is a database with bring your own internals. So right so it take carees of all that front end of understanding how to parse SQL, how to optimize the query, how to pick the plan. So you plug your physical plan, you plug your logical plan, you plug your optimizing uh optimizer rules and it's going to do all that complex front end side of the database. Second uh substrate is the more up and cominging uh project in this ecosystem. It's focusing on standardizing how you represent a plan whether it's logical plan or physical plan. facilitate push downs and facilitate exchanging this information between those components because you start from the front end

You have the execution but usually you want to push a lot of things to the storage like filters or possibly aggregation limits a bunch of things needs to go be pushed down to the lower layer components. So you need a common language to communicate between various things in that Korean engine or if you're communicate communicating with other storage system you need a standard way to do this. Arrow is this inmemory standard inmemory representation. So it's both the columnar representation in memory to enable vectorization and an efficient exchange system between components. Right? Arrow is designed so that the immemory representation is the same at the onwire representation. So you don't spend any CPU cycle serializing derializing from one node to the next. You just copy the data over because it's all based on relative pointers. So it doesn't matter where it is in memory and so it's relocatable and allows a zero copy network exchange right like so you have this efficient exchange that you don't spend any CPU transferring data and um any unnecessary CPU and it's cross language compatible and it enables this fast inmemory access for vectorzed execution we talked about at the beginning data fusion has become the standard for reusable component that knows how to do a bunch of vectorization um that we used with the arrow kernels that know how to do the basic vector operations

It knows all how you implement a query engine on top of arrow um in a way that's really fast and state-of-the-art uh vectorization and so that it's very pluggable and makes it easy to extend to various type of query engine like for example a time series database or uh a streaming database or a regular a regular SQL um execution engine and then in the storage layer that where it gets a little more interesting. Um so a storage layer it's a distributed data set with push down right it enables distributed reads um for multiple streams in parallel we know how to segment the data and read it in parallel from multiple machines and it knows about push downs like projections filters limit aggregation how to avoid reading more data than you need right how we push more of that to the storage layer so there's two layers in this ecosystem that uh you can use with different abstraction one is the iceberg park or the open data lakeink, right? That's great for bulk updated, scalable, cheap storage. Uh, and you don't need to operate your service. The storage is the blob storage. So, it's as reliable as the blob storage, which is nice. On the other end, you have a flight or a bunch of other similar interfaces to expose a data set in a distributed arrow. So, it's one level up here. Now you're living you have your own service that exposes arrow as um as an interchange and it enables you to build your own storage like a low latency transactional storage that exposes data this way right so it's a lot more powerful but obviously it's also flexibility at a cost where it's more complicated to operate you actually operating a a system at scale which is uh on the data hot path when you use iceberg you don't like the hot path is s3 Right? You don't have to worry about this

And so the way this works, right? You have execution engine and a flight service. You can have the logic you want, right? You think in particular things that are interesting is how do I ingest data and make it efficient since a lot of blob storage columnar data, right? We still have this idea of immutable files that you rewrite in a blob store. Therefore, you know, it has some constraints. So if you want to do something more transactional um you need to implement your own logic and you cannot just rely on blob storage. And on the other end uh we have the open data lake right which is own your own data and brings a proper table abstraction on top of blob storage. You know back in the day we would just share files in folders and that was very uh constraining right you could not change a table because you could not really add files after the fact right there was this whole notion of I wrote this thing and now it's immutable because if I change it I'm going to break all the things now we introduce iceberg it provides a proper abstraction and we have actually tables we can mutate you know like it's kind of snapshot is relation you mutate them in bulk but you can mutate them there's some notion of um snapshot isolation and you have good semantics uh for um all app database. So and sorry I'm speaking ahead of my slides a little bit. Uh so snapshot isolation you know no phantom read right if someone is writing a new version of the table while you're reading it you don't see those right you see them next time around when you start reading the table again right you see the old version right it knows that there are various snapshots you can someone else can be writing to the table and you're not going to see it until it's committed until you read it next time provides some consistency right you know there's the previous version the new version and things can be rolled back right if you push the wrong version

You modified a bunch of things in the table. You can actually get to the next uh previous version. It's provide abstraction of partitioning. You can repartition things independently of anything that's reading it. Uh you're not coupling incremental processing and partitioning, right? The old way we would do that is would schedule things to read one hour at a time, right? Like it would read that one folder that contain that one partition. Everything was tied together. Nowadays, this is all nicer and abstracted out. The partitioning on the table doesn't have to match the scheduling of the processing and things can be decoupled and changed after the fact

And in understand bunch of the efficient push downs, projections, filters, limits so that you reduce how much data you're reading from there. So in iceberg, the service is your blob store. Like so of course nowadays you have a catalog that knows what the latest version of the table is. is kind of a you you have a catalog that's kind of transactional store to know what the current version but that's all it knows about everything else all the hot path of accessing the data is in blob storage so it's very reliable and very cheap and you don't have to worry about scaling that up and down you know your cloud providers deals with this [clears throat] so to sum up uh the open data lake we have two things right you have on one end the centralized component data catalog governance lineage and the specialized component are not centralized at all. It's all the compute engines and everything's more flexible and you're not stuck in your silo as anymore. And so when you think of that, you know, I put some examples here. Sorry if your database of choice is not on that slides. It's just slides have limited space

Um you can store your data in one place. This open data lakeink. Anyway, Snowflake, BigQuery, Trino, Spark, they all use blob storage to store data. So you may as well own your own storage and be able to use multiple of those tools and not be stuck with a decision you made at some point of putting data in one of those and now you can use only that tool or it's expensive to get data in and out of it. So conclusion um we've seen the key uh component that supports that in the open source side you have two main categories one is the open format that enable that composibility substrate for the plan arrow for the data park iceberg for the data lake and open lineage for the governance understanding where data is coming from we also in the open source you also have those reusable libraries that are actual implementation of things like calite for the optimizer, the arrow kernels for the vectorzation and data fusion for the higher level implementation of vectorized query engine and the other pillar of the system is the cloud services it leverages right like you have this reusable bricks uh storage and compute so you have this abstracted out blob storage that she that is tiered if you're not using the data it's even cheaper you can say ah put it in a low access tier and I'm going to pay less for it and you don't have to worry about the data should I put the data in or not whether I'm using it or not I can just move it from one tier to the other and compute is elastic if you don't need to process the data at all for some time you don't even need compute at all and so in the future I think just wanted to point out a few things I see evolving you know thinking of blob storage as a component but we're seeing blob storage becomes more and more aware of the tables in it so tables you we see already table services showing up and blob storage starting thinking oh actually I can know about those tables maybe uh predicate push down push downs can go all the way to the blob storage um bulk updates versus streaming ingest right like you can see how uh streaming starts assume um adopting blob storage like on confluent uh there's a cafka compatible based on blob storage uh implementation uh and catching materialization there's a duality between cache and material views that is very relevant to the ARO ecosystem. And on that I want to say thank you and I think we have 30 seconds for questions. So sorry for that. [applause] >> Thank you Julian

You have time for one quick question. Anybody? >> A quick question. >> Hey Julian. Uh sorry this will be a super quick one unlike my other ones. Uh, hey, how do you think about uh embeddings, PDFs and images, you know, other uh storage formats? >> Just how you're thinking about it, >> right? Um, so I focused on the I think there's a few of those new, you know, vector database or multimodel databases that exist uh today. And yeah, I'm not talking about any of those. It's kind of uh you can see over the past year um I think it's very interesting right there's a little bit of in columnar storage how you separate the big blobs from things but that data is also very structured right like when people store videos they have like they will store small segments of a certain amount of time of videos and it will have ton of metadata attached to it right maybe they extracted the audio track and they add voice recognition and you have a text version of it you have a ton of metadata this data is actually very structured And so it fits pretty well in this columnar storage and so on. So of course you know you have some optimization of maybe you just you separate a bit the big blobs from the rest because you treat it differently

But a lot of this is very uh applicable. Other things that are happening there are newer encodings that are coming out that are more um aminable to being processed in GPUs or more parallel or getting random access uh into those storage layers. So this kind of I have another talk that I invite you to come watch another day to talk about the columnar storage for a era. So you think about the ALP FSST fast lanes newer encoding that allow random access better in your columnar format. So for people who have indexes on vector sort that's useful or um also be just faster on these newer um computer architecture that are more parallelism uh or GPUs and stuff like that. Thank you. >> Thank you. [applause]