Open Standards and Lakehouse AI Development | Alex Merced, AI By the Bay25
Recording: Open Standards and Lakehouse AI Development | Alex Merced, AI By the Bay25
Hey everybody, my name is Alex Merrced. I'm the head of Devril at Dreo. Um, basically again DMO is a Aentic Lakehouse company. Um, basically that's who we are. Again, if you want to learn more about DMO, head over to Dreo.com. We do have a free trial. It's actually really cool, but um, I'll leave it at that. Um, again, I'm head of DEO Dreo
been at Drummond for four years where I've talked about basically lake houses, Apache iceberg, Apache arrow, many of the topics I'm going be talking about during this talk for the last four years helping people kind of get hands-on and and in situated with uh lakehouse architecture in order to fuel their BI and AI type workloads. Now, one of the cool things I've gotten to do over the years is I' I've been able to I got the the honor or the or the work to write a few books. Oh, this is my event calendar in case I am doing two more events in the Bay Area before I leave on Friday. So, if you're interested in where those events are, this QR code will get you there. Um, but I've gotten to write some books. I wrote Apache iceberg, the definitive guide. I got to write Apache Polaris, the definitive guide. Um, also did this collaboration with Conflin
Those three books you can get for free. Just scan this QR code that takes you also that card that's on the seats that actually takes you to the same spot. Um, and then then my newest book, which is I'm I'm currently wrapping up, is Architecting an Apache Iceberg Lakehouse, which hopefully should be officially released early next year, but uh it's already available for pre-release to read what's available currently on on Manning. And then there's a raffle for a free copy of that book. Okay, only I think there's only one more QR code left. Okay, so with that, and then this is if you if you if you're interested in trying and I'll show this one again at the end of the presentation. If you're interested in learning more about a lot of the stuff I'm going to talk about today, this will be like a nice blog tutorial that'll walk you through a lot of that stuff as well, hands-on. But um again, I'll share this one again at the end
But what I want to talk about today is one lakeouses. We're going to talk about what is a lakehouse and why lakeouses are important, but how lakehouses kind of form the foundation for AI success. And the reason why they able to do so is because they're built on open standards. And those open standards allow data to be able to move freely, to be more flexible. So that way you can build um your AI projects on top of it because one of the big challenges oftentimes is knowing where the data is especially when we're talking about like analytics type AI applications where you're trying to run analytics on your corporate data or organizational data. So I'm going to talk about um some of these different projects like Apache parquet which is a file format which allows us to store data in a columner format in a very efficient way that's very fast for analytical type queries. But of course, some data sets are a lot of parket files. So they're not necessarily they don't and they don't understand that they're one single data set
This is where Apache iceberg comes in acting as a metadata layer. So that way those groups of files can understand that oh wait, I'm actually part of a table. Okay, but the thing is that you're going to have lots of tables. So this is where Apache Polaris comes in as a catalog that can track those tables and govern those tables. And then when you run queries on those tables, you're going to want to be able to do it really fast. And this is where Apache Arrow comes in where basically makes processing these types of analytical queries in memory much faster by having a standard in-memory format and makes it really easy to transport that data between systems without having to duplicate or copy that data making that transfer between systems much faster. Okay. And then of course once you build your lakehouse system using those components we can then wrap that together with something like an MCP server that can then allow us to interact with that through our our our favorite AI clients
But basically when it's all put together what you want is a lakehouse that is complete meaning gives you access to all your data not just some of your data that's contextual that has access to semantics and so that way it can better understand your data that's governed basically make sure that people who are using it can't access the data they're not supposed to access and you want to automate as many of those processes and make sure that they're performant as possible. Okay. So now that we've kind of established the big picture, let's just go revisit like what is a data lakehouse to put these things into context. So to understand data lakehouse, we think about how we've worked with data up till now. So up till now we've generally worked with databases and data warehouses and they're awesome. I love databases. I love data warehouses because they do a lot of things for me. They abstract away how the data is stored
So I don't have to think when I use Postgress, I don't have to think how the data is in there. I just create some data, insert some data. it's there. Okay. How does it know that this data is part of the customer's uh table and this customer this I mean this data is part of the orders table and so forth and so forth. I don't think about it. It just knows it's it has a way where it knows hey this is the data that I've stored. This is the data that belongs to this table
This is the data that belongs to this table. And how does it know who which tables exist and who has access to them? Well, there's a catalog built into the database. There's all these layers of things that just make it work that we don't think about when we're using databases and data warehouses. on top of also the processing layer where I can write my SQL and that or SQL wherever the preferred is uh but basically take that SQL turn it into a abstract syntax tree uh turn it into an execution plan and then execute that plan all that stuff is just abstracted away from me lovely I get this great experience working with my data whether it's rowbased transactional data or columnerbased analytical data with one big catch every system does it differently the way postgress test does all these things is different than the way that my SQL does all these things or SQL server does these things how Snowflake does it is different than the way Redshift does it so the problem is you end up in a situation like this where basically you have different departments at the company producing data organizing their data doing good modeling doing all all the right things but they're doing it in different systems one department may be using red shift another department maybe using Snowflake and now their data can't talk to each other so you end up in a situation where you're oftentimes having to do movement of data from one department of another so basically you're copying the same data over over and over again to different systems. This creates inconsistency in the data because sometimes the data doesn't move over exactly the same way as it's in one place. This also creates cost because there's a cost of storing that data and running those pipelines to move that data. And then also this makes it really harder to build sort of AI projects. So if I'm trying to build like an AI agent that gives me an analytical view of my entire company's assets, well, I have to now go hunt down where all these different data sets are, make sure that they're available, and it's going to take time for me to deliver that
So I don't really get the same sort of real time or really sort of live access to the data that I'd like. So when it came to trying to solve this problem, we had different approaches. First was the data lake. So the idea behind the data lake is saying how about we don't have a system that just kind of locks up our data in this black box. How about we just store the data as files, CSV files, JSON files, paret files, and we store that on some sort of open storage layer like an like an S3 or an ADLS, some sort of object storage, okay? or Hadoop. Now that works and then again I can sort the cool thing is I can store other stuff here. I can store unstructured data like images and text files and things like that. The problem is it's it's not quite the same as using a database because you don't have all those abstractions there
So running SQL is a little bit trickier. Um doing making changes to the data set I don't get the same sort of asset guarantees. So there's chances that two people try to change a data set and they they they mess each other up. you don't get the same sort of again optimisity, consistency, isolation, durability that you would using a database or a data warehouse. So that quite was a step in the right direction because now different departments could use different tools to access the data. They don't have to like agree to use the same tools because that's always a fun political fight. Um but not quite there. So when we take a look at it, I got my data lake where I store data in a storage layer and I use tools to run queries on that data like a Dreo or an Athena to run SQL queries on that data
But again, it's not quite the same experience. But when I take a look at it from this angle, I realize the reason it's not is because I'm missing a few layers. I don't have layers like the table format. I don't have a way to recognize these different groups of files as tables. and I don't have a way to track these different data sets and govern those data sets. If I add those back in, I'm back where I started. I now have a system that's open. Okay, different tools can talk to it and interact with it
So people don't have to use the same tools, but it works like a data warehouse. That gives me all the same asset guarantees, gives me the same sort of like SQL fronting fronting face that I would with a data warehouse. And thus we call it the data lake house. you turned your data lake your storage layer into your data warehouse instead of having a separate system. Now this is great because now once you have and the thing is that this data lakehouse is built on certain standards for table format because it's not just doing this but it's also doing it in a way that h that builds on standards so that way different tools understand those standards. So I'm going to talk about what those standards are because that's what allows that complete picture of your data because everyone nowadays like Apache iceberg has become more and more so the standard pretty much now you're you're seeing like every tool be able to kind of send data as iceberg making it easy for everyone no matter what they're using and what they're doing to use iceberg makes it much easier to build on top of all your data. So the end result is you get something like this where you have all your data sources you know your Postgresses your uh your MongoDBs your Salesforces and then you're going to ingest all that data into your data lakehouse and now all your different tools that actually do the data processing can then talk to that data and then you will see later on then I would build like an MCP server on top of that or use native MCP servers to these platforms and then that allows my AI frontends to be able to talk to my data in with natural language. Cool
And then you end up with a nicer picture like this where basically now all my different departments instead of having to like worry about moving data between each other, they're all just moving data into one place, the lakehouse. As long as they put it into the lakehouse, they can then share the data with each other. It's just a matter of going to the catalog and sharing this data set with the right user. And then they can plug the catalog into their tool and access all the data. Makes life a lot easier. But again, one of the core standards that makes this possible is Apache iceberg. Now, what is Apache iceberg? So, Apache Iceberg is a table format, not a file format. So, at the end of the day, your data generally is still going to be saved as Apache paret
It's a great way to store data. It's really fast, really compresses well, really works really well. But the idea behind a table format is to say, hey, you know what? I have a thousand parket files that are one data set. I'd like my system to know that because the problem is if I don't know that, then I have to go I have to go pull all thousand park a files, read all thousand park a files to run a query. But maybe I don't need all thousand park files. Maybe I only need 10 of them. So by having a metadata layer, a layer of sort of data about the data. I can then an engine like do or spark or snowflake can then understand, hey, there's these thousand files
Here's statistics on these thousand files. And based on those statistics, I really only need to read these 10 files. So then I'm making less requests to like S3. So I'm reducing cost there. I'm also transporting less data. I'm also and executing these queries much faster. So it just allows me more performant access to my data. And generally iceberg works like so
So in an Apache iceberg table, the basically the layout is this. So really the actual iceberg spec specifies how you work with these three categories of files. Okay? And I'll talk more about that in a second. But generally the assumption is that there's going to be some sort of catalog. Okay? So you're going to have some sort of catalog that tracks your tables. So then a tool like Dreo when it wants to read your table what it would do is say hey uh catalog uh where's the sales table and then the catalog would check the credentials and say hey you you have access to this table and then it sends me like an address in storage where that metadata exists. So then I would go to S3, pick up that metadata file which is a JSON file and that metadata file, this top layer file. What it does, it tells me how is it, what's the schema of the table, how is the table partitioned, basically the whole sort of setup of the table and then tells me all the different versions of the table because what happens is that one of the great things about these table formats is that they have snapshot isolation
So every time the table changes, they actually maintain that history. So in this case I can see hey there's two snapshots in this case. So went got to the metadata file there's two snapshots I want to read the latest version of the table. So I go to snapshot one and then each snapshot is then dictated by a manifest list. The name is kind of a tongue twister because the manifest list is a list of manifests. Well what's a manifest? So the actual listing of parquet files is in these manifests. So those actually list the individual files, their locations, their stats. Now the reason why they don't just build one big list is because then you would have to rewrite the whole list every time you make a change to the table
Okay? You make a change to the table, you have to redo the list. So what they do is they write the list in chunks. So that way I can reuse those chunks from snapshot to snapshot. And then each manifest list just tells you which chunks belong to that version of the table. And then there stats on those chunks. So I can actually take a look at the stats on the chunks on the manifest and say, well based on the column ranges for the files covered in this manifest, I don't need any of the files in this manifest. So I can just skip this file entirely. Okay? And that's the beauty of this
The idea is that as I go through the metadata, not only am I discovering which files are part of the table, I'm discovering which files are relevant to my query. So again, it could be a thousand file table and I only need to actually pull 10 files, read 10 files to actually do the query based on my specific filters. Okay, so again allows me to and this is essentially how databases are working out under the hood. It's just now you're doing it in an open way that any engine can take advantage of because it's an open standard. Okay, every database has metadata like this that it tracks under the hood. Again, this is just creating a standard way to expose it. Okay, so again you have metadata the top level that describes the table. From there we can discover the different versions of the table which are each entailed in a manifest list and then that'll point us to the chunks of files that belong to the table
The manifests. Cool. But I mentioned there has to be a catalog at the top. Okay. And that's an important piece because a catalog is how your tools discover your tables. So this also becomes the place where you want to do governance. like this is where you're going to sit there and say, "Hey, when should I reject sharing that location of the table? How do I determine who has access to the table?" And when iceberg was first created, it didn't really have a recommendation on how catalog should work. It just said you should have one
Something should tell you where the stuff is. Okay? And because of that, you ended up with this, you ended up in this world where originally all iceberg was everything was done in Java. So you basically had like an iceberg implementation for like different cataloges like AWS Glue uh using different databases as cataloges using um Hive as a catalog and all of those were separate libraries in Java. That's fine at first and then new cataloges started coming out that were specialized for iceberg. Now there's a lot of them. Um and they would have to write rewrite those libraries again too. Create a library for their particular catalog. And then what happens is that iceberg kept expanding
Now you have iceberg in different languages. You have iceberg in Python. You have iceberg in Rust. You have iceberg in Go and all those Java libraries don't help you there. So guess what? Now you got to rewrite all those catalog libraries, all those in Python, then in Rust and then in Go. That was not sustainable. That was not maintainable. So the idea was how can we make sure that there could be a diverse ecosystem of cataloges, but you're not having to rewrite all this logic over and over and over and over again
So what they did is they came up with another standard. So we had a standard table format but then they created a standard catalog protocol called the iceberg rest catalog spec and this is an REST API protocol. So the idea is that if every catalog implements this REST API basically you know there's a YAML file that describes here's here are the endpoints this is what the headers need to be this is what the payload should should be for all these different endpoints. Long as you implement that then all the same client for every cat can talk to every catalog because every catalog server will work the same way. So now you don't need to write five different libraries for five different cataloges in Java. You can write one library in Java, one library in Python, one library in Rust, one library in Go and it works with every catalog. And this has allowed there to be a much faster growing diversity in cataloges in the iceberg world. So now you have tools that read this standard which is pretty much every iceberg tool which is now pretty much everything like anything that can pretty much write analytical queries has some sort of iceberg access
So things like Dreo, Snowflake, Apache Flink, Apache Spark, Duct DB, um Star Rocks, Polers, Presto, Trodeo, Dataf Fusion, Arabte, Puppy Graph. All these things have the ability to connect to these iceberg cataloges that follow this protocol. Okay, giving you interoperability. All these things can be used to run queries against that data. Depending on what workloads you're doing, you might prefer one tool or the other, but that's the beauty of iceberg. It allows you to use the tool that's best fit for the job. And on the open source side, there's a bunch of open source cataloges that allow you to deploy and maintain your own catalog for your iceberg tables for your lakehouse such as Nessie, Gravitino, uh, open source Unity catalog and Polaris catalog, Apache Polaris. Okay, that's actually literally once I'm done with this talk, I have to head over to an Apache players meetup
Okay, so I I I talk a lot. Um, cool. But there's also a bunch of commercial offerings in catalog in the catalog world. Like Dreo, we have a built-in catalog. Minio has a catalog now built into their storage. similar to what S3 tables does. Okay, Snowflake has a catalog and again Dreo's and Snowflake's catalog are actually built on Apache Polaris the one the open source tech that I mentioned in the previous slide. Datab bricks has Unity catalog
They have like a proprietary version of that catalog that's a different codebase. Uh data hub um Amazon S3 has two different cataloges. They have their S3 tables protocol which is a separate thing than their AWS glue data catalog which they have they're both fine. uh Microsoft Azure has one lake and Google has big lake. So lots of options. Okay. And this is made possible because of the Apache iceberg restpack because it would be impossible for all these tools to build support for all these cataloges if they have to do it catalog by catalog. But because there's a standard you have this reach
Okay, cool. Now for Apache Aeroflight. Now when it came to data processing um one of the things that used to happen is that you generally want to represent analytical data in columns. Analytical queries are going to be much better using columner data but most data warehouses and analytical tools would create their own way of representing the data in columns for their system. So in the same way that everybody represented their tables or their table metadata in different ways people used to represent their in-memory data in different ways. So the goal of Apache Arrow which was sort of a some basically is it was basically the co-founders of Remy were the creators of Arrow and then they they created that kind of out of another project they did called Apache drill. Um so there's this whole big data lineage there but um basically it created this format that allowed much more to be able to fit much more data in the same memory. So you can actually process a lot more data with the same level of memory in a in a system
But then the second piece that it added was this whole arrow flight idea because the problem was even if you could process the data faster on your system, I still have to then send the data over the wire to my destination and back what was the what was the previous standard? Well, it was like JDBC. Let me just get all this up there. So again, arrow is the in-memory format which is now pretty much baked into everything. So while Deremy was one of the places where Arrow was first used, now you have an open source engine called Dataf Fusion that everyone's starting to build a lot of startups around is um as as entirely built on Apache Arrow. Snowflake has added an Apache Aeroflight endpoint. The Spark, there's a lot of projects with Apache Spark that are trying to replace the Spark internals with Apache Arrow processing. Um and that for like a lot of other engines that weren't originally built with Arrow. So arrows really become sort of an industry standard for really fast processing and parallelization
But again when I was talking about moving data just get those images up there. Okay. Now Aeroflylight's about how you move data. Before you had JDBC ODBC you still do. I mean JDBC ODBC work great when you're working with rowbased data. You take rows you move them over the wire and you receive rows. problem is when you have columns, you then have to turn to use JDBC ODBC, you have to take those columns, turn them into rows, ship them over, and then turn them back into columns on the other side. Okay? And all that sort of switching format around cost time because you're reerializing, deserializing the data over and over again
Okay? And this was a bottleneck. So for example, there's actually a video that I have on YouTube where I compare like PIO DBC to Aerofllight. And I actually same exact query. So both two queries from dreo same exact query so same exact data the amount of time that it takes the data to transfer over the wire is the same the difference is the protocol and it was it's a significant once you get to like a million rows or more it's a significant difference in the time it takes to actually use the data on the other side because of the time you're taking converting the data back into columns and converting the data into rows in the first place. So by by having a transport a way of transporting the data that just is natively arrow. So that way the data doesn't have to change formats as it moves is saves you a lot of time. Okay? So you end up being able to move data around a lot faster. So you're processing the data faster and you're moving the data faster
Okay? And you're also moving it in a smaller format across the wire. So you're saving money in a lot of different places and saving a lot of time in a lot of different places. And now they've built an wrapper around that that's like JDBC ODBC called ADBC arrow database connectivity. So now you can actually use ADBC not just to connect to arrow type sources but any columner source any kind of columner data source can now build their own ADBC driver for their custom format and you can use this protocol to keep your data columner over the wire. So that way you're not you're saving time and again the benefit of this is just faster performance. Okay? Because at the end of the day, imagine if I'm using sort of an AI agent and there's the time that takes the LLM to respond and then there's the time that takes the query to respond on the other side. All of that stuff can add to a poor experience and working with that with that AI tool if I'm waiting a long time for queries. So again, the beauty of Aeroflight is that we're keeping the data columner throughout the entire movement cycle
And again, here's just to show you like the benefit. Like in this first one, you see like 2,000 records. This first chart you see like the difference in amount of time that it takes to move the data is like not that different like for small movements of data the difference between using OBC JDBC is minimal but as you move more and more records so this is like 1 billion records okay the time that it takes to do so with Apache fly is fraction of what it takes to do so with ODBC or JDBC okay when you're talking about analytical data that you you want as columns cool so everything I've talked up till now has been all about okay has all been about um the actual analytical system the actual thing that actually runs my queries stores my data whatnot but at the end of the day if we're building an agentic world you know we kind of want to minimize people we want people to just say hey can you just tell me what the average my average sales is for the quarter and get a response and for that you need some sort of thing that interacts with the tool itself you could custom build tools and you know use a framework like lang chain, build a custom tool that runs a query against a system. But the beauty is we have MCP now, okay? Model context protocol. Now, instead of me having to rebuild all this logic every single time I create an AI agent, I could either one build an MCP server that has that logic and then connect it to multiple different interfaces. Or two, many of my favorite things now have built their own MCP servers. Actually yesterday I was at a Fiverr meetup and they were talking about how they had just they're just they just released an MCP server to work with their connector builder. So one of the things Fiverr's done is used for is for moving data from one place to another place but they don't connect to everything
They connect to a lot of things but they don't connect to everything. So what if you wanted to connect to some pull data from some random API that they don't have a connector for? They had an SDK but then you'd have to write the code yourself. Now they built an MCP server that'll actually walk you through and then actually write that code for you to build your connector to your source. That's the beauty of like MCP that instead of me having to build my own agent, I could just take the MCP server they had, plug it into my they plugged it right into cursor and basically just started working with it. Okay, I could take any MCP server, plug it into chat GPT into Claude or create my own custom interface and just get right away to working with it. Okay, and this just kind of basically rehashes what I just said. But now you end up in this world where basically now I can create that world where the data is complete because the data is if we standardize on iceberg then all my different departments can use whatever tools they want to work with their data and they can more easily share the data without having to recopy the data. If they're using a catalog like Apache Polaris then that data can be governed and the right people can have the right access to the data
And if you're using systems like use Apache Arrow the processing of that data can be really fast. So you get this really fast data system that can then power like with with a with an MCP MCP server on top can be the foundation for really fast uh really robust analytical system. The only piece I haven't gone too deep into is the semantics of it all. And again how a semantic server can be delivered can be a lots of different ways. That's one of the big conversations nowadays isn't it? Um but I mean I mean of course if you're using a tool like Dreo we have a built-in semantic layer but then there's also services like at scale or u cube that have sort of like these separate MC semantic layers you can get separately. There's a Python library called like boring semantic layer. All these things that are there to build context on top of your data so that way your LLM can understand what data you have a lot better and what your business metrics and what's the context of your business. But the idea is that generally it's going to be your MCP server that's going to go reach out to your semantic layer to go pull that context in
And that's another really nice piece of the the MCP protocol is you can build those extra steps in there. So like I know in the DEI MCP server will actually go fetch like the wiks for your data sets before we um you know run at runs just random SQL queries. So that way the LM can come up with the right SQL query before actually running it. So you get you're not running as many like unnecessary queries as you would otherwise. So you end up in a world where you have iceberg tables. Your data is stored in iceberg. It's tracked in a catalog. Apache Polaris
Now a little bit more about Polaris is that Polaris has these two ideas of cataloges. There's internal cataloges which are the cataloges you are n that are natively in Polaris and Polaris manages. But what if you're using AWS Glue already? Okay, you have data as glue catalog. Okay. Then there's something called external cataloges where you can connect to those external cataloges to um Polaris. So that way all your data can be visible from one place. So so basically you can even have multiple cataloges and still give your users a unified view of all that data. One place where they connect to all of it
Then you have MCP servers. Then generally you're going to have some sort of lakehouse engine that actually runs the queries. So that could be like a DMO, a data fusion, uh, duct DB, TRO, some sort of engine that's going to run queries on your lakehouse, a semantic layer of some sort, and then an MCP server that can interact with both of those things. And then that connects to your A or agentic interfaces to create your nice analytical sort of agentic world. Okay, I think there we're just about at the end. Now, if you actually want to see a lot of these components in action, that's where this tutorial comes in. So here you actually see like all working with iceberg, working with Polaris, working with an engine that uses arrow, using an MCP server and actually seeing like literally within like few minutes actually asking questions of your data of of of working with this tool set. So you can scan that QR code if you want
Um but with that we I think I have like two Yep. two minutes left. And so with that, I'll open it up to any questions. Hopes, fears, dreams, raise your hand. I'm here. Go once. Going twice. Yes, we'll go with we'll go with applause
Yes. Thank you. [laughter] >> But >> anyone has questions? >> Yep. >> If if no one has questions, I mean, I'll still leave it open to questions, but just to point out again that those cards on the chairs, that QR code will take you to where you can find the free books, the Apache iceberg and Polaris books. You can find this some tutorials and some other resources there, too. So, also the link to where you can get all my slides. So if you want the whole slide deck, you can find the whole slide deck in PDF form there as well and along with like every other slide deck for all my other presentations. Cool
>> Thank you so much, Alex. >> Oh, thank you. >> Yeah. Uh so how does the performance of uh rest catalogs compares to uh the custom libraries because like I think uh rest is probably slower than the JPC calls that are implemented >> theoretically but like the payloads aren't like super large like because basically you're not sending the data over the catalog. You're only sending like a loca like an S3 address. So when you're only sending like a string of that's an S basically an S3 address with a few other properties. Um is the the difference is most of the time negligible and having sort of a standard interface just allows the ability to kind of build functionality and more tooling faster. So I would say the trade-offs are are well worth it
>> Makes sense. Thank you. >> Everyone having a fun AI in the Bay? Yeah. >> Any more questions? >> No. >> Well, again, I I super appreciate wrapping up or being, you know, and w spending this time with me this last hour or this last half hour. I hope you guys have a very fun rest of the AI in the Bay. If you have any questions, feel free to follow me on LinkedIn. I'm always glad to answer any questions at any time
Um, constantly putting out a lot of content, blogs, podcasts, videos. So, if you want to follow the lakehouse space and a how AI relates to the lakehouse space, uh just yeah, just follow me on LinkedIn. It's good times. Thank you very much. >> Thank you.