Designing Data Infrastructure in the Age of Generative AI | Lisa N. Cao, AI By the Bay 2025
Um, I'm going to be giving a slightly altered version of an O'Reilly talk that I gave a little bit ago called designing data infrastructure in the age of generative AI. Uh, it says generative AI in there because apparently you can't get people to show up without generative AI in there. Uh but ideally if you're in this room, you're interested in data infrastructure in general and sort of some of the ways that we're fundamentally changing what we consider to be core pieces of infrastructure and what we consider to be critical uh in future proofing our infrastructure. Uh a little bit about me, I'm currently at a a very tiny unknown company called Data Bricks. Uh which means that a lot of people take what I say very seriously sometimes as indicative of the company. Some of this is, some of this isn't. So this is I'm my own person in a lot of ways. I also do a lot of stuff in the community uh in the open source space especially in the Apache and Linux Foundation communities
Uh really have been in the catalog and open table format space for a little bit now. And yeah that's a little bit about me. So you know my perspective and this is kind of where it's a little bit relevant here where what is happening with data right now. I feel like we had a really long period where we discussed, wow, there's just so much data and we have this big data revolution and that data is still growing, right? Tabular data is still being used more than ever. I don't want to say that it's going away. It's not and it won't be anytime soon. But the percentage of tabular data that we're working with or structured data is now having a smaller piece of the relative pie because we're now introducing so much more multimodal data into the mix. Right? I think we've seen this a lot in consumer applications that are now much more image-based, video-based, audiobased than just pure textbased applications
And we're seeing at least within the generative AI market that so much of this data is quite multimodal in terms of the way that generative AI applications are generating new data as well. Uh there are some really interesting statistics here. Um basically we expect that generative AI solutions are really concentrated in this multimodal space. And I also want to make a point here in that it's not necessarily multimodal or structured. I think that it's a hybrid of both, right? Like we have structured data that we store in metadata format at the very least or open table formats that we then want to link to, you know, our, you know, raw like sort of multimodal data, our text, our images, our videos because that's how we actually govern and manage these sort of things. And that's kind of the stage that we're at now. Um, it's not to say that we didn't have solutions for this before. I think no SQL has been around for a good minute now but the expectations in terms of latency performance of the breadth that we use uh a lot of these like data that's kind of being generated is quite different
Uh so really this begs the question do we evolve existing paradigms so we do we take the legacy data infrastructure that we have now legacy I don't I don't agree with that word um despite the debate yesterday or do we like start from scratch do we rebuild the entire thing right um I think we kind of know the answer here and that's that infrastructure doesn't work like that like you just don't rebuild things overnight even if new and better tools do exist it's just very unlikely and we've seen this time and time again And I think that this kind of veer towards multimodal data management is just a natural extension of the life cycle of data engineering that we see. Every once in a while we have these generations of tools that come through and we build the ecosystem around it. You know if we look at like hive meta store or parquet for instance uh these tools are still around like people still you use hive metas store quite a bit but we've decided to make it a pinnacle of the ecosystem and build around it until you know bigger and better things came along and although we're still migrating to things like rest cataloges uh generally speaking a lot of us are still deploying solutions side by side that makes sense to me because infrastructure often goes through phased migration approaches versus all or nothing. It's not like everybody just went to iceberg overnight. Um, and even with parquet, parquet, there's been a million in one columnar, you know, formats that have like come through. After parquet that maybe are arguably better than parquet, but we still use parquet for everything because it's the basis of the ecosystem, right? Delta, lake, and iceberg in particular, Apache Arrow, uh, all of these things. So, uh, just because something new comes along doesn't necessarily mean that we rebuild, uh, everything from scratch. In fact, uh, arguably what happens is that we put old stuff and new stuff together
I'll get into that in a little bit. Um, if we look at kind of changing existing paradigms, and I hardly would want to say that Delta Lake and iceberg are legacy in any sort of way. They're not. Um, they're actually still very new in my opinion. Uh, generally speaking, we're seeing changes within these open table formats that are trying to adjust to more of these multimodal use cases. Uh a great example is the use of binary columns where you can store blob data in line within the open table format itself. But also, you know, that's not the most performant thing because we have a query engine that's trying to scan this table. Uh you don't want it scanning a giant blob in the middle of that over and over and over again
And so what you can then do is maybe store the path to that blob within object store uh in the column instead and then you know use that as a reference. There's pros and cons to both of this that we'll talk about in a little bit later, but there's, you know, how do we manage these references without getting a ton of orphan files. Uh, but also how do we make sure that they're easily accessible and that we're not jumping a ton for latency and performance reasons. Uh, we also see, you know, some new things coming up like variant, uh, which I'll talk about in a little bit and file format APIs that are currently being developed. So, this is like very new, uh, in especially in iceberg v4. Uh there's currently a proposal that's being like almost fully merged in. I would say it's very close uh called the file format API and I expect that this is going to be extended beyond different uh open table formats too that will follow trend. And then of course if you're looking to go beyond the table format uh you know the catalog also will do volume management things like that of different object stores and file systems
So yes, there's like a lot of variation going on here mostly centered around blob. Uh if we look at variant types, so this is a new like basic primitive that is shared between parquet iceberg as well as delta lake. And this is great because if you have all three uh in agreement with each other, that's your standard right there, right? You don't really need anything more than that. Um which is great because that covers I think most use cases of people who are using you know an open lakehouse format. And so what is variant? uh you can store you know the binary data uh in more of a compact way directly within the table format but this is really meant to support less so purely unstructured data but more so semistructured data so like your CSVs your JSONs your XMLs things like that and then it becomes very efficient to be able to query that data but one of the cool things that you can do is actually shredding so what do you do when you shred let's say you have this very deeply nested JSON that you're then storing as a variant type within like iceberg or delta or uh etc. And then what are you going to do? Maybe you want to access one of those nested fields. You can then uh extrapolate that out into its own column. That's what we call shredding
You're literally shredding uh it out. Uh which is awesome uh that that is supported uh at the parket level. So what if we look at specifically blob data management and what are the characteristics of maybe multimodal data in general as we think about you know how we're storing uh this data. So multimodal data if you're storing it tends to be very wide especially in some sort of table format. We're going to get into lans in a little bit and maybe how that changes things and it's often GPU bound right and this is because a lot of this data is bound for some sort of like a IML workload which requires a lot of features a lot of rich metadata and so there's a lot of constraints that can come with this. Uh it's not necessarily narrow and cheap like you necessarily would with uh tabular data. It's quite expensive to store as well as to process and we kind of know that uh which is why you know Nvidia is making a ton of money right now. Uh but generally speaking blobs can be very massive right they can be huge especially as our expectations as people uh exceed and we want things to be bigger higher quality better and we don't want to give up any latency uh for that as well
So this is just the reality of what consumers are like. Uh and traditional query engines aren't necessarily always the best equipped uh in terms of blob awareness. And so it's really important to keep in mind that there's new waves of engines that are coming through, but it's not like Spark is going anywhere either. So how can we find this middle ground again of changing old paradigms to fit these new use cases and still being performant while also being able to leverage new paradigms as well. Uh if you want to look at ways that we can make query engines more blob aware, we can do things like lazy loading. Uh we can handle different streams and we can break up blobs uh to only fetch certain portions. There's a lot of different things that we can do to kind of you know ch change and make things more performant uh if we want to and how we actually query optimize uh a lot of these blob reads as well is really important in that planning process. And so I think it's just going to be very punishing to not optimize for these sort of workloads
Uh but we also have native multimodal formats as well. So we'll talk about this in a little bit. Uh but generally speaking, if we're going to do multimodal, we also have to think about, you know, how we're going to manage the blobs and how we're going to create the best system that works best for us. I don't think this is anything new pattern-wise. Uh like in terms of zero ETL, in terms of hybrid storage, like this is just sort of standard data engineering work. like the actual goal sometimes changes but generally speaking the patterns remain the same. Uh things like that that are meant to know. Uh two shout outs here are definitely Lance DB and Daft
Daft is a multimodal native engine and lance is a multimodal native format. Uh and why are they so interesting right? So, Lance DB uh performs something like chunk uh compression that will allow random access to be a little bit more efficient with these uh native table formats, but also it's just really meant to, you know, store features and embeddings in a way that's more easily accessible. Uh one of the cool things about Lance is that and we'll get into this a little bit later is that there's a big question with how do you manage uh at least the governance of unstructured data? Lance DV actually offers its own namespace that can then be managed by cataloges. Uh we'll get into how Unity catalog can manage uh Lance in a little bit, but that's really cool as well. And then there's also GPUware compute engines uh like Spark Rapids project which came out of Nvidia as well as Daft as well. So really um again, are we going to use a CPU for everything? No, we're probably going to start introducing GPU a little bit more. How far do we want to take this optimization? And again, uh, will something more hybrid come out of the gate, like the NPUs, which I feel like are not totally mainstream yet, but are really interesting to keep an eye on. Uh, and we may see more innovations of this at the hardware level that maybe we don't have to overcompensate for at the software level
And so, interesting thing to keep an eye on, right, that now we're having to be very much more aware of the hardware implementations as well. So this begs the question, do we want to hypers specialize and hedge all our bets into one table format? Uh I think generally speaking, this is always pretty bad practice, especially with something that's untested. Uh I personally really like daff and land. So I think that they're great projects, but also they're still pretty new, right? And it it's very unlikely that we will want to move our entire infrastructure them if they're not tested, you know, battle tested in production in like really large enterprise scales depending on who's backing these open source projects, etc. And so, you know, what is the right balance of where to kind of hedge our bets and for what reason and how do we manage all of that as well. Uh, I think we're really moving away from all or nothing type of formats and engines. Uh I think we've had a generation of really specialized tools after a generation of really general tools and we're kind of moving a little bit more towards you know modular sort of set where we figure out okay what is featurerich enough for me to be usable and to be interoperable without necessarily having to worry about you know being siloed. Uh which kind of brings us to our fundamental question again which is you know how do we actually design an open lakehouse that can fit all of these needs? We've kind of established here, at least I've convinced you in the last five minutes, that tools are coming, tools are evolving, old tools are changing
There's a lot going on at the same time. And I don't necessarily think a lakehouse is going to be the best suited to manage all of this uh at once. I think it was a really great idea, but we still need to put in some guard rails there and there's still a lot of performance optimizations that a traditional lakehouse doesn't provide. Which brings us to this concept of a composable system where beyond just modularity in the data space and data storage space. Can you bring that idea and especially in terms of like reusable assets uh in the compute space as well? How do we build connectors that can fit each other without having to necessarily be so rigid? And this brings us to this idea of you know can we have shared material views for instance can we have shared governance can we have shared execution uh all of these things that make it really interesting to to share these internals that are usually quite complex and are really unrealistic for a single tool to be able to manage on their own and then unrealistic for data engineers to expect to manage between these different tools themselves. Uh this is really important because we've seen two separate stacks emerge, right? And I think anybody who was at the debate yesterday probably had a good sense that this was kind of what was happening is that there's a very pythonic AI stack and there's a very GV uh JVM heavy, you know, data engineering stack, but the data is still shared between both and you still need governance between both. I think that's why you'll see in the middle there catalog is shared between both layers technically uh because if you want to scale out AI applications uh not just necessarily for the sake of legal compliance but also for the sake of observability evaluation all of these things how do you then you know bring all of these into the same fold right because data is still data and you don't want to create more copies than you need to but also you want things to be able to leverage each other as well uh which brings us to this idea of catalog for architecture. Uh generally speaking, a lot of the catalog commit coordination right now occurs at the file system level and in the file format
So iceberg will have their way of like handling snapshots and metadata trees. Uh Delta Lake, you know, traditionally uses the Delta log. And so can we actually move that up into the catalog to have the catalog managing all of those commits so that then uh and I think the IRC is a really good example of this paradigm shift uh to be that then source of truth and this is really cool because then if you have say Lance alongside iceberg alongside maybe even Delta if that's you know kind of the architecture that you're working with with whatever else you have can you then have all of those commits coordinated and all those policies stored all of the you know security measures that you want to have all the governance within the catalog space so that then you're not having to worry about whether or not the table format can do it all in itself. Um this is really cool because I think the IRC has paved the way for a lot of this too. That's why we've seen uh lens name space uh being developed. And so then you know you also have this really cool idea of like multi-table transactions being unlocked as well. So, this is something that's really cool if I want to be able to leverage, you know, multiple table formats and I don't want to worry about necessarily exposing the user uh to all of these different complexities that may be really hard. And so, how do we then uh pull that off pretty much, right? Uh this is where I bring up the delta kernel
So the delta kernel uh is really it's a rust library pretty much at its core uh that's really meant to be embedded within different engines to then enable you know any engine to understand natively the delta protocol which is used for delta lake including delta sharing as well and all of the the goodness of the delta ecosystem and it it's rust native which is really cool because it means that it has a high level of compatibility between different engines within the stack. It's also available in Java and it's really meant to be this one-stop shop where you can build an engine that works to suit your needs and then you want to leverage the engine in whatever it is, but you don't want to manage the different versions of the engine because engine XYZ might be the only one that has Delta Lake um you know support, but then maybe Delta Lake protocol changes and then you don't want to manage, you know, another version of the engine and all of these different things across these different use cases. And so that's what kernel is really meant to provide is a way for the engine to inherently understand the table format and its complexities as it evolves without necessarily having to worry about the complexities. And then this is what we mean by composable system because then this can be shared and extrapolated into all of your engines if you want to. uh which is really cool and you can still then leverage uh the actual you know spec specifics of the engines in terms of query planning uh that make you want to use that engine in the first place right uh and this is really cool because Delta kernel actually integrates with Unity catalog's commit coordination service and so this is the catalog coordinated commits uh and it will route the the commit coordination to the catalog in itself so then you're bringing everything into the catalog uh which is really cool. And why is this so important? Well, let's think again about what a composable system is and kind of the reality of working with so many different engines. I think there are some really exciting engines uh like polars and ray etc on top of like spark trino uh and we don't necessarily want to deal with the complexity of every single engine dealing with these sort of primitives in the open table format in different ways right that's just a huge amount of complexity but we also want to leverage the old tools and the new because everything is developed for a very specific purpose and so in an ideal case you know delta kernel would just be a layer that sits on top of the catalog Fortunately, that's just not how it works in implementation. It's just very messy
And so, it's better to embed it within the engine itself. And then that will create the connector and then it'll be able to understand, uh, the the data format a little bit easier and better. And so, uh, kernel is inherently designed to support multiple formats. We're trying to nail it with delta obviously because we have a lot of stake in making sure it thrives. But this is really cool because then if you can see the the future picture here uh maybe kernel will be that place that can be really modular and allow for a composable system to thrive between different table formats between different engines and then we don't have to worry about which one is the best anymore right and we don't have to worry about what's legacy as much uh delta kernel has had a lot of development work on it already which is really awesome so uh everything on this ecosystem that you see within like Java and Python are actually able able to plug in and leverage Delta kernel and then of course when you're using Delta kernel you can then leverage say uniform which allows you some interoperability between iceberg hoodie and delta lake um through unity catalog as well as delta sharing so you can share these views as well which makes it a little bit easier on these different cloud providers uh so yeah it's really plug-and-play and it's pretty easy and straightforward to to run and so if you're interested in Delta kernel uh I'm pretty sure that QR code goes to the GitHub repo. This is a community project by the way. Uh this wasn't born out of data bricks but we are really excited to see it. And so again there's a lot more that we're seeing in terms of the development of Delta kernel
So do keep an eye on it on the next year. I think it'll be really exciting uh to see us explore more of the rest ecosystem. Uh which brings us to Unity catalog in general. So Unity catalog is kind of going to be I think the center of gravity for managing data and AI assets. I think people are a little bit cataloged out at this point. So I'm not going to be hammering home what a catalog is, but generally having a catalog that can then provide multiple options in terms of these open APIs uh including like the IRC, including Unity REST, including delta sharing will provide you a little bit more flexibility and and then when you're leveraging Unity catalog, you then have, you know, at least in the open source version, the ability to plug in your other tools, right? and you can build it and create something that suits your stack a little bit easier. Does take a little bit more leg work, but also if you're using open source, that's kind of expected as well. Uh, and really that's sort of the vision is that if you're managing all of these assets, including these models and tables and views and volumes, which we'll get into a little bit later, uh, you'll then be able to use any format with any tool and then you're coordinating it
You have the governance in place. You're able to manage those assets really effectively. kernel is really great but it has its specific scope in terms of what it's trying to achieve. Unity catalog is a catalog right and so if you have all of these together then ideally you can then serve it to your AI uh systems later. This is where the generative AI part comes in. I know you've all been waiting for that because multimodal isn't totally generative AI uh in our world yet. Uh and then we can leverage things like functions. We'll get into this a little bit later
Uh Unity catalog can manage volumes. So if you have uh either a managed or external volume as long as you have the path for your external volume, you can then manage and govern uh sort of that blob data and where that's being stored as well. And then you can still leverage uh all of the things that Unity catalog is known for such as you know credential vending uh governance policies aback etc. So uh same with userdeed functions. So if you define a function you can create it as an asset in Unity catalog which will then be a governed asset. So it's a first class asset. uh and it's then reusable across different systems as well and you can have different iterations of these functions and then it can be exposed to different AI systems as well which we're going to be talking about later but uh it's really cool to then have them be dynamically called according to what your use case may be. Uh Unity catalog also offers uh AI specific SDKs in the Python ecosystem
Uh so pi package that comes to mind is Unity catalog AI. uh this acts as an interface between normal Unity catalog which is uh Java implemented and you know maybe a Pythonic system that you have to make AI systems and functions and tools a little bit more easily translatable to the Unity catalog layer. There's a lot of integrations uh such as like lang chain open ai anthropic uh and you know coming into this is MCP support as well and then of course because it's unity catalog it has uh actual integration with MLflow right this is like two systems that are very coupled MLflow has a lot of amazing uh new features around you know generative AI you know evaluation judging scoring observability tracing and has anel implementation as well and So uh there's a lot here that I can't get into uh because that's a different talk from a data infrastructure talk but I do recommend looking into it because MLflow is completely open source and it's full you know fully featured uh seriously but you know the original you know MLflow versions that we saw you know did model uh tracking and training and all these things that were inherently already you know registered with Unity catalog that integration was pretty much like from day one and so now uh you get to see MLflow evolve and unity catalog evolve for these pieces of information that you might have for these new use cases. And so you have your data there, your multimodal data, your tabular data, your functions and whatever tool set you have and then your actual models that you have and then you have them governed uh for AI which is really cool. Uh if you're interested, there's a Unity catalog road map out for 0.4. Uh and then there's Unity catalog AI as well. Uh but let's get into agent quality tools. And so generally speaking, uh if you're working with agents, you know, you have no idea whether or not they're doing what you want and you just know if it's good or bad, right? It's kind of like a past fail sort of thing
Uh how do we actually evolve and develop agents so that they're actually useful for us and actually create custom agents that work for our businesses, for our implementations, for our personal use cases. Uh MLflow has a lot of these custom uh building judges that you can do. So you can build a judge with MLflow that will then actually collect feedback. It will then change uh and tune itself depending on sort of like the feedback that you give it. And this can even be done through another LLM as well. So you can just say I want it to be XYZ. Let's say you have a specific domain expert that wants to leverage uh and score and tune things uh to their liking. You can do that and then it can trace across the application layer which I think is really important
So this includes you know TypeScript, JavaScript uh and things like that and it leverages open telemetry. So you're getting actual instrumentation for your AI applications and then of course this is all open source uh with agent as a judge as well uh to evaluate your AI applications a little bit better. So then you have all of that governed and tracked within Unity catalog and then you're actually able to iterate on that with MLflow. And so this is uh the way that MLflow does this. If you want to evaluate typically you know if you want to do a manual evaluation it's possible but it is a little bit tougher and so then what you can do is after you build your initial evaluation system monitor it you know using the MLflow UI or whatever else you want to use and then actually build and iterate so that you're actually building an application that might suit you a little bit more. So uh the summary of this and I think I might actually be ahead of time which is crazy because I usually am very behind time. uh the summary of this is that I think the fundamentals are changing in terms of what we see as a critical pieces of data infrastructure. I think we thought very much in these sort of like waving motions of what is the fundamental and I think the open table format was a big part of that fundamental and it makes sense right but now I think we're seeing with this rise of composable systems what is truly the fundamental piece that we need to invest most in so that we can create a futureproof architecture for ourselves so that we can actually allow ourselves to use and leverage new tools while realistically keeping the old ones because they're not going away just yet
And I think reasonably speaking, we want to create phased migrations. And so, how do we think about where our center of gravity is within the different components of our data infrastructure and what is worth investing in? And I think that's really at the catalog and connector level that we're seeing here. How are these engines interfacing with this data? Uh, etc. Which is why I think Delta kernel is such an exciting project uh because it's something that we just have not seen before to be honest. Um and then also you know if we're going to discuss the inter interoperability of these tools we're seeing new libraries that are much more memory safe like Rust uh come through that are making it much easier to create these embedded libraries and then because it's so much easier to say embed like a rust library within like you know an engine it's a it's much safer uh to move forward and and I think this is a very stable path and re very realistic path to think about the way that we design data infrastructure uh catalog managed commits are very crucial. I think that the more that we move into the catalog, the better uh because the catalog is the source of truth for everything and it's already well equipped to be able to bridge across different stacks, manage all the assets, give you the observability that you need to within your system in terms of usage, you know, who's calling what function, uh all of these different assets and probably become then that place for modularity. And really if we want to look at any single catalog or any single tool, any single engine, we have to think about the ecosystem and how it actually interoperates with each other, right? It's not just necessarily is you know spark better than trino or is you know polar is more useful for me than ray. It's more so about well what is the ecosystem that's supporting that particular tool and how do I actually plug things into each other that futureproofs my own infrastructure my business my ecosystem my organization and this is something that's really important because if we take for example you know something like hype metastas or parquet the strength of these really are the ecosystem that's what makes it a robust and productionready tool sometimes uh it's not necessarily just that on its face it's the best performing thing ever
I think realistically we've noticed that maybe that's not always the case. And so the the whole is greater than the sum of its parts here. Little bit of a cheesy line to include, but I think that it it's really important is that you know is it just Unity catalog or is it Unity catalog and Delta kernel and the open table formats. Is it just Lance DB or is it Lance DB plus you know the catalog implementations that support it? you know, what is that particular project doing to allow itself to plug into the ecosystem and who's backing that I think is really important and why we're seeing so much power be behind, you know, these new AI tools that come out. So yeah, uh questions uh concerns, complaints, compliments uh yeah >> this transaction multimodal transaction are these fast enough. >> Yeah. So you mean like mostly in that sort of like NoSQL world that we see in terms of trans Yeah. Like whatever
>> Yeah. I so I think we're trying to figure this out right now which is like that that's the caveat here and that there have been a lot of you know databases like serverless Postgress for instance that have tried to fill in this particular application space that's probably the best running the really big question there is then how do we connect those serverless instances to our lakehouse um if we want to right do we want to use something like you know I think Moonake Labs is a really good example of this where they were really creating a transactional layer on top of iceberg. Uh, and that has its really cool use cases, but it can also have a lot of caveats itself and generally speaking can be hard for governance. It can be hard for other engines to understand. So, we might just see a divergence of the stack and that might be where we end up going where we have new tools and a new wave of tools that actually are specialized for this transactional layer, but then we try and connect it back to, you know, the ecosystem somehow. So, honestly, I kind of don't know. We're still figuring it out like that. That's the there's like inklings around and there's promise, but like you're completely right on the money where the concern is that this is where applications are going, right? And we kind of need to figure out how we're going to work with it because these data applications are so intensive right now
>> I never dumb questions. >> I think we're all confused. So, right on the money with it, though. Yeah. Any other questions? All right. >> Thank you.