SBTB 2023: Dipankar Mazumdar, Apache Iceberg.

SBTB 2023: Dipankar Mazumdar, Apache Iceberg.

Recording: SBTB 2023: Dipankar Mazumdar, Apache Iceberg.

[Music] um thank you so much everyone for joining this session um I'm going to talk a bit about Apache iberg today uh and introduce a Leos architecture uh we are not going with the jargon too much but I'm going to try to break it down technically uh and then we'll also talk a bit about interoperability that's a huge aspect uh in today's table formats world uh so I the I'm currently a staff data advocate here at one house uh before that I was at Dro I was involved with Apache projects like Iceberg Apache AR as well uh mostly on the advocacy side and uh you know making so sure like uh developer experience is greed and like you know people uh get easier for people to get started with those ecosystems uh more recently I'm involved with Apache hoodie as well and you know one table that's something uh new um that we just launched today um and my previous roles has been in um you know different you know different roles different segments like software engineering uh data visualization machine learning as well um so in terms of the key takeaways uh we'll focus on uh we'll start with the evolution of kind of data architectures and we'll then go through how it has evolved over the years uh then we'll come to our prime topic which is like Iceberg and um you know table formats and then we'll come to uh do a kind of a deep dive of the architecture of Iceberg and try to understand how uh various SQL queries work under the hood and you know how they leverage the kind of architecture um finally we'll end up with some design benefits and talk about about one of the important aspects that is interoperability so over the years organization has been kind of heavily investing in building like centralized reliable and scalable data platform uh the whole idea was that uh if they could basically let democratized data uh they could have more data consumers like data analyst or data scientist uh to leverage it and further uh kind of enable analytical workloads such as Pi machine learning streaming those kind of workload so data warehouse we're kind of the first one uh you know data warehouse or ol databases are you know same to me uh they were enabled to deal with that kind of like you know workload more structured and relational workloads but then it also lacked ability to deal with unstructured you know data for more advanced use cases such as ammo and stuff uh and I know today you know we are able to probably deal you know solve a couple of things around that uh with new cloud data warehouses uh so then came the era of data leaks uh with the Hardo ecosystem uh which is which is on promise so data leaks were kind of great right you know organizations could basically store all of the data whether it's structured unstructured didn't really matter uh and you know you could dump it in a file system like a Hado file system and then with the rise of cloud uh and it's kind of cheap storage uh we have ended up with kind of object stores like U you know S3 all all three major kind of cloud providers like you know Google storage Azure blob as well so um you know the B main benefit with this kind of system today is that we can basically scale out the story as we would like right and and you know the other benefit is that we also have the cost Factor Associated towards that um so the cost in general is kind of less and obviously there are other you know cause factors involved in that but in general the idea is that you could basically scale out the storage as you would like to but there were a couple of problems and of course that's why we are here today so one is that you just can't dump all of your data whether it's you know structured unstructured and you can just just can't have your analyst or scientist to deal with that kind of data physically stored data right so if you wanted to do something like bi you know you still needed to organize the data and get it out in a form uh that is basically understandable by these different various sort of analytical tools and then you can do your reporting analytical you know use cases around that and the most kind of like the most important like uh pinpoint thing that led to the evolution of the architecture that we're going to talk about today is that data League that they lack those kind of atomicity and consistency guarantees with transaction so you know anything that you want to do whether it's updates deletes even reads uh you know you cannot do asset based transactions uh on the data leak you know for example like two concurrent writers are trying to uh write on the same table at the same time that's kind of like tricky uh you know you have those like you miss out those guarantees you know you don't know sometimes partial updates and those kind of things uh also it was getting really hard uh you know I have worked in various roles in around this with data Leakes you know it was getting really hard to make sense all of the data right because you know uh there is no real governance there is no data quality checks you know usually you know that kind of a data Lake environment is like you know a swamp environment um so really how can we build like a you know prediction model to like uh solve something uh if we do not have that trust in the data or you know how can we you know rely on some kind of analytical reports if we do not have you know that kind of quality or governance or data so this gave rise to a kind of a new architect platform called lak house and you might have guessed it right you know it's kind of like you know the best of the both world so you take a data Lake uh and know take the best part of it and then have a data warehouse data warehouses have great data management capabilities uh so it allows you to take those kind of best abilities and you know um form a new form of architecture that provides most importantly flexibility for different customers different use cases so uh this is more of a visual picture um I like to you know see things visually um you know here are a couple of technical components involved and that we going to discuss today so I'll start from the bottom up and then we also do kind of a comparison to like existing data architectures like data warehouses and you know going beyond that kind of marketing jargon you know leak house and like you know how do we relate to a more technical component like a olab database so obviously on the bottom you have the data Lake storage this is where the data lands after ingestion from operational system from transactional systems uh you know Azure blob or know S3 bucket whatever it is or it can be something like a Hado file system as well so basically it supports storing any kind of data and facilitates with like performance and security and those kind of aspect so that's kind of the bottom layer where we have the storage and then we have the file formats on top of and this is where the actual data is going to be stored right so you know mostly you know in the recent world we have been using columnar formats like Park U you know wherec which basically provides advantages with like uh for example the metrics right you know you could basically use those metrics to prune your U you know use the metrics for pruning your queries and you can basically take advantage of that kind of thing um now the most important thing that lies on top of that is basically the table format the leak house table format and and we will learn a bit more about table formats in the next slides but in general what a table format does is that it provides an abstraction you know it provides that kind of abstracts that physical data structures complexity and allows different engine to basically work on the same data on the same table so Apache Iceberg you know Apache hoodie Delta Lake are kind of the three most popular table formats today uh and are kind of get gaining the widely you know Enterprise adoption uh but obviously this is new not new at all like you know in the past we have had the hive table format Hive cap out of like Facebook at that point of time and basically led us to you know Iceberg and today actually one of the main motivations towards building something like Iceberg was the hive table format and the problem Netflix were dealing at that point of time with Hive table format and on top of that obviously we have the computer engine you know we need the engine to basically crunch the data so they're responsible for processing it you know and depending on the task uh depending on the use cases you you can use something like spark uh trino or flank or even you know most recently you know some some of the popular computer like duct TB you know you can basically plug in there uh recently had a support for Iceberg as well to read the table at least and you know so really really like depending on the use cases the idea is to bring your own computer engine and connect to the table format versus the other wheel and finally the final component of the lak house is basically obviously the analytical tools you know that are responsible for like you know Downstream um you know analytical applications like bi reporting AML based tools as well and and these tools are the idea is that you can connect to these tools where certain systems like you know in the past we have used obbc and jdbc but more recently we have seen use with like something like Apache aof flight you know flight SQL that gives you the semantics to query those data and be able to deal with it in a more efficient way so one question that always came to my mind when I was kind of getting into this kind of Lake housee architecture understanding a bit more technically was that how does it compare to an existing data AR architecture like like a data warehouse right so the main distinction between uh data warehouse architecture and something like this is that each of these components that you see here they're no different they're no different in a data olab database or data warehouse basically the storage that you have the table format the file format the compu engine they are all the same in a data warehouse environment as well the only difference here is that here you have you do not have each of these components bundled into one single system whereas in a data warehouse these components are bundled into one one unified system so that that brings us to like the question okay what is the difference in that so the difference in that is that these data warehouses have been like using table formats and file formats theyve been building their own table and file formats and they are optimized really for their own Compu engine so that means once you load a data into their own table and file format it's really hard to get like you know bring a different engine to that same kind of data and table format and you know uh plug into that so that means in a way you are locking the data to your own environment uh however you know in the recent days we have also seen a very good move around that like you know cloud data we vendor like Snowflake and bigquery uh they have started you know supporting external table formats like Iceberg you can basically you know um you know manage them as their own way uh and that allows you to uh that allows customers like you know to deal with various kind of use cases that they were kind of uh you know restricted to in in the beginning so um I recently also published a paper about it um you know and in general you know that's the idea like just to establish like how it compares to like data warehouse technically and how this relates to a data warehouse and obviously like there are on top of that the benefits out of it like we talked about like the scalability you know the data management capabilities that we kind of see now so what is the table format uh so usually with databases we have obviously seen the concept of tables it's not a New Concept at all you know like vendors like Oracle you know have been doing it or Microsoft have been doing it with relational databases uh basically it's a way to organize uh you know data sets file to present them as a single table so users don't really have to know the kind of the physical bytes and the files you know and they can just have that abstraction right on top of that uh and in the past like I said vendors have had their own kind of storage engine that kind of handles the data uh on their on on the files on their disk and the execution engine interacted with their own files right uh now in the Big Data world uh with Hadoop and hdfs we ended up ultimately with the hive Hive table form it now Hive like I said it allowed you know more users to be able to uh get get hold of the data that was stored in a file system like Hado file system so in a way it allowed the democratization of data using something like hi query language uh which is pretty similar to SQL uh so with Hive uh the way it was designed is that with a hive a table is basically a set of directories uh you know and each of the directories will have the files which are basically the file that we the engine has to deal with right and so like you see here it's it's a table with you know a couple of like the folders here and you know the folders have their own files and the tracking is kind of done at the folder level with the hive table format so let's talk about it in some more details um in the hive table format like I said you know the tables are one or more directories you know obviously we see here one you know one of them or a couple of them but at scale this would be like a lot of directories right so this has been kind of the defacto standard since like the M adoption of Hadoop and you know um so definitely the it being the kind of the defacto standard until now at least you know I know a couple of organizations still leveraging on hive that has some advantages towards it and of course we are going to talk about advantage and then slightly move on to the problems that kind of Netflix face around them so one of the advantages was that it basically you know works with every engine right you know until like recently there was no other table format that allowed you to do that uh so that was great uh secondly it had more efficient access pattern so for example things like partitioning schemes you know and mapping to that column you know bucketing and so you don't really have to do uh Full Table scans for every query that you run right so that was great it was also file format agnostic so that means you know you can use any kind of file format whether it's Park CSV doesn't really matter you know it's agnostic towards that uh and the most important thing is that it it could atomically update a whole partition so for example you can basically take a you know partition change the data whether you want to do some you know insert some records or delete or upserts and write data into a new Partition and when you're writing it uh there is uh no one see the changes right no one sees the changes and then you atomically swap that reference of the you know the pointer uh in the Hy metast store and this kind of gave us the Hy metastore right and allowed us to deal with things like atomicity uh consistency guarantees Etc so the Hy metastore provided kind of a single centralized answer U to what's in the table like if if I as a you know user or you know any kind of query engine comes to the table the H metast store basically allows you to basically give have a single centralized answer and kind of every you know user every query tool got the same kind of answer but obviously like this kind of advantages also brought us to certain caveats and you know this is not me throwing shades on the hvep table format at all obviously like this is a a great inspiration to what we have today uh but some of the problem that I also personally say s saw during my work life was that you know the smaller updates were kind of very inefficient uh because you can only atomically update U you know whole partition if you only wanted to change a few records and a more number of time you know that's very inefficient with hi table format um then you had no way to change data in multiple partition safely so because no because if you want to insert let's say two a row in two different partitions um you know you cannot possibly do that with atomicity guarante in Hive same things with multiple you know jobs doing the same data set you know if there are multiple jobs trying to you know access the same data set and write do things like concurrent rights it's really hard to do that um most importantly the in the hi table format the files are under the folder we we kind of like established that right so engines basically are not going to read the folder they're going to have to open up each and every folder and then open up each and every file and read the file close it and imagine this doing at scale like this could be like a a lot of time taking task another important aspect of like where Hive kind of lag was that user kind of had to know the physical layer of the table and that's kind of a really like a cumbersome thing to deal with like for example if you have a table Partition by day and if your query has something like a time stamp Hive really will not know that kind of trans like mapping H didn't have that mapping so unless and until you specify that partition column that you have explicitly defined for your hive table format um your users will basically if you run a quer you know you know pass a query they will basically be doing Full Table scan of the query of the table so ultimately you know it's also a matter of educating people like that's what I also realized that you know let's say for if a table is like like I said if a partition by day why do users really need to know about that kind of like okay 2022 SL you know 08/01 why do you why do they even need to have that kind of like knowledge around that and most importantly the table statistics and hi will kind of often stale like you know we have realized that you know in at scale you know the hive allows you to basically gather this kind of you know statistic in a kind of asynchronously and these kind of read jobs are very expensive at scale and to collect the stats we have to run this job most importantly and you have to schedule and do some maintenance around it so they are hardly run you know in practice but this table statistics are really efficient you know in able to like you know query planning for query planning and pruning the data and accessing the data in general so these problems are pretty much common like you know at every organization at scale you know it is are not new at all and Netflix were kind of the one to look at it and they went to the drawing board and you know they thought about it and then they decided that instead of putting a Band-Aid to kind of each and every problem of this can they really take a step back and understand the actual issue because in a long term it was going to be really hard to you know manage it so they had this goals they wanted to achieve and you know these goals are again very similar to every organization that you know we want to achieve it so one of the first goal was around table correctness and consistency I you know talked about it you need to have the trust in data you know otherwise it's really you know it it it becomes difficult to make that decision and you know irrespective of the query engine that he use all the queries should return all the same result uh next is around faster query planning and execution so one of the query that took them basically 10 minutes to plan a week of data for just one metric it's just one metric of data for a week of for a week's filter and you know it took 10 minutes to uh basically um you know query that data and users are not really going to have to wait for that you know performance is really a key around that uh next we talk about the physical layer of the data and we understand that you know it's super important to have that like with partitions even if I have to query using a date you know user should not be having to worry about that so you know we can just have the software take care of that uh there were also a lot of problem with table Evolution and this is something we usually see right you know tables grow over time we should be able like you know business requirement change you know we should be able to be able to adapt to new requirements that comes in the table and be able to evolve the schema without doing things like you know for example rewriting the entire table that's a very expensive thing to do so you know these kind of uh it was an important aspect towards them as well and to accomplish all of these at scale you know if we can we can take care of this probably for a smaller data set but you know imagine dealing with like a huge parab table data and it only grows so it's really hard to deal with this at scale so when looking at this kind of the Band-Aid you know problems and intermediate Solutions you know all these cons they went through you know they realized that they could solve this using a pretty simple solution so the solution was that instead of tracking the you know the files at a folder level they just get rid of the directory structure and you know track at the file level so that is the only difference that kind of exists uh between like table formats today like Iceberg uh versus Hive and that's really the core like you know that basically you know set the Precedence for that so uh this is where Iceberg comes and you know kind of addresses this problem um I'm going to talk about what iceberg is and is in cuz you know it's better to have that context because we have seen that with Hive you know it can be quite nebulous and mean different thing to different people so what iceberg is uh it's obviously a table from its specification it's a way to lay out those bytes uh and in the form of table giving that abstraction on top of the file formats uh you know like perk uh so it's a set of libraries and epis and basically if you're execution engine like spark you will basically have to leverage those apis um in a more technical sense and be able to you know interact with those libraries it's not a storage engine it's not an execution engine or any kind of like a service of any form so I'm going to do a kind of architectural Deep dive and go through like how iceberg is like you know developed and how it's set up architecturally so as you can see in the IM image here they are basically three layers of you know at a high level for Iceberg so we'll go from like the top to bottom and you know understand the various components the first is the kind of catalog layer which is basically a super important component and Iceberg needs this component if you're trying to deal with things like uh atomicity and consistency guarantees uh the second is the metadata layer and you know the the metadata layer has things like the metadata file um the Manifest list the Manifest file I'm going to talk a bit about what they are as well and finally the data layer this is where the actual data is going to be stored these are perk files or or files um and this is how the overall architecture looks like so the first thing here is the iceberg catalog and uh you know so if a user is trying to basically you know read or write data for a table uh they first need to know where to go and catalog is that place so in a more technical sense this can be anything like a data store a database the only requirement for something to be a catalog in iceberg is that it should have um that you know it must be able to support that kind of atomic operations right you know basically what we basically what we want to achieve through that is that once a query engine writes something to iburg table it swaps the reference of the version of the metadata file with the latest one so for example if version one is the current metadata file and if I do some upsurt I'm going to as an engine I'm going to change the metadata file in the catalog to version two so now that allows me to get basically all the latest St so for example now if I bring any other query engine like trino to read that data I will still see the fresh set of Records because my version is now the version 3 which sparkk has already basically reiterated and changed it so that's the overall concept with um you know catalog that you can basically store as much of data as you want in a data leak storage like S3 but you need some kind of a locking mechanism for those atomicity and consistency guarantees so most data leaks they don't provide that you know and Iceberg catalog they serve this purpose so the catalog can be a lot of things like I said you know it can be technically any data store um you know Haru file system as well of course we don't recommend using the Haru file system as a catalog in production because we have seen losing those kind of guarantees when we're doing uh concurrent rights and stuff but there H metast store is one of the commonly Ed if you're in the AWS ecosystem maybe AWS glue is a very common like catalog as well uh or there is something called project nessi which is open source as well uh that allows you to get you know deal with things like you know a git like code so for example you have a production table you can create a branch of it just like a GitHub branch and you can basically create that branch in isolation like a local development environment and you can write your ETL job or whatever it is in that particular environment and no one SE sees that when you're doing that and once you're done with that you can basically say hey merge this particular Branch into the main branch and then you make the data available in production so that's where things like you know write audit publish and you know those kind of guarantees come into the picture where you can do more um you know detailed data quality checks uh if I may and like I said in a more technical sense what the catalog is storing is the version of the the current the location of the current metadata file like okay this is my version one and this is where that location exists so let's say if you're running a select quer like a select St from something with some limit uh you basically go to the catalog and say hey give me the current met file for for this particular at this particular state of the time and it gives you that particular location so once you get that particular location I have numbered it with one you go and find that particular like the snapshot and you know you get the current metadata file from that so the metadata file as the name suggest it's a metadata about all the tables you know certain properties like schema like partitioning specs and those kind of things so you see obviously couple of the contents here I don't know if it might be not visible as well totally but in general what we are trying to have here is things like snapshot is a very important concept here obviously snapshot means the state of a table at a given point of time so basically whatever operations you do like creating a table you know writing it any kind of insert upsert everything is captured as a snapshot in iceberg uh and finally from that particular current snapshot ID so basically in the catalog you got the location of the metadata file let's say this is the version one and from that metadata file you got the current snapshot okay this is my current snapshot and using that current snapshot ID you can also see the Manifest list Lo so that has a lan age towards the Manifest list so it points you to that manifest list and brings us to the Manifest list Pro so what is Manifest list basically manifest list contains information around things like partition most importantly so you can do like at a high level you can do things like partition pruning here so you don't really have to you know go and read data from each and every partition you can basically you know read the query engine can read the Manifest list and it can get information around that and from that it can basically find the Manifest file which is what we are looking for for the next step of our uh you know Journey so the Manifest file uh is it's at a lower level it's tracking those data file like those par or the or orever whatever file is that they're tracking it at that particular level and obviously you know we have information such as the data file part U and most importantly the statistical information that I mentioned about like for example Park gathers those park has that information in the footer right you know the lower and upper value bounds the N value counts and stuff but instead of like going and opening each and every perk file what Iceberg does it does second level of aggregation so it basically Aggregates those statistics into a manifest file so that you just can come to the Manifest file and you know basically prune your data files based on the Manifest file so you don't really have to open and know take the overhead and the cost of opening each and every perk file even though you know that you can do that and you know take advantage of things like you know pruning um so this may seem like a lot but you know at a run time this is way faster of course U and you know we can see more about it as well so I wanted to run through a couple of the queries that are you know very basic ones just some couple of writes couple of reads um and then we are going to you know see how the iceburg architecture look like how the system look like in the data L as well so let's say first we create a table uh it's a simple create statement and you know we have couple of fields here we have partitioned by order of by hour of order time stem as you notice here we have we haven't introduced any new you know partition column here the order time stem is part of the particular you know table right the only thing that changes here is the our transformation so Iceberg it allows you to do of transformation and keeps a track of this transformation from here so on the right hand side you have the architecture of Iceberg on the left hand side you have the catalog and the file system the whatever it is S3 or whatever file system it is so as you can see here because I don't have any data inserted at this point of time you just have one metad DAT file that's why you don't have the Manifest list the Manifest file and the data file because obviously we haven't written anything towards it and you know that gives us the ability to yeah see that and basically now we have one version of a snapshot that is the current snapshot so now let's go ahead and create or like insert some record into that so if we insert some records here now we'll see the the tree of the iceberg that we see because now we have inserted some new records so now we have a version two of the metadata file as you can see there's a different metadata file and you also see that it has two Snapshot in that metadata file right there's a snapshot zero that was from our previous create Table and there is a snapshot one that is based on our insert statement so iberg keeps a track of all the older snapshot that brings us to the question okay what does it mean for storage you know at a point of time it might be a lot of storage like the snapshot so obviously you know there are maintenance procedures to do that if you're using something like spark you can call a procedure and you can schedule your maintenance to like get rid of all the older snapshot so you know this is basically used for things like time travel right because you want to time travel to a particular point of time and you want to leverage those use cases and finally if you see here uh the cat catalog if you see now the catalog is updated with the version two of it not the version one so because whatever engine use use it to write the file basically change that reference to version two from version one now uh I for the lack of time I'm going to skip the upsert but it's kind of the same thing you know you do upsert there's a different metadata file now we have a different snapshot now and you have a different kind of tree but ultimately The Thing Remains the Same one thing that is important to understand here is that with upser on the left hand side if you see the the files here we have some grade out files and we have some newly written files in a new Partition the reason being that is because you are trying to basically use a strategy called copy on write and you can also leverage something called merge on read which is also very efficient for streaming basb load so here basically all the new upor all the new updates are basically written to a new file instead of the older file um so the read statement follows a similar way you go to the catalog get the current meta file we talked about it we get the current metod data file from that we go to the Manifest list and find the exact data file that is required for a query to serve the purpose uh time travel Works in a similar way you know the only difference is that let's say you know for in our time stamp you know we have specified a particular time travel you know field and in that now because in the metadata file Iceberg keeps track of all the snapshot in the snapshot in the metadata file it will keep uh it will have the time stamp that we specified in a query and based on the time stamp it basically go through the same route and find out so the difference that you see here is now that we are not fetching the latest setup record it went to the snapshot one and you know pulled out the other set of record that was from our insert statement before the upsert that we did so again I'm going to skip this this was basically you know link um Netflix basically they had this time that was taking 10 minutes to just you know plan the query and with Iceberg it took like them basically 42 second wall time just to execute the query overall uh so that brings us to the main question how do we choose one you know we have like discussions today with different table formats like you know we have Iceberg hoodie Delta Lake as well um that are really greatting adoption uh so the decision to choose a specific table format is very challenging and can be can be involved in a lot of cost like I was personally involved in a lot of this discussion in the past year uh you know one of the main thing that we' have come to realize is that obviously it comes to like feature level comparisons complexity of implementation how easy it it is easier to get started and scale it right and of course the main thing is the support by the query engines and the ecosystem overall so if you're in a you know like in a Google environment you might have a different set of computer engine so you want to be able to get support from these different environments and with newer workloads we are also realizing now that these table format needs to be inter interoperable one of the customer that I was talking recently is that they have these two different table formats that are they using and it's kind of has also led to kind of a siloed environment and they want to be able to to leverage them so that brings us to the focus on interoperability can we make these table formats interoperable um so the question is really do we have to choose or there is other way and there is another way uh because you know at a very fundamental level if you see this metadata structure they're basically a metadata layer on top of the file formats right and all these different table formats like hoodi has this hoodi metadata folder on the par file Delta has the same log file and the perky file Iceberg has the same kind of thing so ultimately it's an abstraction on top of that so basically you know it's easier to be able to take those metadata file and translate them into another format and that's another project that I have been working on and with you know set of commuters from my organizations and Google and Microsoft that's called one table that basically uh we released today and it has focus on interoperability so you can basically the idea is that you can take a particular Source table format like you can write in your particular choice of table format like Iceberg and depending on your use cases or your choice of computer engine you can translate them into one or more formats um that's the idea behind one table so one table it's omnidirectional the the reason I specify on the term Omni directional is that data brakes also release something called one uniformat and that basically takes and works in one single way whereas one table will focus on like you can take you know go from any direction to the other direction so it really isn't dependent on one way of doing things so uh the idea is that you can take one any table format of your source and you can see the spark you can read it in different different table formats uh so if you're interested in this particular uh open source project it's very new you know if you're interested in contributing or applying it for your use case as well this might be something interesting so you know give us um a look on that and you know we are also plan planning to incubate it as Apache project because we have been involved with various contributions from Google and Microsoft today and we are seeing more active contributions from Apple Adobe and other companies today as well so that is pretty much it that I had today and thank you so much for having me I might take any questions that you guys have thank you I know I exhausted that I'm sorry about that no no worries you're you're still on time so we can take a couple questions before um yeah kind of going for lunch thank you for this talk uh my question is um does any of the table formats that you described uh support the multiquery transactions like where you begin the transaction then make a couple of updates then commit or roll back it all together yeah so quite recently the iceberg with with the res catalog specification it's a it's a specification that allows you to you know use something like rest API that is something that is being worked on that I know of that allows you to do kind of like if I'm not wrong you're talking about multi-table transactions right where you have you know a couple of transaction table that are dependent on like for example fact and dimensions and you make changes at the same time and you expose to the new production environment right so that is something that is being worked on Project nessi is something super interesting as well like you know it's it's open source project that allows you to do that but by default hoodie or Delta L doesn't allow that if I know it but I know hoodie something is being worked on on the hoodie side as well that's a great update thank you of course thank you yeah anyone yeah right thank you U Can you say a little bit about all the different indexes we have from hoodie to uh Delta Lake you know data skipping Global index and so on related to and and Iceberg specifically where it's lacking right so Iceberg doesn't have any concept of indexing at a high level so hoodi is one of the table format that really brings out those kind of Rich indexing you know experience uh but Iceberg basically you know tries to deal with the Manifest files that's the way they took it it and so basically you do not leverage an index to do that whereas hurri is more dependent on different kind of index like Bloom filter you know simple index or those kind of things whereas that's something that is kind of like as far as I realized that you know that's something that is not and present in the iceberg environment and I'm not sure about the Delta you know space as well they have data skipping as well right it's it's the same kind of not all of it's open source unfortunately right yeah yeah but yeah hoodie of course has a more diverse way at the beginning you talked about uh small updates issue right of course U I'm not seeing small update issues as a data Lake issue in general okay but uh I'm not clear how you are solving that based on the presentation so far yeah I think the whole idea behind this is that for example like upars right and those kind of thing right now you don't have to like be able to like you know for example like I talk about like how you have to basically atomically Le we have the reference every time you do those kind of updates now you can do that without doing that right with Iceberg or any kind of table format so basically you can write the data and this copy on write or the merge on read update so basically you can use this technique and balance this kind of write and read latency trade-off so for example if you have something like Flink you're writing quite you know frequently right so you can basically use technique like merge and read and you can what Iceberg or you know whooody does it in a different way as well but what Iceberg does it basically merges these things on the read side versus the right side so that kind of trade up is being involved in this kind of new techniques I see thank you of [Music] course