SBTB 2023: Julien Le Dem, Ten years of building open source standards.

SBTB 2023: Julien Le Dem, Ten years of building open source standards.

Recording: SBTB 2023: Julien Le Dem, Ten years of building open source standards.

so I'm Julian um this is going to be a little trip down U Memory Lane I'm going to talk about past 10 years of building open source projects um and in particular open source standards so I structured this in three chapters um starting with talking about Park and then arrow and then more recently about open lineage and then each of them I'm going to talk about three lessons learns about how open source Community Works how you can Define standards in open source and how that all works so this is what you're going to see so first uh let's talk about the birth of par and um a little bit before that um so 15 years ago I was at Yahoo uh back in the day working using Apache Pig so I don't know if you familiar with Apache Pig who used Apache pig in the past not not oh yeah there's some people um that was nice uh so you know it's a bit on the downward trajectory it's been replaced by spark mostly for what he was doing and so at the time I was using pig as a better map produce right instead of writing XML workflows to orchestrate my produce jobs we' use Pig which was much better um and I started contributing to it became a committer that was my first committer ship on an Apache project and that's also connecting me with the wider open source Community around it which included uh Twitter and the Twitter infrastructure team and that's how I got my next job and so working on Pig you know I got to all the levels of moving from contributor to pick to committer to PMC member and being the PMC chair for a year as we were rotating that role role and so going through all the steps and all the experience of what it is to run an Apache project uh so was a very um good experience for me and learning all of this and also around the same time that's when the Dremel P paper came up and I first read it you know it's kind of part of good data engineering and keeping up to date of what the competition is doing um and so then I was at Twitter right and so that kind of triggered the burst of parket of what we had at Twitter so on one end we had Hadoop that could store a lot of data you know it's kind of store a lot of machines at it store a lot of data uh having high throughput you can process a lot of data but it's also High latency right so you would start your job you would go get coffee and wait for the result so you could process a lot of data but it was not particularly fast on the other hand we had vertica when the MPP U massively multi a parallel um database and you can have interactive queries it's SQL it's faster but it doesn't scale as much you cannot store as much data it's it's more expensive and there was always this chicken the neck problem like oh can we put the data in vertica what we can only put the most valuable data how do we know it's valuable well we know because we use it but we can't use it until we put it in there right so they it's kind of always that friction so basically it came down to how we make Hadoop more like vertica right so on one end you have Hadoop which has two big components um the map reduce processing uh on top of a distributed file system right so you have the things that's really good at scanning and doing a shuffle across uh the network and then riding back on the other hand you have the design of vertica which has a columnar storage underneath for efficient data retrieval and making a lot faster to get the right data and a distributed cor engine on top right that can execute SQL and not just map produce so how do we learn from vertica and um to build make Hadoop more like this right like how do we have more primitive uh primitive Primitives uh here so at Twitter we have this nice paper reading group right we' meet every week and we kind of shared of papers that were interested there's lots of papers that go around but how do you as an engineer you know you can't really read all of them so how do we pick the few that are interesting and maybe some of us read it and share what they found so we did a lot of reading about there's some research paper around vertica SE store how it was built why there's the dremal paper there's the monb paper that was also a seminal in the vectorization how what are column stores uh why do we need vectorization and all of that stuff so thank you about this architecture and you know how some of it can trickle down to Hadoop and making less you know like this blunt tool to process data into something a little more uh solo uh that can benefit of the learning of vertica so one of the key things right is the columnar layout and that's one of the key design in park um that's what it provides the columnar layout so when you do a raw layout you just put each raw one after the other one and so you interl data of different types and you know if your column A is a string column B is an integer C is a date you're going to interally leave different type and they go one after the other and it's assuming you know you kind of reading all the data every arrow one at a time in the column layout you put all the data for a given column together right so in the end when you store on disk uh it's a linear storage um so you can of you have this two dimensional table but it needs to be stored in one way right so in column layout you put all the data for a given column together which gives you everything of the same type it's more homogeneous storage so you have lots of opportunity to encode it you know if it's certain integers that fit in a certain range you can maybe go do bit packing or other techniques uh you have a string and there's lots of repetition you can do diction encoding so there's lots of opportunity to do cheap simple transformation that will compress it better and also when you retrieve data if you're let's say you're doing a SQL and you're selecting only a subset of the columns it's a lot faster to retrieve them because it's a lot more efficient for uh skipping the data you don't need right so you're reading only the columns that you care about so those are some of the benefits and why you know columnar St storage was invented because it's a lot more efficient to retrieve the data so there was this discussion why okay so how do we start making hop more like vertical one one step is to make the storage more columnar make uh data retrieval more efficient uh and so this red Elm project came as looking at the intersection of a bunch of existing file format RC files t files CF Trev uh that were existing that kind of hinted at looking at this but were not like true columnar um state-ofthe-art implementation yet the schema systems we're using at Twitter with Thrift and the pig project and like the the findings in the dremal paper and in the dremal paper you have two main things one is the columnar representation of NY data structure and the other is a coronin on top of it and so that's how you know started this Dremel project this redel project sorry in August 2012 um and I kind of pass following this paper reading uh exercise as I started implementing the algorithm in the Dremel paper and the bus ride uh to a from work so that was my side project and um the first comment so I just dug up to know when this happened so it's a little bit more than 10 years now we're a bit you know that's the second time I give that talk sorry about that um and um it was quite ambitious because if you notice red Elm is an anagram of Dremel so the the idea was to build everything um but we kind of resop the project after a bit and then to kind of something more reasonable uh and at Twitter old projects were named after Birds but I guess red Elm is a tree and birds live in trees so that was kind of a good alternative to change a bit from that and so really looking at this I knew I didn't want to R Implement all of this on my own that was not quite doable and also living in this open source ecosystem you don't want to reinvent a storage layer and have to implement it with everything you don't really want to build proprietary things like this uh you want it to be open source you want to find Partners you want to be part of the ecosystem so that we all work together on this and it's actually doable right it's not just this giant Endeavor of reimplementing uh Dro from the Twitter infra team um so I kind of so that's was kind of my tongue and cheek uh way of reaching out on Twitter and finding people and the reason I know there's an error in the figure five in the Dremel paper is because I used that as an example from a unit test and my unit test were not passing I was like I have a bug and no there's a bug in the paper uh so that was a fun find and so that's how I connected with um the Impala team at clader because of course we were looking at okay okay like how do we make Hadoop more efficient you know and less of a blunt tool and we're not the only ones looking at that right so the cladera team the Impala team at cladera was working on this columnar format and uh they had a prototype I had a prototype and we just join efforts so on my end I was looking more on the jvm ecosystem you know Hadoop was very much in the jvm ecosystem uh integration with pig map produce all like building NY data structure on their end they were focusing on building a native code implementation SQL and Hadoop thing um distributed cury engine native code generation with uh llm sorry llvm sorry mixing my acronyms here um and so we merged designs and that was great because on my end I was bringing all the interoperability with the jvm ecosystem on their end they were building an actual cor engine on top of that so that fit the design to make sure the design of the format was uh going to be good for building a efficient cor engine on top of it and that became paret right we needed to okay now it's not we not reimplementing redon where like we building this file format so par seemed to be a good uh analogy uh for the bottom layer of the thing uh so that's what it became uh and so from that you know looking a little bit at the Timeline we started from uh the first commit in 2012 then we work together with cladera um and then um we did an announcement in two 2013 uh this entered the Apache incubator the next year uh along the way there bunch of adoptions spark SQL was a adopter of Park alien when they built spark SQL on top of sparkk they needed a columnar store as well and so um this became a good collaboration Apache drill was also an early adapter um Netflix uh implemented Park support in Presto that came out of um um um Facebook that same year and then all the way to the graduation you know in 2015 becoming a top level project uh collaboration for with West mackini who worked on the C++ implementation uh when he was at cler at the time and then you know if we fast forward now every major Warehouse supports Park is kind of everywhere so it's kind of magic at the time so we put all the effort to make it happen but it's still mind-blowing that you know par became um the defo standard for storing data in a lot of cases so lesson learn from this um so when you're growing this project right it's important important every contributors become a stakeholder right if you want to grow the ecosystem if you want people to keep contributing if you want to be the project bigger than just what you can do on your shuttle ride to and back to work they when people become a contributor to the project they own a piece of it right they're not just like contributing code and moving on now you you take responsibility you own part of it we're building this together right like and it's been really important in building parkage you have co-conspirators right like with Wes mckin and Ryan blue and a lot of others to lipcon uh ngi um a lot of them from cladera or uh Netflix of or other companies and really building this together right so it's a strong part of it they're not just contributing to your project we they become stakeholders they become part of it and they own a piece of it and you know with great power comes great respons ibility so I think that's one of the key factor how you grow the project well you share ownership is an important part and there's a real Snowball Effect right like often people say oh it's like you want this to be adopted everywhere it's like boiling the ocean or or you know it's like moving mountains is just like it seems undoable but there's a real momentum happening right at the beginning you push a small thing you're not trying to Bow the ocean you just do a little bit and then it keeps momentum of its own right it the more momentum it gets the more it it goes faster and there's a real acceleration factor and that's the thing that you don't have to carry all that weight on your own right it just gets momentum on its own and you just need to push it enough that it uh benefits from this uh ecosystem the last part that I wanted to talk quickly about in this first phase is understanding about what open source means right because because you hear people talking about open source people have lots of opinions about what open source means and the most basic definition of it is that um you can read the code the code is available but by default that gives you no right to do anything with it so the first thing you look for is what the open source license like and in there there's lots of licenses so the Apache license or the MIT licenses or those type of license are permissive licenses basically they say you can do whatever you want with it you cannot hold us accountable if something breaks because you use it um and the only constraint is you cannot pretend you wrote it you need to give credit um and if you change it you need to call it something else right and you say oh it's derived from this project U and then there are other license that are less permissive like GPL basically if you build anything on top of it it also needs to be GPL it's also needs to be open source so kind of force people if you benefit from open source then you have to give what you build on top of it uh which is very different uh constraint and then governance you can have an open source license and still you don't know how to contribute or take ownership in the project or maybe you cannot right it's owned by a company and it's fine um you just need to be aware of it right what's the governance like so if it's a uh something you can become a maintainer of or not and it's just you're only ever going to be a contributor if you're not part of this organization that owns the project uh and the last thing is foundations uh like the Linux Foundation or the Apache Foundation who uh ensure that nobody owns a project I mean the foundation does um but nobody is going to control it is going to stay neutral and I think for those type of standard project like parket Arrow op lineage it's really important to be part of a foundation and it's a testament that it's always going to stay neutral like everybody is going to be on an equal footing in the project so going a bit on uh Arrow as part of this discussion about building this columnar storage format right like the bottom layer of the columnar store that it came in the same community that we needed an inmemory store uh on top of it like in for different reason right that CES more from the vectorization monb how you make a vectorized Cay engine so we also needed a standard uh in memory format um and that's because modern processors don't just execute um exe um instruction sequentially there's a pipeline where they kind of predict what the next instruction is going to be and the memory cache is also very important when you need to keep the data in Cache to be efficient because it you lost lose a lot of performance every time you need to copy data from the main memory into the processor cache so all those reason makes colar presentation vectors in memory uh a lot more performant to execute instruction um and so and I highly recommend reading this seminal paper to understand better about it so what we did is we started with okay let's find a group of people who care about this and bootstrap the project uh what would the uh spec look like and it happened that the drill Apache drill project actually already had uh inmemory columnar representation that looked exactly like what people were wanted so we started from bootstrapping the project with that and spinning off the inmemory columnar representation from the drill project into its own project that became arrow and that focused on this fast query execution so even if it's also a columnar representation it optimizes for very different things than Park Park focuses on fast disc retrieval and projection predicate push down and atress compression when arrow is focusing at vectorized execution zero copy transfer and fast in memory processing for the Cory engine um so that's kind of the initial requirement and then you see so this timeline starts right immediately after the end of my previous timeline in 2015 right so bar graduated and then there was a kickoff on reement on the aror project it directly became a top level project out of Apache drill and then a lot of collaboration happened as well like Arrow uh was in pendas where the paret C++ implementation is tightly integrated with the arrow C++ integration they just work together it's basically the same Library um to get very quick pendas integration integration in spark and nowadays uh now also all major warehouses uh support Arrow as a fast interchange um in between of result sets um you have things like dug DB that uh make it very popular to have vectorized execution on your machine and uh uh it's interesting to see the full circle right we started from these big databases uh that we do on one machine things we mve to distributed databases to scale that up and now we're back to uh single machine ver is high parallelism and uh do processing really fast on the really parallel machines uh with fast in memory processing so lesson learns from this part so I talk about this you know Snowball Effect but really when you have an exponential curve at the beginning it looks really really flat right at the beginning the ecosystem grows very slowly and then suddenly you know there's an inflection point in accelerate sorry um and that thank you for the water bottle and then then uh what you can do at the beginning is really bootstrap with the bigger Community right Reach Out leverage we l leverage the fact that there were a lot of open source project companies connected together around the development of paret to bootstrap eror with a bigger community and you know for Park we can stumble onto it with having two contributors with uh Twitter and clader at the beginning and then slowly growing getting Netflix and crto and um the spark people to use it and grow around it but for Arrow it was more like thinking about okay like we need this collaboration early on and how do we make it happen faster and through that you can find like-minded people right to who will Drive the vision with you right you don't need to convince everyone that this is a good idea you just need to convince a few of them and you convince a few Visionaries right that will oh see the vision see why it makes sense help drive to what this they will convince early adopters and the early adopters will convince uh the other people right so there's kind of a scalability aspect to this because it's always you know feel like boiling the ocean but this's really like a network effect that happens um and that really um enables this growth and it's you know and it's all about the connection we built along the way right so it's I'm describing that like as if there's an evil plan this is how we're going to take over the world but actually you know it works because we enjoy the process right like someone was telling me are you're doing open source the right way and I think in their mind they were were thinking the hard way you know where you collaborate with a lot of people and you need to get people together you know this I'm cheating because I'm actually really enjoy this I really enjoy you get to work with amazing people that you wouldn't get to work with otherwise and it's great to be able to build those relationship collaborate in open source you know nobody has to be there they're doing it because they they want to uh and it's really a rewarding uh mechanism so um on the last part um moving with open lineage um so there's you know if we think okay now we solve the data interchange uh from disk um to memory and from memory to memory aspect what's the next other Pro big problem that exist right and I think this the big data is one problem like scaling to data but the big data organization is the other problem like people depend on each other within each team they know what they're doing or usually they know what their teammates are doing and how they depend on each other but across teams there were a lot of friction happens right because they're consuming data and they have no idea where it's coming from they have no idea who's consuming the data they produce and whenever something a lot of change things change all the time and um that creates a lot of friction I my data is broken because someone changed the schema I was not aware of it and all these kind of problems and so often when you see there's this I've built that that data hierarchy of needs you know there's this Mass lows hierarchy of needs is if you're looking for happiness of becoming the better form of yourself you can be first you need to solve basic needs right you need to be fed you need to have shelter you need to have security uh and then once you have those things then you can build on you know being a good person and building things and uh having an impact on the world so adapted to data like before you can do anything right and we often scramble in all the noise of things are broken constantly first the data needs to be available and then it needs to be fresh right it needs to be like up to dat and then it needs to be correct and once you have all those things then you can keep your head above water and you can start saying oh maybe you're working on your business optimization reducing cost on something or finding new opportunities building llms or whatnot and building new products but most of the time we have our head underwater right and we constantly fting well the data doesn't show up or it's showing too late or there's a bunch of nulls in the data and it's just like this is a lot of thrashing that prevents us from achieving Higher Goals and so that's where you know like moving on at the time so I was working at weor and uh so you know in this phase where um we're do seeing a lot of growth and uh a lot of investment and building data platform and figuring out how to do streaming how to do batch how to store data how to do schemas uh instrumentation all of those things and the one big missing piece in the ecosystem was building a map of all the data sets all the jobs and understanding how everything depends on each other right and that paved the way to solve uh data reliability so then you know we were blew up like very uh publicly and that led the way to the creation of data K uh we figured well we can all go our own way or we can push that vision of how we build data observability you know it's so much more mature in the service world when people know how to talk about slas they know how to do operations and there's so much more that is being done in the data world it's still the stoneage right it's just like as long as everything works it works as soon as something's broken it's hell on Earth no it takes weeks to resolve figuring out where the data problem is coming from so that led to the creation of open lineage as we need the parallel of open Telemetry we need a way to standardize how everything collects lineage or exposes lineage like before and I give here the example of an ecosystem like through open source but the same for all the proprietary data catalog as well like everybody reverse Engineers everything just very expensive it's very brutal every time one of those things changes your Warehouse spark anything it breaks lineage and they need to update it and make it work and it's just been like a giant waste of resources and uh repeated uh investment to make it happen so if only we have a standard way to expose lineage now every everybody works against the standard and there's a lot less duplication because instead of n Square communication you know it's a linear growth every new Commerce to the ecosystem can just Implement to get the spec instead of agreeing with everybody else how they exchange data and it's also a lot more stable because everybody tests against the standard and they don't break each other because they're unaware of the dependencies they have and so open inage becomes like this common layer to exchange metadata in an asynchronous way either through CFA or thanks and I again have my little timeline so it also starts you know at the end of the previous one so we start from Marquez this initial project that joined the lfi and data that was both the instrumentation and the storage and the analysis of data we split off open lineage as just just the spec and instrumentation because that's a common part that unifies the ecosystem together that's the standard representation that everybody benefits from and so that's the part of the project that's the most going to benefit from this uh ecosystem and grow so and quickly you see a lot of adoption from Microsoft from Google recently from Manta from aeria um metaphor data Hub uh because there's this same common need so everybody needs this right so all you need is to push this product and make it happen um so going a little faster on the model there's a core model of lineage and then it's extensible with facets and everybody can focus on the part of metadata this is how you attach metadata to your lineage whether it's data quality whether it's the version of the code whether it's uh the version of the schema and tracking those changes so on data set you can track stats like how many rows uh quartiles how the SCH schema changes on job you can check how the code changed uh how the query plan change on runs you can check you know how long it took how did the Prof Query profile change what were parameters hyper parameters that were passed and you can extend that so just going a little faster on this path so I can uh talk about my con conclusion and this is just an example on how you would do that in Python there's also a Java client on how toet open lineage and you will find resources on the open lineage that are your website the ecosystem how to implement this how to leverage existing Integrations there lots of job work already done with spark um airflow uh Flink uh DBT um SQL all of this and the the possibilities are endless right lineage is not a feature lineage is a means to an end uh people need lineage for for data reliability they want lineage for their uh machine learning reliability for compliance with gdpr CCPA um banking reg regulation bcbas 239 uh they need for governance they need to know are we Computing our dashboards based on the correct data there's so many things you do with lineage and everybody needs that so it's high time we standardize it and you know it's not like you build when I talk about you know people are telling me you know was talking about open lineage and saying oh I've done this before and you know we're following the Playbook from error and like what makes you think you can do it again it's just like oh good point right like so par it kind of just happened the project was successful I didn't know what I was doing but really the trick is we're not just like having an open source project and doing Mind Tricks so that everybody adopts it the the initial thing is what does everybody need so whether it was parket for arrow for open lineage there's an unmet need right that and you have know that from inside for trying to build those data platforms and like being in the middle of it and seeing what's missing and like okay everybody needs that and it's not happening because Engineers don't like to talk to each other right it's just like we're kind of nerds and we like reverse engineering stuff and it's actually a people problem right so if you create the project that everybody wants uh you empower the community to become owners in this and we you put degrees in the project then you get success so it's just the the trick is is to start with what people need and I makes things easier and if you know the stone soup uh Fable and um this is this story this is children's book someone comes to a village in the Tom Square and they start heating a pot right and they put and people come to see them is like what are you doing well I'm making a stone soup meaning they have hot water they put a stone at the bottom making a stone soup and of course the stone doesn't add any flavor to the soup right it's just neutral and so they're just a soup at nothing but and they say but you know you can bring your own ingredient and make my soup better and you know we'll share the soup and it's for everyone and you know so people come with leak and they come with carrots and potatoes and whatnot and just the actual making into soup so that's all I'm doing all I've been doing has been steering the pot encouraging people to join and participating become owners in this and making it happen and you know there's lots of magic that happen happens when you just provide the focal point for making it happen right and it's just about aligning incentiv understanding everybody's incentive align them create this network effect and the magic happens not when I convince people hey you should join in but then when people start talking to each other right the magic happen is just like oh yeah we all trying to do this it makes sense uh and then they you know everybody gets convinced by the group more than just me talking around and convincing and so yeah lesson learn every contributor is a stakeholder there's this sble effect open source communal shapes and sides you can bootstrap the community and like create this bigger group at the beginning to bootstrap this um collaborate with uh innovator Trailblazers and it's about the you know the the journey not the destination uh finding what everybody needs feels that need and align incentive and Magic happens thank you