sfspark.org: Alexy Khrabrov interviews Jean-Daniel Cryans about Kudu

Recording: sfspark.org: Alexy Khrabrov interviews Jean-Daniel Cryans about Kudu

hello everybody I'm Alexa crabber off the organizer of sf's park in France and here we are on location at quantcast a hosting talk on kudu kudu is a new system out of Cloudera we have one of the core members of the team with us here JD crayons and we just heard about kudu announced the thing we are happy to host one of the first talks yeah well my my first talking public actually that's fantastic right so we are super fortunate to hear about it and we'll ask GG a little bit about like a system about his interest then you know why code is the next thing we should try welcome so let's go a little bit with the the history right so so I know you were you are hbase pmc and you wear at stumbleupon where is base kind of one of the places for hbase grew and so can you tell us a little bit how this happened why stumbleupon needed something like HBase and can what are the crucial lessons he learned working on each place there that's a little lot of questions so if we start from the beginning so stumbleupon was funded like both for Canadians one of them garrett camp who is now the chairman of uber so back then would stumble upon what he saw is that they have a business model that can scale and Finn Utley as long as they can get more traffic they can make more money so they're like so well we have this infrastructure with my sequel it's scales I mean you can make it scale but like is there something better out there that we can like put the whole application on it and continue scaling up he identified actually if he I didn't feel a bunch of things but then he hired ryan rawson who wasn't at Google I believe mm-hmm and he tasked Ryan back then to find what what would the database be he eventually picked h place and then we he built a team around that so i was the second harder to work on HBase there yes so how did this happen right basically you know you have a standard kind of data requirement for web scale storage never body did Charlotte my sequel it was good enough for facebook you know how did you guys decide to to implement edge base and a pistol was all the work to do this and how did you manage to do this kind of when while maintaining stumbleupon going yeah that's me interesting story because when when we started using hbss stumbleupon so these are using it just before I came on and exactly six years ago in 2009 and back then hdfs didn't have data durability guarantees like so if you lose a machine you actually lose data so it was in the beginning a very wild ride so they just first identified the things that they needed to scale the the most like that that just didn't fit in my sequel anymore and then and then like the rest was just like enough flow there's more and more stuff from my sequel and what remain and eventually my secret was just what really made sense in the relational world all the other stuff like all the comments they were received serve straight out of a HBase I'll declare the counters the badges that was all done in HBase mm-hmm interesting so they took a big risk Garrett camp actually he believes strongly in H place even though like back then as I said didn't have their ability maturity was like it was a pretty young project but you you you read the big table paper and you think man maybe I maybe I can have my own little like Google infrastructure internally right mm-hmm oh so i just realized when you mentioned the canadian origins the base is a canadian thing no it's not it's not a understand herself okay well i was kind of funny like it all came into focus it's interesting it was my call stack and Jim Kellerman at our set like but actually they took it over from what's his face will be washing girl remember but it's really stack and Jim who back then picked up the project a power set because Chad there Chad Walters I remember correctly like they had pretty much same issue and they had the same goals mm-hmm but eventually parcel was bought by microsoft yes and then stack also came on board that's not ball up on and then we had our own little HBase Commodore a team i stumbleupon and what we liked about it is we could basically like a play with the rest of the community as neutral hmm individuals instead of we were and vendors like we're not making money directly from selling HP right all right I think this is fascinating right and so so the way kind of it was adopted and I mean clearly serves the king he tries I quit it serves the need of data store and we see this theme recurring and it's an avoidable you know people build different engines that have stream intentions and spark right but still you need to source so so it's very anxious for me that that you now work on CUDA so maybe you can tell us I mean you will have a technical talk but but kind of obviously HBase was not enough right because the fish way to solve all the problems you will just do is base right so I mean this is this is basically what I'm coming in with this man song from right and I think it's very interested you of all people bourquin kudo right so maybe you can explain basically how this happened and why we need to do is kind of next thing is it a success sir touch base is it a complementary thing how how we're could is on this spectrum of data databases right well so like stonebraker said one size doesn't fit all so H plays very solid database but most of all your problems all at the same time and on a more personal level I had been working on HBase for four or five years and when I started working on kudu and I was ready just for another challenge because HBase was like the first thing I did out of university okay so I just welcome to challenge also a different language c++ and okay so what could you tries to do it tries to be the one system that can replace your lambo lab the architecture so people will build systems where they're going to have that's a stream of inserts also updates and deletes that going to maybe parque files then they put impala on top or some other query engine hive and then they run their analytics and the problem is well I gfs is right runs read many kind of system so if you want to do updates and you don't have a system like HBase then you got to handle your own compaction of the files and making sure that the data that it is then uploaded into whatever a query engine that you're using so on on the other side you can use HBase for that but then HBase on the scan side like serious long sequential read it's very slow because that's not really what it was built for soku do what it offers is like being on one hand really doing really fast sequential reads mm-hmm while being able being an updateable store mm-hmm so instead of having maybe a mix of HBase and follow or your whole system around all your the managing of the parkade files you just have one system it's coo do it's almost as fast when you do sequential scams when you do scans almost as fast as parque and almost as fast for random reads on the other side as HBase interesting so we are claiming to be the best right we're claiming that it's going to be enough for your needs so if you can do very fast sequential scales it means so it's kind of it's good solution for streaming data so if you have some like Kafka right when you have a lot of stuff so normally people debug the stream systems by having that dating Kafka and then winding and we're playing everything right look like this is typical sparks Truman's a top right so you have Kafka going to spark so you can I mean it sounds like I should be able to use kudu for the same thing like can it be like a permanent storage or streaming data can I just kind of store it all there and an endlessly and then I can you know can replace some of this when i'm debugging and then is it the kind of a feasible model I mean yeah definitely we definitely want you to be your one data store and in those kinds of use cases because what we're trying to simplify is like I think all those different storage engines that you have to use in order to accomplish whatever go that you have but like so in the demo that I'm gonna do today it's 10 mollusca he wrote it and it's you have like basically online games are being played in the winners and then there's points all that data is being streamed likes it directly sent to Kafka so then Kafka can like it's a permanent Q so you can buffer up like if there's some kind of pushback and then it use a spark streaming to pull from kourou on one side and pull from kaka and the other side and do all the aggregations and then push it back to cool interesting so that that's really one use case that could is perfect for interesting interesting it's streaming like not a whole lot of interest reaming all the new players on the new data points coming in updating your data on the fly and on the other side you can use what you can use sparks equal are you in my case I'm going to use in Palo directly query data live including so you'll see the changes coming and young there is no delay of one hour before you can see the new updates like you get a new updates as soon as the line you just saying super interesting actually so when you mentioned that you know kudos meant to replace your lungs architecture that was very interesting to me because you know we have this conference called big data scholar and you know in order to kind of emphasize why we need them to end reactive systems we did this site called know et al dot org kind of prepared in no sequel right and and so the the ocean which scale community has we have this reactive systems you know you can have a cow as yo a TI you can have Kafka in the middle as message boss then you can have spark right and so you you're never dumping data back into HDFS as text for instance like this so there is this typical etl workflow where you grab stuff for one thing you transform it and think you dump it usually into HDFS text and then you read it back and parse it and then something else does something else to it right so so there is this piece of it I mean in etl is inherently step Orion basically first to give struct you know transform and a new law so so we kind of want to emphasize the fact that you don't need to do too low that anymore is just keep it there so I wonder from the CUDA standpoint and the architecture proposed would you agree that you know this kind of ETL old-fashioned I dlf HDFS text storage is obsolete in kind of your place and so you have now can you're supporting kind of streaming continual process and solution can you talk a little bit how how it like you envision kudu as a part of this architectures I think for the the live data crew will be able to replace a lot of the current architectures but it's it's not it's not a one size fit all so if you have like really really cold data that's going to go like stained his gfs in your pocket files it like if you're never going to update it there is no reason to pay the price of like all the managing that could is going to do like extra on top right and and if you're trying to serve a website then again like that's for each base so what lens in the middle is exactly what could use for so what about preserving schemas right so one the things we want to avoid within old-fashioned eight year we don't want to extract anymore we want to you know receive an object through the API we won't assign it to a type you know well fortunately kind of in a strongly typed language usually hopefully and so we want to we were validated it right we want to preserve that and hopefully you know that some consistencies checking and we want to pass the object bicycle through the whole system and I really want to store it in that way right is it possible like it's good to kind of built with this assumption that you will have typed objects we all have schemas like Kafka right combo Kafka has a schema server how does could work there so kuru unlike central rage base is strongly typed like that when you use a normal relational database you have multiple columns they each have a type who is exactly the same way there's nothing new there and the same same thing with your keys instead of having a special columnist your key you pick the columns that you want as your keys we had composite maybe there's only one and again that's something that people are very familiar with we don't have nested data types yet but that's also something that's going to be on the road map interesting how big other objects I do have some optimal object size like would you handle large objects like you know we have a lot of PDFs in my use case like can you send binary blobs you know several you can but that's not really the kind of use case we're targeting in the beginning if you have an analytics use case you're really doing analytics and PDFs maybe you are but I don't know about your use case okay well we can extract a lot of sex from a data but I mean little bunch of metadata which we can which is not as big party no and but there is a binary blob associated right well I mean you could I mean it really depends on the size and if you're talking about megabytes here and I would say the kudu it's gonna have a hard time since there's like no optimization what so ever done for dark medium or big objects but there's a lot of stuff in the back end we there's always this problem like in other systems of writing we are always doing background confections and you if you want to update them you have to like an HBase if you do a major compaction you have to rewrite the whole data set so there's a lot of smarts inside kudu where it's only your the data that's the most fast changing that we're going to wreak um pack if you have like large blobs that you don't really change fan because that's your text like you might be computing things about that text that's going to change but not the text itself we're never going to try to rewrite the text if you have updates of the other columns because good is a columnar store a true corner store unlike its misery which are more like Cohen family family oriented mm-hmm so we really store each column separately and do the appropriate encodings interesting so let's say you know I have a lot of documents and then you know in my use case I will constantly improve my machine learning and extract more metadata and more I mean I all the different extraction right so for each ID document ID I will have a better and improved version of various kinds of metadata this is kind of a useful good store for this kind of stuff where I want to have basically I want to replace you know a lot of stuff insert columns and I'd say depends if you're if if like the one thing you're doing the most is updates and deletes I would say like use another kind of data store like HP's or Cassandra there are we're like writing is free you don't you don't need to write when you and you don't need to read when you're right you just write update as a right delete as a right present coo do we have this drink enforcement of the rookie so if you want to write something or did something we first have to check if it exists yes and then there's a lot of optimizations we do on this on the scan side like if you want those fast cans there's something you gotta trade off right so if you have a lot of updates we use what they call Delta reduced Delta undos so from your base data we have to apply all the updates that you've done in order to get you your final data so if you only have a few that's fine if you have a whole lot and that's your main use case then it a kudu will not be as performant as interesting so I would say you know we can sell a little bit in the middle right so let's say we're like we start a bunch of text and that probably say the same so we will improve our algorithms iteratively right and so so a typical this case will become whole collection of staff and the the middle it will mostly stay the same so but we might improve our name extraction so you know maybe ten percent maybe five percent of text will change so can I give you kind of catechin I like let's say I maintain my metadata kind of an adjacent file right and I know on my side like I know how to deep my metadata right so I you know can you take my d function and would it make your life easier if I would tell you I know how to update this column with this field because it's kind of my metadata which is structured and I only kind of you know I have a list of you know people names in this document so like inside of Jace and there is an array in that array I did three my name's and a replacement like so that like I know specifically she records changed would knowing how my my fields differ like having a diff function for me which is most efficient simplify life not really right now if you try to update a field like you just you have to provide the whole field that you want to update so but as I was saying earlier we want to add those nests nested data type so you have one field which is actually composed of many fields and you only want to update one thing in there these are that's the kind of optimization that nested data types usually handle and that's not something that we have that's something we want to have oh kudi is only a little bit and see how can i still only in better and we release the three weeks ago so we still but the thing is the the reason it was closed source I've brought her for a long time like almost three years all right and now we're releasing we're not really seeing a final product we're really seeing as a public bit oh and that's because we were in a mode where we want people to tell us about their use cases and see like a we don't want to prioritize like things too soon yes oh like just by ourselves you want to write based on feedback what should we be building for this data store cool so it's good you know to become like you know if you like somebody wants to work well for the use case you know this should become Apollo customer and let's try it with just contribute feedback and also it's it's an open source database of the if you wanna contribute the next data type implementation patches are welcome as we say yeah well I did a lot of C++ and very to thousands and I vividly remember right in corpus or cincy possible yes who actually proposed to add a flag to g + + cold air there's not an idiot which will remove certain warnings from auto-generated korba so I here's a question for you I mean this is pretty big difference right so you work in Java and it's understandable for each base you almost have to kind of fried jar which is almost like assembly like its job which is almost like see right like we have to do low-level java and and so kind of his writing the database in C++ liberating like do you really kind of like you don't want me to pretend it to see you can actually write it and see is it is it you know and I think you know the same thing goes for in Pollock is this is this like what is your take on this you know choice you know Java on the one hand and all the whole ecosystem versus applause pause for the database yeah we always get this question and again there's like two sides for it like there's a personal side where for me I have been working in Java for basically my whole professional career and like hard core java right like where you're handling and terabytes of data yes and now making this switch to C++ so again on the personal side that this is like this was like a new challenge for me I wanted to learn something you don't want to do something that I've been doing but in a different language helps you like understand other things better I'd say like it gives you another perspective can you elaborate like what is the perspective after the first perspective is like the garbage collector in Java man this is like this is the one thing that really kills it's useful it's useful but in big systems where you have a lot of data going through the JVM mm-hmm it's it's it's mostly a hassle so you're always trying to put stuff outside of the heat like in each phase like the first thing we got outside of the heap was was not i'll be outside the heat like we use dark byte buffers for them or just by buffer for the mem store so now you're managing your big blob of bytes and then the next thing was like for the block cash because you want to let like four machines today 256 gigs of ram is it's a lot but it's not it's something like crazy right and years ago was 64 but like try creating a JVM that's fans those like 250 something gigabytes and then you hit the gc's LGC it's like end of the world that Noah's gone right so you want a black ash that fits in ram but not on the JVM that again you're going off the heap so all that all the effort like in C++ as long as you're managing you're managing your own memory and there's a lot of tools now that exist to make sure you don't have memory leak that you're always destroying the object you're supposed to be destroying like our built environment has like we have memory checker we have address checker we have thread checker and we limit ourselves to a set of libraries that are like well known in c++ that we're not actively trying to show ourselves in our food okay so but you know gvm has come almost an extreme linked with the big data like a system ipod us and we're soon big data is kadu panas JVM and basically assume you have a distributed system and GV m and everything we should details how do you find a system written in C++ interacting with this like do you do you hear is it is it a disadvantage that it's you know you don't have everything with Java gives you for free already how did you find it in practice it's a trade-off I would say it's definitely a trade of like I wouldn't say like oh my god C++ best solution why isn't everybody going c plus plus there's a lot of people well like me like three years ago that wouldn't touch c++ projects so that right there there's a bunch of contributors that you can't really get on the other side there's a lot of people are like oh my god finally something in C++ on the storage side and then we have Impala which is already in C++ yes and other projects are also looking at like spark the tungsten project isn't isn't isn't it like also going away from from Scala well and I think its memory management kind of i mean i think people people want to manage memory kind of an artisanal fashion but but i think i'm a lot of them us trying to stay within GV m I think flink did this right so I think the question like what you've done with HBase right so if you can do a reasonable job from within Java but yeah that's interesting and there's a lot of things that we're doing kudu that we you wouldn't be able to do the JVM like using the SSE instructions orders in the instructions you'll have to go through the J'naii interface in order to be able to get there right so this is something that we can just naturally do in C++ ok no this is interesting so so I don't realized that kudos in gestation for three years and maybe like the best kept secret because I haven't heard about it so you can go on there's a question on Quora I think was asked two years ago and it was what is that lip con doing these days uh-huh cuz toddlercon is the tech lead for kudi net it's basically his brainchild like he thought about this for three years ago he was the one working on it mostly and uh huh he's he's a he's the one that knows like everything about cool so how come we never heard about it was it like a skunk works project was it kind of you know slowly brewing or like we're still a very small team like six people mm-hmm so we were just the first thing that was built was just a really fast call nor datastore and then on top of it we build the rest but we we wanted to be able to iterate really fast internally before going outside and then like going for contributions which is like a model that drill took like they went from like let's design by committee then and then build that way which is also like another very valid way of doing things that's the way we chose ok and ok so I'll get the wrap up with a very philosophical question and even kind of very abstract so I've had the viscera variance conversation with Evan Weaver who is a has a comical fauna DB which is you know a different take on a database and kind of its you know hasn't launched properly yet so you know I don't know much about it but we just had this channel discussion of Evan about you know the role of the database and so he built it a sort of Twitter in handle dramatic throughput so he has got a first-hand experience with the systems and an obviously with reduced us and kind of they have their own strument so they need to build the whole big data stack because they have a user so he actually better some interesting ideas in he basically said like you know he's thinking of you know using query level processes as granularity level right so and in kind of if you think of this whole land architectures whole stack so you're assuming that you have distributed systems such as aqua Kafka spark cassandra and there but kind of you know they have this big knowles the assuming they have a machine to put a demon all and do something there but in the end everything or the only thing they're doing they're running queries so why not focus on queries and basically have this whole cluster instead of kind of putting together all the things and putting them in docker containers and using masses to to balance them and schedule them why don't we drop down to the level of the final work result which is a query and think of it as a process right and then basically when Anna schedule is queries on the cluster we know exactly what we're working with instead of having tag to move like little Linux light containers around it's also so it really struck me as a revelation right that you know if in the end if all we want to do is run you know queries against some stream of data why don't we optimize the whole thing for this purpose so I'm just wondering what your take on this having built kudo for this criminal case right I mean this is like very new to me I'm trying to understand you know the meaning of this so I'm just kind of narrated relating it back to you and kind of am wondering what what what's your take on that I would say I'm exactly where you are actually that's the first time I hear about this idea and I definitely need to spend more time thinking about it well I wonder what the trade-offs are because there's nothing perfect right so if you're doing this where are you not able to do that that would be my first question right right right so but in that sense so you do is kind of it's enough the distributed system right so like so you you know you want to coexist effectively like if you run Kafka spark and guru you will have the demons on each node right so so like in the current situation like will hope you know like the locality will be achieved by somehow making everybody together right so yeah I mean just something I kind of final five fascinating well I mean the design by composition I think everyone working on their own piece that's basically what the Linux model is right mm-hmm and I think it's a good model all right well thank you very much for sharing and we're looking forward to talk thank you very much