sfspark.org: Jean-Daniel Cryans, Kudu: Data Store for the New Era, with Kafka+Spark+Kudu Demo
hey hello everyone my name is JD I'm going to talk about kuru a new storage engine that was released by clatter and not too long ago but before I go into who i am i like to know a bit more about who you are so quick raise of hand if your data analyst so all day long you run queries and try to understand data no data analyst all right raise your hand if you're on the data infrastructure side so you're actually like setting up the pipeline's building the systems all right not too bad raise your hand if you're building databases okay and final question who has heard about kudu before reading that the meetup was going to be able to do after you all right perfect perfect okay so yeah so myself I'm a software engineer at Cloudera I've been working on the kudu project for two years but before that I was working on the Apache H by H based project for which I've been a Commodore since 2008 and the for joining platter three years ago I worked for drink for three years for stumbleupon where I was working on HBase doing open source yeah so kuru was still sourced a clatter for two years now and now we've open sourced the project in September so we can finally talk about it this is then my first public talk about kudu I can finally go back on the on the meetups meetups track talking about my stuff so yeah we're going to talk about kuru which we've released in September this is still a beta product so everything I tell you about today if I sound too enthusiastic I shouldn't be because like we're trying to not oversell and under the liver I want you to know that you're not supposed to go download kudu and then put in production tonight because you might hit some weird problems but it is Apache License it is code that you can download right away and if you want to provide patches and whatnot at the end of the talk i'll show you how so why why build qi build yet another storage engine in the whole Hadoop ecosystem and so three years ago toddlercon my coworker and tech lead on the lacuna project he sat down with our CTO and they had a discussion what is the one use case right now that is not being addressed by the hell Luka system so we would look at what our customers are doing and what is their doing that we could we could provide a solution that would just like simplify their life and also like apart from those use cases how is the hardware landscape changing so that we can take advantage of it so if you look at this mba graph here so you have you have access but you don't have numbers right we're just trying to place things on a graph if you look at fast random access you have h base which is like all the way on the right you can do really fast reads really fast rights scans and not so much rights if you do ball clothes yes but then it doesn't excel as much as Hadoop whereas a loop you can just directly done the files and you can use parque and do really really fast cans so that those two systems are really good at what they're doing but there's no intersection really where the one can do what the others does really well so we need something that's in that red box in the middle that can do both and unsurprisingly we came up with solution here which is what I'm talking about today which is cool so we want something that has high throughput for those big scans that you want to run normally like on parking on HDFS what's something that's low latency enough that you could do whatever you're doing right now on HBase like you do your random reads your updates or deletes of your data and not be so slow that you wouldn't want to be using it we want to build a system that people will be familiar with so it's not yet another interface that you have to learn about that's completely foreign to you something that feels familiar so we're going with a relational data model we want you to be running sequel queries on top of kudu we want you to be using types want you to be using schemas so a bit about the changing landscape there are many things that are happening right now first we're seeing a lot more SSDs more fashioned in our machines the we use the design system around spinning disks this is still what you'll see the data centers but in five years going to be very different we also have technology like 3d cross point which is coming from Intel which is flash that's a thousand times three orders of magnitude faster that the current flash technology not just imagine having disks that are a thousand times faster than what we have right now and this is coming from Intel next year in theory and I'm talking about Intel because they gave us a lot of money last year right also Ram is getting cheaper you not so long ago 64 gigs was like the high end of what you would see like in your commodity machines but now i can already know it would be maybe 256 gigs of ram and this is just going up this is not slowing down ram is getting cheaper every year and the jvm start struggling at those levels it's rare that you want to start a JVM and like just go use those 256 gigs of ram and please don't GC so the first takeaway is now that we've removed all those bottlenecks on the i/o side CPU will become the the next bottlenecks like it used to be that we design systems thinking like we we have so much CPU on those nodes like we don't know what to do with them can we just put more spindles in those nodes and now all of a sudden it's like the iOS fast and the CPU becomes a bologna because the system's haven't haven't been designed around that and the second takeaway is and that might be a little little stretch over what I just said is that column stores they're feasible friend um access what this means is you would do a raw reinterred data store because you only have to do one seek on your disk and then read whatever you have to redouble video the whole row right and HBase cassandra sort of worked like that whether column family orientation you would have liked the whole family sitting next to each other on one desk and that's one seek all right so if you want to do a columnar data store if you want to read the whole row that means you got to seek in each and every of your car them in order to get the materialized the rule but if you're honest as these if you're running on flash seeks don't exist so it's free almost so it becomes feasible not maybe as fast but feasible sokku do what it offers it has a sequel like schema so you have a set of columns that will be typed that will be fixed and you have to alter your table although that is quick unlike you don't have to lock the whole table but it's we're getting away from the HBase and Cassandra model of putting as many columns as you want inside one college family of those columns you can say that you have a primary role key again this is nothing new this is what you've been used to with your relational data stores they also offer at the same time no sequel like AP is so you want to run a scam you want to run insert update delete this is also easy as you can find with kudu people that we've been working kuru come from the rest of the Henry Baker system so we're also at the same time offering MapReduce bindings spark bindings and impala bindings since we're well from clutter and there's more to come so use cases the kind of this case is that coo do is going to be really good at you want a lot of sequential reads you want analytics like this is what we we see on the website and I believe on the back of my t-shirt this is fast analytics and fast data so this is like analytics so long reads sequential sequential reads on data that's changing you're always ingesting data but also modifying it so for example you have a time series your you have data that you're always interesting that's coming from a bunch of notes and sometimes you have to do modifications sometimes you have to do some compaction of your data so you have to be able to update it but you're always reading the big sequences you want to do online reporting so you have someone that's currently using impala and you want to update the data that's Impala but right now will be impossible well with good will you be able to do it and at the same time that only that's crazy update your data so if we look at a typical architecture that you will find it in the wild let's say fraud detection so you have incoming data that's going from your messaging system and you put in HBase because like it's random right you don't want to append to a file or you could in this case let's say we put in HBase and then eventually when you have enough data you dump it all to a park a file and you want to use that porky file when you have enough of them or maybe right away eventually but load it into another system it could be Impala it could be high view so you want to invalidate some files and Impala and then so use these files instead this is like the new data this is the updated data and finally you can do your reporting on the data that you ingest it so there's a delay here right this is delay between the time that you're ingesting the data and the time between the time the interesting in between and the time that you're actually reporting it there's also like what happens if it fails you have this whole pipeline of things that happened like your your set of cron jobs that you have that's running and then like there's some there's a bash script that you just modified it broke and now you have like hours of data your how is updated behind what happens if there is data that like it has never been putting that into that that pipeline you have to make sure that every single part of the system understands it and can process it without failing so what we're seeing is with kuru you have your data you put in KU you and then you read it you only have one system to operate you don't have this whole crazy set of cron jobs or jaw joints in the background that you're running if you have data that arrives late let's say you have your processing data by day so you have the data for today is coming in and then you have the switch at midnight now you're now you're tomorrow or longer today we still you might still have stragglers that are coming from yesterday but you've already rolled up the whole day what happened then well you need to do that that compaction of your old data previous the data with what you what we closed right and since it's already in kudu you can directly read from your data as soon as it enters so how it works I I'm going too much into the details but if you have questions later I would be really happy to answer and by the ways you have if you have clarification questions during my talk I'll take a few if you raise your hand you have one such occasion right now okay all the way in the back all right yes so the question is what about nested data types this is not something that's currently in kudu but that's definitely under world map we know it's important there was a question in operation this is basic Cassandra edges and Cassandra scan operations today work yes they work so how general generalizing separation because folks in different ways for educational and how do you differentiate generalize well so let's say you want to just do like a set X star of your table all right that would be like a full table scan that would be exactly the same thing in Impala or HBase you you would read from the beginning of each range tablet or regen and then through the end and then we turn out the data or just let so you just want column and then read that off and then we turn the data like that that's the kind of scans we're talking about here that's one example right and what we're saying is and I'm going to a little into that why kudu has made some traders that are faster that make it faster right other things we're going to be slower I'm going to talk about right now the data so it's not really designed for five columns regular number of columns so for time series you were just at the huge number of all right so the question is like a kudu were expecting more tolerant table than wider tables yes then that would be yes that would be the case alright so the way that you like so we're using a parlance here that's similar to big table so instead of talking of regions of ranges we're talking about tables and tablets so your tablet's there horizontally partitioned into tablets your tables or partition tablets exactly like the way you expect it to be like in HBase so you can do range partitioning you it's like you would sort all the rookies and then you would simply say like this is my split point so the data that falls in between those two bounds will go to that region well in that case that tablet or you can do hash partitioning and you can simply apply a hash function on your roki and it will be distributed on your cluster we can support both the same time in kudu we're using RAF consensus in the back end to do the read data replication so what this means is you can read from any of the tablet replicas you don't need to be always bound to go to the leader of the corn you always write to the leader but then you let's say you want to support queries that's like read five seconds in the passcode I know I don't care so much about the race updates you can go to any of the replicas and get your data and then how we separate those tablets simply hosted by tablets servers they are storing the data directly on this not using HDFS and I will stress this out stress this out kuru does not rely on Hadoop at all you can deploy kudu without any other dependencies as long as you have a formatted disk the XC 4 XFS preferably the metadata we're using a master-slave kind of architecture so you have a master that holes on the metadata where the tablets are where the what the tables are with the schema is but it's pretty light the Coram's themselves are responsible to report what they are so the master can die and of course the rest of the cluster will continue functioning it's just that you won't be able to create new tablets and you did new tables all they did is very small and weak a in ram so on a big cluster we did some tests and we can usually enter all the queries within a few dozens of microseconds or at the worst case a few hundred of microseconds so the master is not a bottleneck the master is also replicated but in the public better that we just released its kind of crippled so single master flow moment but this is not against the design the master just uses raft in the background so when you click a few your client you say this is where the Masters are it's going to figure out which one is that leader and then bootstrap itself from there so this is this would be your your typical deployment you have all the Masters on the left they all have what we call the master tablet one of them was going to be the leader its console gonna be the master from which you update the truth and then you have a set of tablets servers with that all hold some parts of each of the tablet so let's say tablets over one tablet server w will be the leader for tablet one it's going to be also the follower for that three won't have anything from tablet to and this is how we do the load balancing also so a little primer and how raf consensus work it's it's basically paxos if you're familiar with that but mid simple so let's say you have a client it needs to write data so first of all have to ask the master where is the leader and then it will say well the leader for for this de stab let isn't that what server a alright so let's send a rudder pc when the tablet server which is the router pcs got right to hit right ahead log and at the same time in parallel to all the followers replicate the data the followers will write the data to their own right they had the right head log and as soon as you have one of them that replies that means that you only need two of three servers to accept the right then that means that the leader has achieved majority and can say success i have written your data and we ensure strict arabic serializability on your rights so fault tolerance first transient transient transient let's say you have a follower failure so it's not your leader your leader replica for your tablet that fails the leader still has majority if you have three or if you have five you can support even more failures than that so everything Phi functions correctly as long as you restart the tablet server that was holding the follower within let's say five minutes the lab the leader will have enough logs to just stream back the follower and get that get that note back to the speed to where the rest of the cluster is so you won't even notify you won't even notice it now if you have a leader failure the followers are expecting to hear a heartbeat from their leader if it doesn't happen three times every one half second they will elect a new leader between themselves so after after a few seconds let's say after 10 seconds they'll double out they will have a new leader and you will be able to take inserts right away there is no logs to replay or nothing like that there's no session timeout and let's say that you've restarted the node that was the leader it's going to join back as a follower as if nothing happened it's gonna come back say hey guys in the leader no you're not okay so we can handle n minus one over two failures so that means if you have three of the cuz we can tolerate one failure now let's say you have a permanent failure there's a no that's just gone the leader was going to notice that the follower has been gone for five minutes it's going to start it will have to kick him out because if even if the follower comes back after five more five and a half seconds well it doesn't have the logs required to get that note back up to speed so we'll take the follower tell the master about it the Master will pick a new replica in the mat in the cluster and the leader will copy the data to that new node so there's a bootstrapping process bulks and all the data from that further tablet to the new replica and eventually when the replica has all the data you switch mode to follower starts getting a stream of logs and and then it's as if nothing happened the tablet design it's similar to lsm but yet different so we even though we have Dracul on disk and we don't have a right once read many kind of limitation there's still a lot of good things about immutability so we don't we don't try to rewrite the files or change them in place so we have a just like HBase we have something we call the memo said that we're going to flush the disk durability is guaranteed because we have a right head log when you do use when you do updates we're going to use mvcc and then we're going to guarantee that you have a consistent view of your tablet see we can't even guarantee a consistent view across your tablets using the bag extending in the background that's called Harbor clock couldn't will perform better if you're something that's near current time so it will always try to optimize to put the latest data as the data that you can reach the fastest and if you have updates that just came in or you want to go back back in time then we'll have to do to do more operations to get you to what the data looks like at that time so the performance as you update your data will worsen so the kuru has to do some compaction the back ground to get that based it up as the data that doesn't have to apply updates on so if you compare lsm versus kudu kudu shares some resemblance with lsm but it has a it uses something that's very common in other data analytics database what uses what we called redo deltas and undo Delta's to track the updates so when it flushed the data that's in the memorable set that was inserted in memory instead of if you have an update instead of going into the memo set it's going to go to a special in memory store that's called the Delta memo the Delta M store and then it will apply the update in the deletes they're flush them into small files that are that contain all the other deltas and then apply them on top of the data you're reading parse in lsm when you're reading from well you're running to memory and then you flush then you're writing against lets you update updates going to go in memory the same remember upset and then flush into an h-file where's a stable so when you want to read from it it means you have to always merge all those H files are those SS tables which is pretty cpu-intensive so the trade-offs your random updates will be slower because like the HBase model DSN model does not require reading your data before you write there's no primary key constraints we don't have to make sure that when you're doing an insert that the data wasn't already inserted if you do an update in HBase you could you can update data that doesn't exist because actually there's no update in HBase you just always right delete is also right included we are you have we also have to always have to make sure that the key exists before we do anything so that means you want to write we gotta read you want to update we gotta read to make sure that the data exists and find where it is so you random of this will be a little slower than the system say like HBase and the signal single row readable also be slower so it's a corner design we're optimized for scans I won't just want you want to scan that column for the whole database we can do that really quickly HBase cannot but if you want to just read one row fully it's visible answer before kudat us some benchmarks so we're NT PCH on a 75 76 not cluster to all spinning disks using the CUDA version of Buddha pre-release at September and Impala 2.2 with kuru support so this is something that's available not with it in the Impala it's currently released you need a special build that we call Kim Paula so we use a scale factor 100 so 100 gigs it's a small data set that fits in memory I have a query here that's the kind of query that t PCH will be running and here are the numbers so we're comparing in blue parquet so this is Impala acquiring the data that's stored in parque vs. impala that's squaring the data stored in kudu and it's on the x axis you have all the different queries that contains our containing t PCH that you have joins you have where's and then you have you have the latency so if it's lower it means it's better and as you can see I was saying earlier that we want to be almost as good can everyone see correctly the slides will be a little online we say we're should be supposed to be almost as fast as parque but in this in this test we were generally faster and sometimes much faster than parque and the reason was that there's a lot of things you can do in kudu that you cannot do in parque if you do break it push down we don't have to materialize whole rows to then filter them out so there's a lot of smarts like that we can do but also the data set fit in memory so impeller will win if you put that data on disk but then there's a lot of tunisians that we haven't done yet and it also there's a lot of musicians that Impala hasn't done for data that fits in memory because usually doesn't so what about the Apache Phoenix is if there's people from Salesforce here by any chance No so we compared it on a 10 mil cluster HP is one point 0 with Phoenix 4.3 a few things first loading loading the data we do a data load for the line item tables that's only six billion roles in this case because the smaller cluster the higher you are the worse it is so lower is better and this is a logarithmic legarreta log scale so 1 10 100 1000 left Moses Phoenix so in the load phase kudu and the Phoenix which base perform pretty much the same and could Park a is way faster if you look at the tpc ashbury one so it's a normal query that has a few where classes but no joints Phoenix did it in 219 whereas could we did it in 13 seconds quite a difference more than an order of magnitude and then park it was just below that now just doing a contestar Phoenix actually has to read all the roles to the council at 76 seconds could who did it in 1.7 seconds that's because we were just reading metadata there's like a quick optimization you can do here and well it doesn't exist there because you don't know what the rules are in HBase and so the only place actually Phoenix wins against HP against kuru is if you do a single roll look up and it's as I said before so it does it in 40 milliseconds compared to 150 may seconds for kudu and parky has to like it has a lot of it doesn't have an index so that has a lot of data to read to find that one row so it doesn't one second point three and this is the kudu is not always winning and we include this slide just to show you so we did a no sequel style of random access benchmark which is in this case why csb so we use all the workloads that are provided and rerun it on the same tunnels cluster comparing YCS be on kudu versus ycs be directly on HBase lower is better actually higher is better in this case so it's throughput operations per second you can see the HBase in this case inserted way faster than kudu zip phien workloads like the more rules that the worst is if Chen the faster HBase was compared to kudu uniform kudu actually won in one case and then the other cases it was more equal and I don't remember what the decoy but as you can see generally HBase will be faster at these kinds of queries kinds of workload so kudu is carew is not trying to replace all the workloads who has a sweet spot in the middle and this is what we're trying to demonstrate question them this use this does this test run out of memory yes on the hundred million rules I believe it's all it also fits fits in memory ? it right it reason right or is it a mix yes it is a mix I think it's just the workloads it was that lip contour under the test but I believe it was just using like if you look at the workloads folder and why CSV it's just running those yes but what about the cage it feels like the secret sauce is actually extremely expensive SSB from info all right so good question like it seems the secret sauce like what if it does have to hit disk I mean that's that's really a test that we have to do there but like those could rely on specialist SP if you haven't seen those special assist these the three crosspoint from intel yet so unfortunately i can't even tell you but we'd be faster for sure i can tell you that because we're optimizing for that now the question is like is that does you do you require that no obviously not so the will be still slower or faster like someone do the YCS because i would fully expect ycs be to be a faster on HBase definitely looking at t PCH results i would expect impala to take over kullu in most of the cases except where we can be like if you have a fairly restrictive where clause and you can do a lot of predicates splashdown Park it would still have to read a lot of data that could will never have to read because we actually know like what we're storing and we have indexes and that thing is impala we'll do like it megabyte reads at a time from disk whereas kudi will do I think it's like 64 kilobytes or even four kilobytes so right there like there's a lot more round trip between process and disc on kudu right so Impala is going to be way faster the question would be slower for critical updates I mean that that would be the ycs be worth of the thing is we keep most that stuff like if so we use bloom filters at the at the rowset level looks like I didn't go to those details but there's a lot of tricks we we use like to have to prevent going to disk unless you really have to and we can use like really large heaps if we want to I give us a give us all your memory and we'll use it and could you if you want each base would be will have a hard time did that's a are you satisfied with my answer hey all right let's move forward so not trust me I'm a vendor so those are slides that were presented at strada in New York when we launched when we launched a product so it was being led from xiaomi they've been working with us for about a year now on the product so they're currently using kudu not on a really critical path because obviously kudu is still in beta and for them was in alpha and we're regularly breaking the RPC the API is everything right and so what they do they're one of the one of the big Telecom's Falco's in china fairly recent company so they have a big dinner analytics pipeline so this is what it looked like before kudu it's going to be very familiar with the other slide that I showed earlier so they had a data source running into H base running into sequence files merging that into hi mr spark and then you have offline analysis you had online analysis long pipeline the data could take between an hour and a day to eventually be queryable and there's no ordering in there so that this is the case that were referencing earlier where you could have data that's like a day or three days old that comes today and you might have like already closed up that data and now it's coming so now you have to invalidate some files and then re-upload them so after kudu they have a much more simple pipeline or do I can see when you get your data and where you can query it is like up to 10 seconds and they have to they have like a direct path if you're if you're willing to accept back pressure you can go too fast path else you will go the slower path through Kafka and storm that could be a sequel that could be spark streaming so they did some did some benchmark for their use case number one which is a mobile service monitoring interesting too so they have all those are pcs are coming from their phones and they're basically doing analytics on that they have five billion record today and they're growing there's a lot of people in China who knows you know so basically they have they're doing Alex but also sometimes they want to reach a single record so you have this this mix of workloads that I was talking about earlier that kuru is really good at so they did some benchmarks on a small 71 note Buster with the following hardware so they use one day of data and then they run a few of their queries and those are the results first they had to load that one day of data into kudu and into parking obviously Park a much faster since there's a lot less things to do when you're inserting data you just dump it and then the query latency so that means lower is better for so that means for the different queries that were running in some cases CUDA was slower but in some very restrictive cases could was way faster like the aquarium verse 6 were in less than a second cool was able to answer with Sparky was taking in 16 seconds and again this is just because kudu knows what kind of data storing and it can directly return what you're looking for so lazy materialization is one talking about like we don't materialize the whole thing and that just to filter it out first we're going to apply the predicates that we're pushing down into kudu and then we're going to materialize the whole row if you really need it so it's very restrictive like one out of every hundred million we're not reading 100 million roles to return only want one are you returning one we can we know how the data is laid out we have we have the tablets we know you know where the ranges are so we if you just read need a certain range you only need to contact one or a few tablets or the next to each other and they're using things that are from the earlier this year so they're still missing a lot of optimization they're not using hash partitioning in this case so it could be web even voi faster for kuya I'm gonna skip that one all right so I'm gonna do a little demo but half time are you guys bored no all right eight we're gonna do quick this quickly so this is a demo that my coworker Ted mollusca he's on the East Coast working for cloud era and he's super super excited about this technology like way more than we are because he's working with customers and he's feeling the pain every day and he's like oh my god it's gonna save everything like whoa hold on hold on we're not selling the software yet like dude so he built a slow pipeline he's a gamer so he's using he's using data points so like let's say you have online games are being played with the the players are winning and you want to run some analytics on what the players are doing so you have he first you have a producer in this case as the demos which is producing random data when I plugging to blizzard whatever and simply sends the data into Kafka so Kafka as a person q and then spark streaming in the middle will pull data as it coming in from into Kafka and also merge that data with the data is include you and run some aggregates that is kind store back into kudu and this is all done life and then from Impala you can live query the data as it's coming as it's being aggregated something that if you had if you only had parque you want to be able to do all right because you can update the data so the code it's it's the the fun it's the foundation code for all the bindings that we're going to need for data frames sparks equal etc it's all living in his github the link is there spark on kudu but if you go to kudu 1214 then you're this is like the tracking ticket for all the work that's kind of covenant the actually following weeks were a little late that so this is where all stops working so again i'm not here to sell software but i use my own software to deploy other components so i'm using Hugh for the data representation and we also have yarn there you are resource measure alright so I ran some application this for all right can you I'm gonna set this bigger so the first the first thing I'm going to run is the kafka producer generator so this is simply something that generates the data as if it was the website itself and then sends regularly sends the data into kafka so I'm doing this live in front of you guys but sending records now I know I crash and burn and if you're familiar with spark i'm doing here is i'm doing a spark summit of a streaming job and the sea at the end means cluster so i'm actually using the yarn manager cluster manager first part and that's so that lunch launches a yarn application all right there's always a lot of outputting so this should show up in my own applications so we have we have the running here started at eight and now we're in spark alright so this is going to get updated you have streaming that's always running so it's always pulling from Kafka and well good is already running so it's already doing the other you aggregates so what if you want to query it so you're going to you and I'm going to use Impala so this is this is just that it's nicer to look at because I could go into an X term and then like run the sequel queries and extra but I thought it would be better to do this kind of representation so let's load the recent queries that I've been running it's a little bigger so I'm going to do a counselor this is just like reading from the the whole table just to show that there really is changing and then I'm going to do a simple select from gamer order by last time played right so the the set of gamers is slowly growing and they're always playing and then we want to see the last 10 players that I've been playing so if you click execute oops somewhat i would you do alright so i have 45,000 437 gamers and those would be the latest gamers that I've been playing now so let's let's remind let's remember gamer ID 5578 all right cliff execute again a few more gamers i believe and different gamer ID that's it we did it guys we have data data in a novel text or Pepa thank you for those that all right but really what I want to show here is that you can do those fast and Alex while you're changing the data if you've been feeling the pain of having the lambda architecture where you have to manage all the updates and the changes that are coming in while running your long queries and whatnot this the kudu is there for you and we're trying to solve that problem and so if i continue querying its execute again there's even more players oh my god and the gamers have changed well this is always happening so this is going to look very different just spark happening in the background just Park happily chugging just processing all the stuff that it's pulling from kefka and calf go just receiving all the other rights from what would be the website it's actually something that's a fake it's just a process running somewhere so that's my demo magic so the project status we've really used that number 28 so this is still fairly recent less than a month version 0 LF five why is it with a five because because it's not zero that one not ready for production don't go deploying this there is no security so if you're a bank or something like that you probably right there that's like a showstopper for you what we want is feedback we want we want we want to talk to you guys we want your use cases we want your we want you to try the software and open jurors and if you feel like doing some C++ you can even provide some patches we have a fairly readable c++ code base we also have a java client if you and a Python client so if you if that's your language of predilection I'll go for it our next release that we're planning will be in November we didn't have mac OSX support now we'll have mac OSX support but don't go deploying a cluster on it will only work for a single no deployment and that's what we think is going to be the case for a long time because of our different requirements and it won't be it won't be a big release will be just a lot of small fixes like we've always been running in psych ladder LeSean me and a few other now that people have been trying it like they try in very different ways there's one guy right now that's been trying the java client with has never been used before and he's finding all these bugs that have been fixing that's great that's exactly what we want and pre-empting that question when is GA when its version 10 next year maybe I cannot make a commitment unfortunately but we will have Kerberos integration that's for sure because you we want it for it to be when we release GA when we reuse what that one dot oh that's going to be production already we need security we need the integration with the rest of this thing so if you want to get started as a user go to get to do the i/o we have a mailing list we just open a slack channel that if you follow the link there you're going to get an invite and then you can join the whole team is there the whole pirate team and all the new contributors that we're hopefully getting every day there's a quick start vm you can download already has kudu and impala installed on it so you can just try out the api's right away we have see SDS and parcels if that doesn't mean anything to you this because this seems specific speak so if you're using clutter manager then it should be pretty happy about that if you love working on the databases this is our github this is where all the commits go we're not we're not like the developing kuru internally a cutter and then shipping things over the wall we're actually directly contributing the code in the open all the time we also have a public Garrett so this is how we do the code reviews you just have to send your pastor Garrett and we will be will happily review it we have a public juror it has all the bugs dating back to 2013 so if you want to see all our dirty laundry and it's the same for all the get our github blogs are all there from all the way back we have barely scrubbed anything so this is all there apache 2.2 license this is real open source contributions welcome and also it is our intention to donate this to the apache software foundation this is not something we're thing right away but this is coming and that's the end of my talk thank you very much questions yeah who haven't I haven't talked to yet you haven't have you asked questions before all right yes so what was the reason behind using impala to do the parade your mo i mean you're quite right there was a reason to use impala could I've use a directly yes i could have written java code to query kudu it will be less interesting though because guru itself is only a storage engine it does it doesn't do sequel it's a bring your own sequel kind of thing so I could have done sparks equal using the data frames I could have done MapReduce will be a little slower than we did in Paula because that's like the one use case that we're trying to solve right now it would work well with our suppose if you have application today a you know as opposed to get it sparkly acquiring it was far sequel you could just swap out hdfs right so the question is you could swap out gfs like parquet basically right you would be querying against Park if you're doing data frames that are frames are scheme are dd's if you have a schema works well with kuru but works well with parquet right and using kudu would give you all the qualities / parkade that I've described before right this is not something special to impala like this like when I when we differentiate could with other things it's really cool to itself and those properties would translate yes now the data frame integration still lacking a bit we're working on it so don't go I mean you know I just want to do it but yeah but yes I could have done it on sparks equal question so do the question is do we ever plan to have configurable consistency we already have it to improve throughput like you would do like in Cassandra like he or calf Korea like right and then forget about it it's a good question that's so right now bulk loading is one of the big things that we're wondering like how we should we do it because it's not like other databases where you can just drop the file in HDFS and then tell the database loaded and then it's done we're running against the disc and the replication is handled by coud with a logical level right gets tricky so I don't have a good answer for that but like on the consistency tuning like you can go against any of the replicas in a tablet configuration they call it in a draft right let's say you want the very freshest latest data you will go against the leader of the corn if you want some data I may be like up to last day you can go against any of the rapper cousin I mean it's going to work like it will give you the data because it will have it and if it doesn't have it you'll wait other questions Frieza you were talking about pretty conclusion what are good predictors that you can move them okay the question is all right I've talked about push down what are the predicate for the moment it would be simple comparisons like smaller than equal greater than do you have anything else in mind that we're missing the question is do we know we don't keep we don't keep an index of all the columns except for the rookie so that means we actually have to read the columns themselves but as I was saying we have lazy materialization so let's say you have your reading 10 out of 100 and you're like doing aware on only one of them will read that one if it matches we read the nine others and bring the whole brain memory the question the other question is have we done any testing on multi-tenancy no but we've done a lot of concurrency testing but now we don't we don't even have for the moment things like namespaces that would be one thing that we're missing that's very obvious although you could manage it at a logical level like Impala ok so the way a fair question what about the different types of work those that are running on the same cluster well yes that's something that we've run but like not specifically looking at what the different impacts would be like in between the different use cases so still a lot of work to do yes other questions I hadn't seen one publication sparco connector can read all this is the part of our connector can it right obviously because it going to because we have a crew our DD I think in that demo it's directly using the client to write it's just a matter of small matter of code as they say in small matters progress probably small amount of programming smoke hood key metadata what is the perky minute that it over ahead I think it's I don't remember I think it's 32 bit it might might be less than that maybe wait isn't that I don't remember the thing is the key we keep like an ordinal index we don't actually repeat the key next to every value like they would like HBase does although that compresses really well it has other advantages let's say so that's the on this is that okay is there a perky in memory overhead the key not the key repeated itself at which cell like an interface now you have the key which is actually are now and if you don't if you don't need it we even we won't even load it like we'll just use the ordinal if you have a very long key for example but I guess my question was like what exists the question what is the moment and there's anything exists in memory so when you do worry back is it too low cops on desk or so so what is it that's being low what is the metadata to create those keys right so we have the we have we have a key index we have bloom filter bloom filters for every small chunks of raw rose that we keep in call them rosettes I I could go into the details we have if you're interested later so if you're specifically looking for key comparisons we have a key index if you're looking just to like it does this where exists you have a bunch of bloom filters that you will have to check like like it is this row in this row set No No maybe okay not now I have to read and then actually check the indexes in the files that's the work is what's actually yeah drives on the back end of Angelo's you fight so who is a corner store right so it has vertical stripes I know it's a really bad joke it's a thick its marketing thought about that but if you look at the picture of kuru it's actually what it looks like but blue but it has vertical stripes if it's blue smoke something other questions yes environment or performance or you know there are any possible problem with running with on amazon amazon or OpenStack or have we done any testing on virtualized environments and have you seen any issues and do you foresee any issues that would be the question so we've done a lot of testing on ec2 not for performance purposes though we've done the testing on docker the thing is the interface through docker to the whole system sometimes it's weird especially for ntp because we rely on ntp for the the clock I didn't go I didn't talk about the hybrid clock but there's this whole mix of logical and physical clock that we're using for the contending the consistency that that comes from spanner basically span are using an atomic clock but nobody hell nobody else's I don't like box in the data center maybe you maybe you do but we don't anyway so have a clocks to the rescue except it relies on my TP being synced so there's some great issues sometimes there how do you are you expecting in issues bucket he is not expecting any issues all right I'll take one last question and then we can have discussions with beer and whatnot all right you guys were a great crowd thank you very much you