SF Scala: Gian Merlino, The Rise of the Operational Analytic Data Stores

SF Scala: Gian Merlino, The Rise of the Operational Analytic Data Stores

Recording: SF Scala: Gian Merlino, The Rise of the Operational Analytic Data Stores

[Music] hey everybody so yeah I'm here to talk about I'm here to talk about your thank you for the people that said very nice things about Jordans last couple intros I really appreciate it I saw that a lot of you raised your hands for being here for the Scala Meetup George is not written Scala sorry disappoint you but I did write a lot of Scala in my last job and the purpose of that Scala code was to get data loaded into Druid so you know I still feel like there's some kind of connection there and anyway the title this talk is it slices it dices it streams the rise of operational analyte data stores operational update of stores is a class of data store that that I think sure it is I'm gonna talk about what your it is where it came from why it exists what is this class of data stores and what's it good for and and how that fits together in the data world so Who am I I'm generally no I'm a committer and PMC member on Druid which has recently started the apache incubation process so it's now apache german parenthesis incubating the parentheses to be removed sometime early next year and I'm a co-founder to imply a company built around rid and I spent 10 years working on scalable systems I worked on data center automation and configuration management server installation of Yahoo then I was worked on you know like I was saying Scala based ETL pipelines both streaming in batch and now working on Druid so what I want to talk about today first the problem we're trying to solve accurate then I want to talk about operational Linux I said we're an operational I like datastore what does that mean I made up a term and might as well tell you what it means under the hood of joyed alexey tells me that this is a serious meetup group that is really gonna be interested in seeing some insight into why druid is good at what it does and so there's an under the hood section mysterious future if I still have your attention at that point you might be interested in what's on George's roadmap so I'll talk about that and then if I still have your attention after that how to try it at home or to download it how to get started enjoying the community so first off the problem I always start with this it's important to understand what problem you're trying to solve there's a lot of technologies out there that part is fun but we're actually trying to do something and the problem that we're trying to solve is to build this it doesn't have to be this UI this is the start work and imply this is the imply analytics UI doesn't have to be this one there's also other you guys that work on top of Jun like Apache super set but the point of this is we want to have a interactive slice-and-dice visual experience high concurrency low latency we want people to really feel like they're interacting with big data we want to sort of make big data feel like small interactive data and that's very challenging it requires big clusters it requires a lot of advanced techniques that I'm talk about in the talk but what we're trying to do end of the day is build this kind of interactive experience on big streaming data so the problem slice and dice or big data we want to do interactive exploration we want to be able to look under the hoods of report and dashboards so we don't just want to update a dashboard in real-time anyone can do that there's technologies that do that what we want to do is we actually want to be able to have it be fully interactive and unconstrained way and we want to get it fresh too so we don't just be doing this on historical data we want to do it on historical and streaming data together there's a lot of challenges at this I got six bullet points here so first one scale when data is large we need a lot of servers the biggest Droid clusters out there are over a thousand servers and that's in some situations the kind of capacity you need to be able to get that kind of interactive experience on truly Big Data trillions of rows speed we're aiming for a subsequent response time even with terabytes or petabytes of data complexity is a challenge to too much fine grade to pre-compute when we first started working after a time when Jordan first created this was about 2011 2012 a very popular technique was to pre-compute roll ups in things like HBase and Cassandra and having you guys read those out that technique doesn't scale very well it doesn't kill every well because of the next point the high dimensionality if you have tens or even hundreds of dimensions the amount of possible combinations those dimensions is so high the pre computation is just simply not an option you see people saying things like my system works greater than five or six dimensions I can't scale beyond that so to be able to handle that kind of dimensionality you really have to move beyond pre-computation and get into filtering scanning and aggregating very quickly which is what your it's good at concurrency matters too so you have many users many tenants sure it's commonly used for multi tenant workloads where you have a lot of different tables that are just not related to each other from different customers perhaps it's also used for multi user workloads and it's used for heterogeneous workloads so you might have some traffic from a UI that has to be really snappy some traffic that's offline reporting some traffic that's API driven some traffic machine learning driven so different kinds of traffic needs a coexist in the same cluster and not block each other freshness is also a challenge we want to be able to load from streams to get that real-time visibility so this whole architecture has to still work in a streaming world the motivation for all this is that if you can get sub-second response it's on your big data it allows you to have a dialogue with that data it allows you to iterate rapidly on the questions and removes barriers to understanding so that really helps you understand what's happening and it helps you react to new changes very quickly and it changes the way you think about data it helps you think about her so that brings in operational analytics which is what you when you get when you can put all this technology together so I see Giroud as being a part of a new class of data store there's a lot of examples out there I'm biased toward sure and I think it's the one with the biggest community the most momentum it's it's some of the most advanced technology there's other ones out there though one of the examples Guba from Facebook that proprietary it's not open-source you can't use it unless you're at Facebook but it's got similarity secured in a lot of ways they use it for similar things over there slicing and dicing to an operational data something called Pino from LinkedIn architectural II very similar to Jared Doris from Baidu that's a I think it's a fusion of Impala and the custom back in that they built and then click house from Yandex another data store that's the silver Jordan a lot of ways but these all have in common is these they have a few factors in common that enable them to solve the problems that return itself one is it the column oriented so column orientation is super important it means all the data for a certain field is stored together takes advantage of the fact that most queries don't hit most columns so we're trying to deal with complex data that's got tens or hundreds of columns most quarries are going to hit you know six or seven columns maybe maybe a little more a little less we only want to scan and arrogate those columns that are actually relevant to our query that speeds it up by a huge factor high concurrency these stores are all designed to deal with high concurrency and they have to have ways of dealing with that they have to have concepts like prioritization they have to have concepts like you know resource management you want to be scalable so you have to have a big data datastore that can be scalable the hundreds of servers even over a thousand servers millions of messages coming in per second so no matter what kind of scale stream you're dealing with you're still able to load that into the datastore and do queries on that data stream a partition key for query pruning tends to be really important to achieve performance so these kinds of data storage Druitt included a lot of tricks to get good performance you know the core what they're good at is filters can aggregate so there's tricks to be go-to filtering tricks to be good at scanning and tricks to be good at aggregating that career pruning is a very important trick to begin the filtering so when you take event take advantage of the take advantage of the fact that most queries are gonna have a filter on some very common column so in Druid we always assume you're gonna have some kind of time filter and you might have some other filter tubes maybe time plus customer ID or something else and and you can partition by things you commonly filter on to make it so we can prune down the list of data that needs to be queried so you don't have to scan everything these kind of data stores may or may not have secondary indexes sure it does and secondary indexes so the primary one is what you're petitioning on the secondary indexes other columns you're not petitioning on but you still want to have some kind of index within your partitions so I'm druid we index every every field that's that screwable or filter bold within a partition and so we prune down with the time and with some other dimension possibly and then we have bitmap indexes we use to filter with fitness segments they'll talk about a little bit later querying through sequel really common this is sort of the common core language or these kinds of data stores enables integration with the applications that you want to put on top of this to get the best value out of it and then finally rapid queries on denormalized data is something that they have in common which is you don't want to be doing baked joins at query time you want to be pushing that to the ETL process or is in the streaming world should be streaming ETL so you're using something like Kaka streams or spark streaming for example to join data beforehand create a joint stream and then load that into a data source like Jordan so there is some pretty common use cases for this what these use cases have in common is their use cases that have a lot of big data so that generate data at scale and where there is value in being able to analyze them in a streaming way and an interactive way so click stream user behavior things like AP testing and understanding what your users are doing at your site or your mobile apps application performance management understanding how your API is and servers are behaving understanding how your containers and micro services are talking to each other network flows is a really popular use case for doing so you know getting flow statistics from IP to IP port to port Internet of Things Digital advertising is where Derek got started a lot of the very earliest use cases for sure were in digital and then sort of all that business intelligence so the kind of slice and dice reporting on business type data so that's enough about the general class of things operational stores the rest the talk of a talking about sure it's specifically as sort of an example of how we approach these problems and also because it's the one that I know the best so if you go to the Droid website which is currently druid that IO will become George that Apache don't work pretty soon you're gonna see this tagline high-performance analytics data store for event-driven data I want talk a little bit about what that means so we unpack it a bit high performance means we're shooting for low core latency high ingest rates analytics means we're doing core is like counting ranking group by time try at least tend to be aggregation style queries the data store part is important it actually stores a copy of your data and the reason we store copier data is for performance reasons so we're not coring out of s3 HDFS were never actually storing a copy of it on the dirt servers and then event-driven data that goes back to the use cases so you data where every row is an event something that happened you know a click a network flow a user behavior etc someone saw an ad someone something happened on a sensor something happened on the server druid is really widely used the products been around for about five or six years we picked up a lot of adoption in that time we we put ourselves in the center because we're full of ourselves but there are a lot of the people using it too we have a very modest during cluster thing we have a tech node cluster a lot of these other clusters are a lot bigger than ours we got a very yahoo is made a very big investment in Druid they have a few commuters over there they are using jury for a ton of things and they did a very nice piece in info world they they have a quote the performance is great some of the tables injured we have billions of events in them and was getting them in under a second so that that shows you the kind of thing that we're trying to achieve and actually are achieving at scale so the key features of Jered low latency ingestion from Kafka and I say ingestion from coffee then we're not actually querying Kafka so jared is able to connect two streams but I wouldn't call druid a streaming analytic system to me streaming analytics means you know I'm connecting to a stream I'm continuously computing a query I may write and result somewhere continuously jure is a little bit different jared is ingesting data from streams but then storing that data locally in an optimized way optimized for with indexes and column orientation for fast filtering and scanning such that you can do those queries on the very latest real time data but also on historical data and still get some second query performance even on oh that historical data so a little ANCA Dustin from Kafka bulk load from the do so supporting both a batch world history and world we made routed a it might have only been streaming but we made sure in five or six years ago and supported both you can pre aggregate data during ingestion which I'll talk a little bit about later that's one of the tricks that we employ to get faster scanning aggregation schema light is one of the nice features of druid so schema light means that you can evolve your scheme was easily you can change the type of a column you can add columns gonna move columns and none of those operations require rewriting your table they're all very cheap operations ad-hoc queries so I guess thing unlike something like a string analytic system like like Kafka streams or what have you you don't need to know the queries you want to run ahead of time you have to do you do have to give some structure to your data there's a data modeling component but within that model that you've defined you can do any sort of unconstrained ad hoc query exact and approximate algorithms that's another one of the tricks that we have so actually in the early days of jury we only supported approximate algorithms for things like counting distinct and ranking so you know top hundred count distinct number of users etc we at some point added the ability to do those exactly but I think you can really understand the DNA after when you reflect on the fact that initially we only support approximate so the DNA is really to do things as fast as we can and a lot of cases the approximations are the way to do that the ability to do exact stuff is sort of an odd desire for flexibility and the ability to get exact if you need to we can keep a lot of history so here's Visser you're okay going back to the point about we do sit on top of a stream but we're not just coordinates in that stream so even if you have in your Kafka a retention of two weeks you can still store your xandred so I had to sum der it up in a sentence I would say it makes interactive data exploration fast and flexible and it powers analytic applications that's really wide you it exists and and what it can do for you so the under the hood section I promised [Music] let's see so here's the data set I you may need to forgive me I think this might be a version of the slide deck where the things don't totally add up so if you notice that it's not your fault it's mine but this is a network full of data set so we have a timestamp it's very simplified timestamp action protocol flows it's actually coming from a firewall so these are the first one is something far well accepted TCP flow accepted no tcp was rejected UDP flow we don't have a source and destination IP here or source destination port we would if it was a real data set but it's not here because the screens not wide enough but this is the data set roll-up is this so I said I would mention I would talk about roll-up earlier roll up is our aggregation and ingestion time it's one of the big tricks that George pulls so I said earlier we don't pre compute so we're not pre computing every possible combination of Cori you might do but we are pre aggregating and it's different because pre-computation tends to blow up the size of your data but pre aggregation tends to shrink it and because Giroud is geared not towards finding the exact right pre computed answer if geared towards this filter scan our gate motif anything that helps reduce the number of rows is helpful because it reduces the amount of things we have to scan so what we do is we have a model that's familiar to any that has done anything in the OLAP world we have a dimension and metric model or dimensions or anything you're grouping and filtering by and metrics or anything that is an aggregation function so the idea here is that our our dimension model is the hour of the or the day of the timestamp so we're not retaining information within specific minutes or seconds such that just the day and the action of the firewall and the protocol of the flow and then the number of flows we're treating as a metric so we're just storing the aggregate number flows so we're not storing every raw record we're storing if two records have the same day the same action the same protocol we're collapsing in the same records summing up the result this is an optional feature injury but it's very commonly used I'd say that the majority of production installs I've seen use this and the reason is because dirt is really commonly used for doing these slice and dice idle MQ eyes and the things you want to slice and dice are the things you make your dimensions for roll up at 10 how much it tends to help varies from from use case to use case and data to data set but in a lot of worlds the compression you can expect from this varies from 2 to 3 to 1 to maybe 15 to 20 to 1 and so it can be pretty substantial if you turned this off then Georgia stores the raw Rose one one at a time one by one and that's useful if you're interested more in sort of a search type use case or a select type use case where you actually want to get the events and if you're an aggregation you want to do this so we support doing this again at add injection time so you don't have to have your your stream processing table do it another thing we do is we split up data by time so a lot of use cases for Girod there's gonna be a time filter on most queries and we split updated by time to be able to prune for those queries very easily so if you have excuse me if you have two years loaded and druid maybe you have seven per day so you got a few hundred segments if you have a query that wants to filter on a particular week only seven data segments are relevant to that it's a relatively small fraction of the overall data set and we've pre partitioned on time to make it so we can narrow down to that seventy that very quickly I said the word segment a bunch of times then we talked to what it is just a brief aside as to what a segment is enjoyed and said it's one of the core concepts in truth it's the fundamental unit of storage each segment is going to be a few million rows so you know a table with billions or trillions and rows you're gonna have a lot of segments they're usually about a gigabyte or so they're immutable once they're created there's no and because they're meaningful there's no contention between reads and writes bringing it back to the scholar of immutability no connection between reads and writes and that enables very high concurrency which is really nice another trick that we have compression and columnar storage so when we store the way the so I mentioned droids columnar it stores everything that's for the same column together this is something that is common to a lot of columnar data formats like orc and parquet so the way that droid does this storage is no different from the way that something like parquet does it in all those cases you know we're going to take these strings we can Dictionary encode them we take the numbers and we apply some integer encoding on that we don't need to actually literally store the word project over and over again we can code that as a one quote acceptance to zero and store the erase one thing that we do enjoy that of stuff like mark confirming doesn't do is indexing so indexing is something that is a little bit of a less common feature this helps on the filter step so we think about the filter scan an arcade pipeline indexing really speeds up filters dramatically it makes it so when you're drilling down into a really small part of the data to try to get that little chunk you're looking for that you get you get to get there quickly without doing a full scan so what we're doing here is we will store a bitmap for every possible value in each columns over the action column there's two values accept and reject so we're storing two bitmaps and that's so that those first two bits are set representing that the first two rows have accept and the next few bits are set representing that the next few rows that project and if we want to do a filter like accept or reject we know that every row matches if there's a filter like I want to find something that's accepting and rejecting that nothing matches that so we don't even have to scan anything and if we want to find you know all the rejects of TCP then we would combine the rejected math and the TCP bitmap and get a final one and then only then we would scan the data columns for the row numbers that match that so the fact that we do this is can really really dramatically reduce the amount of time it takes to resolve a query that hits a small fraction of data set which helps a lot in a couple situations it helps a lot multi-tenant situations so if you've got you know imagine a world where you have you're running some sort of service you have a thousand customers you're putting their data and ruined each query is gonna have a customer ID filter it's gonna have a small proportion of the data this helps a lot another situation helps is when you're doing drill down if you think about interactive analysis a lot of times interactive and analyses start with a high level picture people drill down drill down drill down the farther they drill down the faster their quarries get and and this is what makes that happen so it's a little bit of a nice duality with caching so we've also I don't have a slide on caching but there is a caching injury caching tends to speed up the high level quarries of an interactive exploration the very drill down ones tend not to be cached but they're very fast as the indexing another trick approximate algorithms they save storage memory and time so in Jordan we've got approximate Holograms for count distinct we actually have two we have well we have three we have a hyper log we have two hyper lock augmentations and one the discretion plantation the the third one the second hyper log log I think is you in our next release I think it has been released yet but it will be soon ranking we got one of October than there an approximate top K algorithm top an algorithm histograms and quantiles so we have got two there and actually two more on the way it's fun to have a lot because I difficult forwards characteristics different trade-offs between storage and speed and that kind of thing so it'sit's we're actually working on a blog post so check out our blog soon for for the histogram as a quantile specifically about the trade-offs between them in terms of storage and performance and accuracy and then stet operations that's a really interesting one that lets you do things like retention final analyses you can say you know how many users logged into my site last week and did not log in this week so that's a it's like a count to stink but it's account distinct with a difference and so that's something that fetus caches can do which was a feature contributed by Yahoo they use it for for that kind of stuff retention FL analysis I talked so I talked a lot about the data format of Turin I talked about that because to me that's that's the heart of it the heart and soul of what makes sure what it is is how it lays out your data how it burns your data how the core is run but I don't talk a little bit of the architecture just so if you do stand up a dirty cluster you know what you're seeing the dirt cluster has a few different server types I don't have a mall picture here these are the main ones for the quarry path so Corey's going to what's called a quarry broker that quarry broker will send sub queries to indexers and historical indexers are in control of reading streaming data and batch data actually the ones that read batch data don't get queries the batch data is sort of its quarry and once it's loaded the thing that makes you in real-time is that the indexers that are sitting on streams and preparing those immutable Giroud segments off those streams will allow you to query them before the segments are done being built and so that that lets you quarry that most recent hour to data the historical czar in the part of turret that in most clusters of the majority of it so you know if you had a note your cluster you're probably gonna have maybe five brokers maybe 15 indexers than maybe 80 historical xiv probably how that would shake out and that's because you're gonna have a lot of retention on the historical and your indexes are focusing on the most recent data this kind of architecture allows you to scale each part independently and it's really cloud friendly so you know if someone who's running this inside a docker based environment or a container based environment you would put each these in their own container and you'd scaled them all independently so as promised if I still have your attention I'll talk about the roadmap a little bit I feel like talking with the room app is nice because it gives you a sense of not just where we are but where we're going one of the big things we're doing the Droid roadmap parallel loading of data files without Hadoop so I mentioned that you know we load from streams from Kafka Kinesis but we can attempt a Hadoop for look match data this is a Hadoop Meetup but I don't know I feel like Hadoop is is dying fighting me we're looking forward to a world where doesn't need Hadoop tool a batch data so George can load data directly from something like s3 we're seeing a lot more cloud first deployments and the cloud first deployment where you have something like you know your public cloud provider has an object store there's no relief or HDFS and then you have specialized data stores like Jared means there's no need for MapReduce and so between those two between those two factors it's there's less of a place we're gonna do but especially in the cloud centric world so reducing our defense IANA D was important to us automatic compaction is a big one so you know that's just making life simpler for people may consider I was worried about so many things when you're running a cluster smaller faster compression so I talked about the tricks that we pulled to get our filter skin our game pipeline very quick the smaller faster compression is about speeding up the scan and aggregate the pipeline subtotals sequel grouping sets that enables query fusion which is really cool so you don't mention that it's really common easier to power you is what a thing UI is often have is got like a bar chart and then on top of the bar chart it says what the total is across all the bars or a time series total totals are part of a lot of visualizations right now you're doing two Coreys to get that one for the total and one for the breakdown but quarry fusion allows that to come into one quarry and reduce the load a lot single standard null handling so right now we handled knows a little differently from the sequel standard says this is kind of like a some people super boring some people super exciting depends on what you low standards bloom filter filters that lets you do approximate semi joins let's talk more about it after the meetup if you're interested and Dan Akroyd ization so I mentioned you know we care a lot about high concurrency low latency heterogeneous workloads then our core so right now view one of the things that enables that enjoyed these set query priorities you say you know how this this workloads higher party than this workload but wouldn't it be cool if Jared could set those parties dynamically so you could just toss a bunch of garbage to cluster and have it handle it that's that's about and then your item here it's open source you can become involved we were just we're starting the batching process we're really you know trying to expand our community and talk me out towards if you're interested and try this at home so our current community site Jared do our future community site generate that Apache dork which we go to it has a bunch of nice words on it and it links to Giroud i/o and then the imply distribution so imply my company built around your head we have a distribution of jerd you can download we also have a cloud enabled one that's fully managed you don't to worry about anything you just click some buttons and starts up a bunch of ec2 instances and it's awesome and so that's that you can contribute we're on github that's where all the development happens we have already moved over to the Apache github org so github calm slash Apache / druid and check that out and stay in touch we're on Twitter and we have a bunch of kidney channel's mailing lists all kinds of stuff it's awesome I think that's it yeah so I can bring the mic to who wants to ask and I'll ask you to repeat the question using this my shirt gracias yeah so that kind of plays into the schema light aspect of Jared so enjoyed null data and a column that doesn't exist or sort of treat the same way so if you don't have a column at all it's it doesn't get stored if you have some column it's part null and part not annulled and the ones they're nulls get treated the same way they get grouped the same way with rows from other from other German segments that that didn't exist if there if there didn't exist there I don't know do that I'm not sure if that's what you're asking oh do we do data wrangling ahead of time so druid has some limited support for data wrangling we have this thing called ingestion time transforms where you can run some expressions on data as is coming to droid if you're familiar with the Kafka single message transforms that they have in their consumer layer and it's kind of like that so you know anything that can apply to one message and modify one field so apply a regex to a field apply function to a field can cap two things substring replace that kind of stuff anything more sophisticated than that and you'd want to do I should have there's there's a slide I didn't include here that I should have included which is what a typical architecture looks like but a typical architecture imagine you have data coming in the Kafka then you have an enrichment layer and the Richmond layer is going to be Kafka streams or sparks streaming or k sequel or something like that that's going to do any data preparation or wrangling that that needs to be done that jury can't do on its own and then that wrangled topic will get loaded in android and you can skip the enrichment if it's only doing things the jerk could already do or if your data is already perfect but a lot of real-world pipelines will have some kind of enrichment step which top of this Java very right on it we tried compiling with 10 and so we didn't work figure out optimization how hands-on a deposition what affiliate yeah so right now bitmap indexes are the only index we offer were so there's actually a community project going on now to add a numeric index is so that would be like a range based index for number of columns the bitmap index has worked better for string columns as far as other opposition stuff there's I would say that that in real world clusters a lot of the optimization stuff people do revolves around trying to get better roll-up so it's being creative around using approximations and sketches to roll things away so you know getting rid of IDs and replacing them with hyper log logs if all you want to do is just ink counting so data modeling stuff like that the way you partition data can make a huge difference so you know if you do have something you're often filtering on petitioned by that makes a huge difference and then I guess the other thing that comes up a lot is if you do have a heterogeneous workload it can make a difference to use priorities appropriately and use the fact that that you can sort of route queries different ways and system to route sort of the queries that you want to prefer to the queries you want to ant I prefer it if you know the going gets tough if it becomes more this isn't can handle what are some of the advantages and disadvantages when compared with Cassandra so advantages and disadvantages Giroud vs Cassandra I would say that Cassandra is used for a lot of stuff so George druids pretty much used for the kind of stuff I'm seeing up here so you know real time slice and dice analytical interactive stuff Cassandra is sometimes used for that and I think it's usually a mistake it was really it was really popular for that because in kind of like 2011 2012 vintage it was the only it was one of the only fat and HBase where like had the only things that scaled to the level that that you wanted to scale to but it's not really technologically a good fit for that kind of project so I see people these days using Cassandra a lot less for analytics and using things like Jude a lot more Cassandra is still good at it's what I would see is its sweet spot which is if you really have like a key value based workload or you're doing lots of you know writes and updates and that kind of stuff which is something transactional almost a question so you know we have thousands of books of streaming including spark and playing right so companies that comment in the nature of everything you mentioned first of all times from Kafka and tundish so why would people and Julie to the pipeline if they can try to implement commenting in whatever swimming it is they're using right what kind of use cases do you see and why you know is it better to use dedicated accounting and analytics layer to some distributed system because that's some complexity right because now in fact manage basically ETL between the systems and so forth so how do you see this practice is why would you recommend you know people do that yeah definitely I would say that so I went back to this slide because this is this is to me the answer to me the answer is if what you're wanting to do is get the sort of like interactive exploration experience on top of streaming data you need something lighter I think that that the stuff there's stuff out there that is good at doing big counting on streams but a lot of the technology there revolves around knowing one count ahead of time like you know having a continuously updated list of top 100 pages is a lot of technologies can do that pretty much anything that says it streaming analytics can do that but if you then want that top Punja list of pages to be filterable by any combination of fields you can think of a lot of so stock really handle that Madrid does so that's that's when it makes sense to bring into the picture partitioning that are charting at five times name okay that's kind of what I'm interested in is accounts for users where a user session is defined by a user of this series of clicks they're less than 10 minutes apart and if they're for 10 minutes so that is that's really good question and I appreciate your a step because it speaks - it speaks the two things it speaks to something oh yeah the question is let's say I have clickstream data and I want to count it's a sort of event oriented and I want to do session or the analysis I want to count how many sessions match certain parameters yeah it's interesting so there are people doing that Madrid I would say that there's basically two approaches to doing that one approach is so you got two session eyes the data somehow you got to take the event or its data and make it into session rate the data and there's two ways to do that one is to do it before it gets in the Droid so you have something that is like a copy of streams job that's basically doing a group by session ID it's building sessions and then it's sort of like a self join on the stream and then writing out a row per session to another Kafka topic and then you load that enjoyed so that's a pretty common way of doing it and it lets you do a lot of stuff you basically have to say you know you might have a dimension for like does this session you know something may I have a metric for like how long the session lasts and a dimension for like did certain things happen in the session and so on and so forth there are some use cases here and there I don't know if you fall into this where pre-built in the sessions doesn't work for ever reason I would see the biggest use case that falls into that is you want to be able to do analyses on number of sessions where certain events happen in certain orders and you don't want to necessarily pre compute that all ahead of time that's the big one and the there are people doing that injured today they're doing it with a basically custom extensions and the way they do it is the way they do it is they partition their data first by day for example and then by session ID so first by day them or set by user ID or whatever and then you have all the rows for the same user in the same day in the same drawer in segments in order and because they're in order in the same segment you can do an optimization where you basically push down the session ization to the to the segment scan process and out-of-the-box Chur doesn't do that so the people that are doing it are doing it with custom query UDF's and custom query types that they do as extensions but it's something that i would really love to make a more out-of-the-box thing if that's a helpful answer I hope it is how would you compare this with LinkedIn Pino is a very different or is it you know five times better or three times yeah I would say that architectural II it's very similar P knows a few years it came along a few years after Jared a lot of the nodes have the same names they relate the same way take that as you will the jury community stuff is a lot more vibrant and it's definitely lot more of a mature project [Applause] [Music]