sfspark.org: Evan Chan, FiloDB: Combining Spark Streaming and Fast Ad-Hoc Analytics
Recording: sfspark.org: Evan Chan, FiloDB: Combining Spark Streaming and Fast Ad-Hoc Analytics
hey thanks so much for hanging around this whole time very sorry I was late you know it's funny I don't live in the Bay Area so is the one time I come down to the Bay Area what happens is I get on a cow train and then the next thing you know like you know with big accident so but I don't know in case you're curious actually I don't think our train hit someone they thought we hit someone and and then they said that we didn't hit anyone but then the train behind us hit someone and when you hit someone what happens that when you know that the crew has to be dismissed so at Palo Alto station you know they dismiss the crew decided it could be like 60 to 90 minutes you know so anyway so thank you for your patience if any point weighs anyone heading back down to the South Bay after this oh sweet that's awesome umm I wouldn't mind a ride so if one of you guys are interested in giving me a ride back to the South Bay I'm all yours and you can pick my brain that's okay that's okay with me yeah so yeah so anyway thanks for hanging around my name will well anyway hopefully you guys are not in a hurry or anything I can hurry through the slides but if you're not in a hurry I can go through all my slides so my name is once again my name is Evan Chan I'm an engineer to put jump I actually just joined two months ago I'm quite excited I have a colleague of mine mine here Helena and it's pretty awesome because we're distributed so we see each other at meetups conferences and that kind of stuff and I just a little bit about myself just my Twitter handle if you want to follow me that kind of thing I've been using doing big reader stuff for a number of years working with Cassandra since very early days since the one dot before one dodo actually dot six and working with sparks since about about eight and contributing to it and stuff I remember when I first started working with spark there was there was a really neat little project but there was almost nobody using it but you know it's been amazing to see like now it's like everyone you can't go to a conference without talking about spark pretty much right and I also maintain the spark job server project which if you're curious I can talk to you about it later it is a rest server for running your spark jobs a little bit about to pohchamp we are a big data technology leader and we do we provide solutions and development partnerships I guess that's a nice way of saying that we provide consulting and you know how help architects and do projects we have a couple of really pretty interesting projects including an open-source HDFS on Cassandra implementation as well as the leucine indexer and file DB is the subject of today's talk and we have an alternative spark isandro integration layer this is a mandatory slide for me to show that you know we offer Big Data development expertise all kinds of different stuff we've done a lot of interesting projects including I mean traditional like help companies ramped up their big data stack as well as some whatever'll interesting projects including porting open source projects to spark and things like that so just this sort of day we'll have we got to have a different cartoon right so give you minute to look over this since about analytics so anyway so I saw this quote that pic did is yesterday and fast data is now how many people have heard of the term fast data like I think typesafe has talked about it the idea is that fast data so big data everyone knows you know you have more data than you can handle on one node the idea of fast data is that you know now we have data that that's coming in that's real-time coming up from more and more sources so you could have sensors right there's a lot of talk about smart cities so that cities want buses to be constantly transmitting information and then they want to use that to maybe improve how they plant a do parking like or like give people you know give people tickets in real time or this kind of thing you know more revenue for them and of course like you have the likes of like you know uber and nests and all kinds of devices right that are constantly sending data and so every company is getting a deluge of data from all kinds of sources and you want to react and take advantage of it right away so that that's what fascinator means is like data that's coming in fast reacting a faster data that that's what that's what it means by fast data right oh I guess I'd have it what it means slide it's not enough to just react o to data as it comes in right like so this is if you want the dashboard so you want to find out how many people are using your service right away that's that's really good but usually you need to combine it with some existing there as well right you need to so you need both you need fast data and you need big data yeah I see I see I don't know if I have anything to add about that but I think you're right that yeah it you you want to automate the whole cycle like you come in and you can derive insights and if someone has this it does have to make business sense right like someone has to actually want it yeah I do agree with that sure so let's talk architecture a little bit how many folks have heard of lambda architecture seems like seems like that's become a very popular buzzword so this slide is from map R so in case you're not familiar land architecture what what it what lambda R Kirk says is I want to deal with data that's coming in and the big data so what I'm gonna do is I'm gonna set up two pipelines I'm gonna have a speed layer that takes data data and process it in real time using a system such as a storm or now there would be like spark screaming and so I'm going to do incremental calculation and then kind of write it into some kind of usually a key value store in this case it's HBase but I'll be talking about Cassandra a little bit and and that's good for the real time stuff but we don't really trust our speed layer so we're going to have a Bachelor that's going to you know really capture the same data and it's gonna compute it and and do the exact calculation right so this is the idea of lambda and at some point I have so now I have data that's computed with the speed layer and a Bachelor and I need to merge them right because now I have kind of two answers I have like this answer for the short term stuff and I have answers that have been compute over historical stuff now I need to merge it so that's the idea of the Lambo architecture and you can see that this involves a lot of different technologies like there's like this the real-time system and the Hadoop system and you gotta like have a key value store and you've got to combine it so what happens when you start implementing these things is that oops sorry you you'll find that this is actually pretty hard to implement you got a lot of moving pieces you have to actually deploy and productionize all those things you have to monitor all those things so this this is this say I mean this is a lot of stuff to implement and a lot of times you might be running similar code into places because if you're doing aggregations chances are you'll probably be doing accusations in both your bachelor and your speed layer so and you have to reconcile queries against two different places so this is a lot of work right and so we've to ask ourselves is it possible to do better so I'd like to start a new hashtag called know lambda and know let the idea of know lambda is we want to build more of a unified system we don't want to build two systems a speed layer and a Bachelor because this is just too much work right so one and we want to explore what does it look like to have a single layer that is fault tolerant that is precise but can give us real-time processing as well as big data analytics that we combine fast data and big data right so I'd like to talk a little bit about the smack stack this is what smack stands for is Scala and spark amazes akka a Cassandra and Kafka I will talk about all those things but I'll mention a few sparked screaming is how many folks have not heard of spark streaming also so so this is an overview right that data comes in the idea of spark swimming is then you compute mini patches which are from half a second on up for most people that's adequate time skill and it runs the same same computations that spark the batch layer compute is this running them in much smaller intervals right so that gives us our stream processing layer we have Cassandra which is a key value database I think how many folks have not heard of Cassandra here right so yeah I don't think I need to go and do this too much but the idea is you need a datastore to write your stream process answers to it's fairly easy to get up and running with doing analysis on top of data and Cassandra we just weed it out of a Cassandra table you can if you want to do SQL you can register it in spark as a data frame then data frame do you have a couple of options to do it you have a DSL as well as you can run SQL query right so this this is also oh yeah right so cash table so data frames has a table so everyone know probably knows you can cache rdd's in spark and that keeps it in memory but sparks equal and data frames actually has a different table caching mechanism this is different from already to caching it's a lot more efficient it actually turns the data into columns and caches in a very efficient way so it takes up less space and it's also much faster reading so this is something that you could take advantage of to make your queries very fast so we can say that spark provides the missing analytics piece to Cassandra Cassandra is typically used for key value lookups but what spark you can run a lot of deep analysis over it but what are we missing so so far so far with the smacks that we have spark streaming that can do stream processing with Cassandra that can do storage with an analysis engine on top so you let's say you're writing events now in the spark then I can analyze it so what are we missing oh sure how do you pull that data we need to keep the real-time streaming you cannot ask that you go down so you need something in between thank you so there's Kafka the the queuing system which provides buffering yeah well that's what there's many reasons why you would want to use capture but that's that's one reason is is that you can buffer and no no no cache tables only inspark yeah yeah so what I would what I would argue is that what we're missing is reasonably fast analytics on top of Cassandra so Cassandra is very good at storing a lot of small values concurrently and reading them out but it's but it's really slow when you want to read a lot of data out that's one thing that HDFS is very good at is reading a lot of data but it's it's kind of the opposite right so basically reading a lot of data out of Cassandra is very slow you'll find that like compared to the same data in say Parque it's about say 100 to 300 X slower and and so that becomes a bottleneck when you're trying to do anything that requires speed like today people use SPARC and Cassandra for things like ETL and loading it and loading it out but they're not really using it for analytics then I would say that's probably one of the key bottlenecks what about Parque so Park is so today a lot of people are you know writing if they do analytics that they would write if they're doing key value lookups they were right in Cassandra for doing analytics they write in the park a and Park a is really good at that for static data the problem is that being a file format it is read optimized but it kind of really sucks if you're trying to write data into it especially for streaming and especially if you want to do time series or ilt or anything like that it's one one reason is that it doesn't support a dumpling right so it's not like a database where I can write into a key a primary key and if I write to a primary key again like Cassandra it you know that it would write to the same place so in parquet you you end up like let's say you have a failure somewhere someone your pipeline you don't know where you end up having to write data again and so typically what people do is they will run jobs who duplicate your data but this adds a lot of overhead exporting data or writing setting up a parallel pipeline from your streaming source into a park and you know it's also another step and what I would argue is that what people really want is the database like abstraction and not so much a file format right like what we really want is let's say that we could have one pipeline for our fast and big data we talked about the smack stack earlier so this is smart streaming as to s and we have 2k here and I don't show meso czaka but it's so it's somewhere in there there's Cassandra like what if we could make the stack work for fast data ingestion as well as for big data for at all kinetics and that's what I want to talk to you about today there's this quote that I like it says all hard work leads to profit but mere talk leads to poverty so for a long time we talked about architectures that there's got to be a better way so now it's time to do something about it and so that's why we decided to to write an open source file of DV so this is the URL here it is honor we poke on to Poe jump what the tagline is that this is a distributed version called inner analytics database that is both for streaming and so rest for tonight we're going to go into different parts of that and how that solves this problem so first what is it it is it is distributed it is currently built on Cassandra as the storage engine which is you know very rock solid and it's been proven with you know kind of like a peer-to-peer design that has no single point of failure and all the stuff don't need to go into that too much it is versioned I will explain what more what this means they but you can think of it as like it's kind of like good like you can control the versioning of your data yourself this is a pretty new concept you can add a few versions of rows as a new version and you can roll back changes it's very very neat but I'll go into this a bit more later it is colander now there's a lot of like Cassandra itself sometimes talked about it's columnar database but what I mean by columnar here is that it is like parquet where I group I store things grouped by the column of my data so that I can pull out for for analytics typically you want to run you ask questions about a subset of your data like I might want the highest price average or the top ten items in this or that right so typically for those kind of reports you're you're only pulling out a subset of columns in your table so why I have to scan every column right why not pull out just data that you need oh I'll explain that in a bit yeah this one yes yes yes really right Cassandra doesn't by default I'll show I'll show you what you do bit later we basically we basically store data in a binary block format so that so that it's a lot more efficient but I'll actually show you what the table day looks like and you can do spark sequel queries on it so you can run all the standard sequel that you're used to including joins and things like that you can create a temporary table in a hive spark thing and basically finally be lets you use the spark data frames API that you're used to for and so you can easily pull data from any source that spark supports you can write into a new table easily using spark data frames and you can pull data out and run SQL queries and another advantage of using spark this way instead of like some databases use JSON or their own query language they said you can usually connect it to industry standard tools like tableau or any other JDBC analysis Center and you can plug into it and you know run run your reports no it doesn't leverage it because the way that we store the data is very different so we need to pull data out and reassemble it in a different way than it's not standard Cassandra records which maybe I should just skip to well if you wait a little bit I'll show you the slide that shows our table layout and you understand yeah we have a slide yep what's in the name well I like food so because it's a version database you know we it's basically based on phyllo dough on layers of data and you can also think of columns as layers of data and you know I like desserts so anyway that's where the name comes from and finally it's built on the types of reactive platform is built on Scala spark and akka actors so it's it's very fairly easy to contribute to so why why do we care about this and you know why should I use that I would say a big reason is performance it offers a huge performance boost over native Cassandra for analytics queries pretty much on par with park' so we get park' performance but you get the flexibility of Cassandra meaning that you get primary key based ingestion you can write data the same key and know that I'm getting exactly once ingestion and I'm getting the benefit of being able to write data in and update it all the time which is I don't fare lis unique and hard to do basically combining streaming and columnar is this is pretty bleeding-edge it is the site for streaming as I mentioned that rights are then put in and it's fairly easy to ingest data from spark streaming or cough card you can today you can use the kava connector in spark streaming but we're thinking of adding support for caltrans action directly and just depends on user feedback so and it keeps the data sorted that's being ingested so basically you decide how you want to sort your data and it will keep keep your data of relatively sorted so this is what sort of fills in this missing part in the very ground basically you can think of it as for example sport streaming rights it's like an extra layer on top of Cassandra so you can write it this way and the idea is you can still for certain things you'll still want to use Cassandra such as I have you know 100 millisecond key values that I want to write to Cassandra so you can still use Cassandra for that but but now you can actually write your raw events or semi aggregate events to follow DB and now you get very rich analytics no very fast just like you would with Parque except we don't need to set up an entire Hadoop stack you can actually use the same stack for everything which is weird is pretty cool sorry right so this is OLTP and this is Ola yeah that's but it's the same stack which is which is pretty cool so for example let's take an event time-series example what what we do is that we model our data using two concepts one is the partition key which you're familiar with if you use Cassandra already which distributes your data around the cluster and the second concept is what we call a short key or this is equivalent to Cassandra cluster and key basically within it all your data that we lost one partner and key fits on one node so the short key is how the data what then a single partition is sorted so for a time series what you might do is for example the partition key might be your events or your machine entity right that might be like one device or ten devices or something like that and your short key might be timestamp and so you would have your data data this way where the the data for your device is kind of laid out over over time it's pretty standard time series stuff the only difference that instead of a normal kiss under layout pile of DB will keep it data written and a very efficient calmer format that allows you to read it out very fast okay there's another option which is experimental but you can also keep all your stuff in memory so let's say that I have a smart streaming this D is coming in in successive rdd's I can actually have this data adjusted into an in-memory columnstore and you might ask how does this compare to like rdd's right so rdd's are immutable meaning the data and there is static right there's there's two big differences which is one is that with an in-memory file DB you can ingest new data in there so it's not like rdd's so you can keep ingesting new data of course at some point you need to get rid of old data but this could keep ingesting data and keep it sorted and then the other thing is that it can filter data in multiple ways so all these are basically meant for food table scans you need to go over all your data but this would allow you to filter in multiple ways such as you can filter along your partition key or a long time stamp or things like that here's an example of connecting spark streaming to Philo TV we have a stream of data a rating stream which is pulling from from kafka using the direct stream connector and we're going to so what this and then what we do is we do it for each RDD which basically as each chunk of data comes in spark screaming is going to give it to you as an RDD and we're gonna take that RDD which in this case is a bunch of strings and we're going to extract out stuff from the strings and basically convert it into a data frame we're gonna convert it three columns a user ID while a source and destination user ID and a rating this is this is this dataset is actually based on kind of you can think of it as kind of like a like a like a dating website or something right where you have like one user likes another gives gives him or her rating something and and this is the key part where I take my data frame and I do have rights and and this one is pretty much identical to how you would write it to Cassandra or to any other data source except here I'm using a different format called file DB dots Park and I decide what they decide to write it to which they days that it's a table so this is pretty much a one-line change from you know how you write the Cassandra to writing in to follow DB it's pretty much you know pretty much identical semantics so I said that would be a slider on versioning so I think versioning this is why it matters databases pretty much have been the same since the 1970s you can think of databases globally mutable state right so in your Java programs you have like mutable state that you share through you know Java util concurrent the database is pretty much you know it issued a version of that right and the idea hasn't really changed since the 70s and now this this is sort of a chance to rethink database a little bit and think well instead of this mutating state I can have explicit version so if I have new data I can write into a new version and I can isolate people from that from the new changes so let's say there have some new change it doesn't propagate all the way I can keep data reading I can keep people reading on an existing version of data and control exactly when that new version gets exposed so so I think versioning like or at least explicit versioning gives us a way to think about databases you know a bit differently and enable a lot of streaming applications question so so what's the what you said it's like in Cassandra if you write old records and you and you overwrite them you have to read over that actually that's not true if it's what depends right so if you write a new record on top of the old record at the same primary key you will sooner or later get the new record only but you might see the old one if wherever you're reading from has not caught up which is the beauty of or dangers of eventual consistency right you're probably thinking of X base X base will let you read multiple versions or you can just version stuff yourself so you can have a primary key where you add in a timestamp and then you so you can explicitly control version of yourself you can do it that way but but Cassandra actually usually only returns you the latest data yeah you can think of them as as videos there's different ways to think of it like yeah you can think of it as videos like basically I can put changes into a version I can implement and BCC that way if I really want or I can think of it as like let's say time series I can think of versions as units of time like I can have each day or each hour of data be a new version right so there's different ways to think of it well it's off by default in the sense that like by default we always write to you and read from version zero so it's it's really just up to you how you want to use it well if you write to a new version what you can or what you eventually we will do is to be able to read data and have it you know merge across versions which which is the way it's meant to be used I'll right now that's not there yet so right now you can use versioning as a separate filter like like let's say that I decide that I'm version my data by our right or by day then then I can use that as a really quick way to to filter and limit stuff right now that's how it's used yeah I think that's that's definitely to go so to to go eventually is to like you know let's say you I only want to keep the most recent 100 versions and then that gives you a way to like deprecated older versions and but but that's not done yet but the more folks say hey that's really cool and you know we want that then you know the faster that happens oh I'll tell you like the versioning came from originally came from because I worked for a company that hosted a lot of public datasets and when your datasets are like sensitive things such as how much politicians are paid you know or things like that you don't want to just push changes out like a regular database you really want to have changes and and let people look at it right so so the review process is very important and the other the other use case was came out of doing geo geo stuff like I might have in public datasets it's very common to have location column like for example San Francisco has a 3-1 one dataset like all the 3-1 or incidents and 9-1-1 incidents that happened and so you get a latitude and longitude but you might want to annotate that right like you might want to find out what voting districts these things belong to or what zip code and so there's there are several ways to do that but none of them aren't that pretty so you can think of computer columns there's one way to do it and sometimes these things change right like say sip cold boundaries or political district boundaries change so versioning again becomes very important so so there's a couple and the way that we were doing it before something like this was that we would use a regular DPM our DBMS and you would have a table right and versioning meant you have to copy the entire table and write things over and you can see what happens when you start getting millions of records right that doesn't really scale right so those are some of the original use cases behind it you can think of versioning as enabling streaming too so let's say I have different inputs and I just arbitrarily segment of different versions that means they result in different versions in file ODB and I can stream them out and croix them separately this thusly sparks equal but I think I really meant for that to be struck streaming or I'm not quite sure I remember what I was thinking so a little bit about how some of you asked about how do we store data so there's multiple ways to accelerate queries in probably be the first one that I talked about was columnar projection which is when I do analytical queries I'm only going to read out the columns that I need that that saves me i/o the second way is filtering by partition key again let's say that I only want to read data on a few like you know entities so my LT or something like that right a third way is by doing sort key filtering let's say that timestamp is my store key so again like I might not be interested in like data for all time in fact most likely I'm only interested in the most recent time period so why read all your data right and finally versioning is last one and what I would say is like you could do you could kind of do this with park' and one of these dimensions like you can organize park' files into a tree structure but you can't do it in multiple dimensions easily right and we're going to go even further with this concept little bit later by doing vertical style projections where you have multiple ways of sorting data in a database and to make that very easy calm the format how does so again how does columnar solve the i/o problem let's say that one of the questions is how much data connector interactively and with columnar it's a lot more than you think because maybe I have hundred Meg's of data and you think you know that's kind of too big to query interactively meaning at web speeds but if I only pray like a few of the columns maybe that comes down to ten megabytes and because I store things in a common way get that gets stored in a in a compressed manner like strings we can use things like dictionary compression and with integers we can store them more compactly so you end up being able to interactively query data that's three megabytes that's a lot more doable right you got to get the slides there but I have an interesting project called Cassandra G tau where I took the G tau public data set how many folks have heard of P tau G tau is the global data base of events language and tone basically it's the worse every single political event that has ever happened since 1977 and rates it on a scale from zero to ten and how much it's mentioned media this kind of thing so it's really really interesting data set I took this data and I wrote it in several different layouts one was a regular Cassandra what I call narrow table a narrow table means then you have a part of a a primary key that is also the partition key and no clustering key I also did a white table layout where I kind of hashed and divided the dataset into n record and n sets and well this out and what I found which was pretty astonishing and one of the motivations behind doing this project was that a columnar layout actually had gave you huge boosts in both read and write speeds so just very very interesting some of you asked about the schema so this is the way that we store the binary data blobs in file ODB we have a partition inversion which is the partition key and what happens is that this is the clustering key we basically what we do is we first we group the data by by the column name first so like all of the you know first names or store together all the last names stored together and so forth that that that alone allows you to read out just the data that you want and minimize io right the second thing is that we have what is called a segment so basically if you think of your data that is sorted from A through Z or whatever we then subdivided into segments and so the data and that's how we keep data sorted as it is being ingested that they keep going into the same or into the same segments and finally they're written in and success chunks using successive flushes using trucks so if you're curious that's how we store data Cassandra yeah so it's it's yeah so so we encode data in a way that we can still sort it using using a plow usual yeah well partition is your partition key so I mean let's say for time series it's your device ID for example would be your partition and so segment ID would be deep subdividing and timestamp so let's say that mint is in my case let's say it's a one-minute right so this would be you know plus 10 minutes Placido min minutes plus 12 minutes you know if successive chunks of data right when you say row right each physical row and Cassandra would be a device ID not the logical Cassandra row which can seek you out which is something else yeah yeah the times that would be perfect candidate for a segment idea yeah no no actually so there's a column name here so we create a Cassandra table for every logical table that you have yeah or at least one of these tables the GDL let's see we actually this is how we actually named it that's it's the name of the table and we append something else yes well it's so that you can it's alright you can pick out different versions right yeah they go together so this is a composite a composite partition key in Cassandra yeah yeah oh yeah yeah so you can filter by version very easily yeah right now you would have to iterate over all the keys so so right now we don't store that but we might in the future yeah so yeah did I trip something okay cool now so the table is for logical table all the columns in your table or store it by the column name yeah oh the chunk ideas is it's for differentiating your successive flushes so basically you know your date the data is kept in a mem table and then it gets flushed out in successive iterations right so let's say the first in the first flush your Chung idea would be zero you know and the second one would be one and so forth so so it's it's so that we it's sort of the right path it's mostly append-only and every time you flush know we flush new chunks of data that's all that really means and it's a way of enabling us to keep data sorted but still keep interesting new data this is great this is a lot of a lot of questions about the Cassander layout I just have one slide on basically we store data in a way that minimizes caching overhead and that's one future way of getting very high performance which is that the data format in cassandra is stored in a way that such that we do have to do very minimal work when you load it into memory it's the same format when we scan through data this is this is not this not the same as its well by comparison when you read normal data from Cassandra it's stored in s table format then when it's read over the wire through tcp/ip that's a different format then when the stored in Cassandra is stored in Cassandra RDD objects which is kind of another translation so every step of the way there's kind of like a separate translation right and you're something else so the whereas file ODB is designed from the bottom up the storage format and what's scanned in memory is the same so kind of minimizes that overhead this is really hard to see but basically the architecture like there's a column store API and when new data comes in it comes in through a coordinator and it's kept in a mem table then it is gone done through a projector which basically we formats it into a columnar format in different sort orders and writes it into comb store there's the concert API is generic so we can implement new backends for it and currently there is a Cassandra and an in-memory back-end but you know if you feel like oh I really like credits or something like that and there's always a few folks that like Redis or whatever your favorite stores you know hey feel free to submit a PR and add your favorite column store I'll go over this a little bit later basically each dataset is actually stored using two regular cassandra tables because in addition to the binary blobs there's another table that allows us to index stuff or keep it basic Aliki track of sort order and overwriting rows and things like that so just a quick notes about how like future plans one is that the phyto repo is actually the the binary format itself it's designed for design for very fast like minimal or no d serialization and it's designed for on and off heap weeding of binary vectors it's very very fast it's basically the ceiling of this library I have benchmarked it you know for reading integers at you know a building this minute or a second so this is pretty much the fastest you can do under JVM without resorting to GPU and things like that or SIMD which yes that's right well it's basically just a design trade off like most libraries like most formats like parquet is optimized for space so it will do all the tricks like Auto compression necessary to get in this field space as possible I guess I have a different philosophy I mean space is important but I believe that RAM is getting cheap and I optimize for cpu speed right so basically instead of using every compression thing under the Sun for example like run nothing coding is very common right so you might use one death encoding to minimize the space that's something uses but instead of running thin coating what we do is that the philosophy of Philo was to split up data into smaller chunks and so each chunk you can optimize separately and and just say you know instead of rolling coding I'm going to store stuff using a few bits but it's a few but it's consistent so that way I can do random access and you know so I can do like binary search and that kind of thing you know without having to deserialize everything so it's just a different trade-off yeah different design trade-off and at the other advantage of this is that again I can use techniques like memory mapping files from disk to memory and I keep it off heap and it can do like almost no translation which again like every step of translation I think that's one reason why parque is fast but it could be much much faster and one of the reasons is because it has to do a lot of spend a lot of CPU to deserialize things and you know that kind of stuff so yeah this can be very very fast by the way if you if you like Scala by the way how many folks new scholar here is this curious also this is not a standard for loop in Scala for i0 until number is optimized this is a macro library called skål XE Scala XY that basically allows you to use for loops but give you native well give you wild new performance basically as fast as you can do on a JVM but in a nice-looking way so spark has a project tungsten which is coming which will speed a lot of things up by passing data off heap and one thing that we're planning to explore is to use an actual columnar query engine and spark today's spark goes row by row what we're going to look into like columnar primitives which will get you even faster performance because one of the things that slow us down is that we have to translate things in the Rose today you got smart to process it well I mean Thompson is something that will just benefit all apps so so yeah it will be compatible but look basically anybody that runs in spark is compatible tungsten if you know what I mean like you just have to support the latest spark versions but do we take advantage a lot of it we probably don't take advantage enough of it right now so we'll be looking more into that another thing is that we'll be looking into caching and tachyon how many folks know about tachyon because so for those that don't know tachyon is a project - it's basically a caching mechanism that it's associated with spark because it's developed by the guys at UC Berkeley also the idea is it keeps your data off heap but it also keeps it it allows you to share multiple JVMs it allows you to share data across multiple processes so it's like an off heap off process cache and some people use it to cache files like park' files what we want to look into is tacking on as a feature for tables to store in neutral columns so we want to take advantage of that to keep hot columns and share them in memory the reason why we want to do that is because right now if I cache data and spark in the spark workers what happens when a spark application dies does anybody know and you lose all your data on your workers right so it's not very itchy right now tachyon is more actually in a sense that even if the spark workers or application dies the Tachyon nodes still exists so you won't lose your data and finally like on the roadmap yeah to support different ingestion api's support more data types support projections how many folks know about Vertica right if you've used verda card and you know that it has different kind of projections and that's one way of speeding up queries right and it and using GPU and SIMD instructions the sort of thing yep yeah right now we support kind of the different integer types you know you know doubles you know strings let me see we support most of the basic SPARC data frame types yes so today we don't support those yeah but but we will look into adding that it's for not so much like well this for having nested data structures and and that kind of thing that you typically find so yeah would be very interested in you use cases and data models so that that would help us a lot and just want to end what they quote that my favorite African proverb if you want to go fast go alone and if you want to go far go together so this is definitely an effort that we wanted to go very far and we'd like the community involvement with which is why I like this is we're kind of opening up an earlier stage to get community feedback so they'll definitely like you know your comments and suggestions and use cases you know and that would help help guide our priorities and things like that nope thank you very much oh I have a quick demo I'm going to try to get this going but you know demos never quite work so we'll see can you guys see that okay so this is the spark notebook how many folks have used broken-up Walker yeah so this is if you don't know about sparking a book this is a this yeah so this might just not cooperate because of what the Mac likes to do yeah spark notebook is a really neat project that let me do something really quick basically spark notebook allows you to get a notebook like environment running on smart you can keep you know especially like you can keep your previous results you can run new results and graph stuff it's it's like it's really neat right okay and okay so and you can open up notebooks okay this will all right well anyway I was this opened up for questions while like I'm not really sure if this will work or not like I said so and we got it working I'll show you some stuff okay anyway something is wrong but I'm not going to debug it now the expected hostname is there Wi-Fi here by the way what's okay so nerd so I can't really read that can someone spill it yep okay okay thank you thank you all right sometimes you know spark like those funny things in a network all right anyone found out like sometimes it doesn't work unless you have networking like networking but feel free to ask questions while like try to get the 7 running no colonel okay so who's behind the project so so so but this is supported by a company to pojo it was a person or toy for a while but no there's there's actually people around oh yeah so I think this is actually running and so what we do is that can sit folksy or should I make it bigger let me make it a little bit bigger so basically we started with a spark isequal context and here we read in a data set so this is very similar to reading any other data set into spark dataframes basically I'm doing a see what contacts that we'd format and it give it an option and I do a load that's fairly standard of use data frames the treat up data set that I have is like 4 million rows times 60 columns is about I don't know one two gigs oops and whenever you run a sequel commands you have to register a temp table so after this I can do I can basically run a so this is the data frame DSL I can do a select count to spark DSL and this is pretty much this is pretty instantaneous this says it takes you know like half a second or so I won't do to run the cassandra' one for you but if we were to do the same thing in Cassandra will take about like say 200 seconds you know sorry oh no this is very dumb this is just this is discounting the number of rows in in like that or not now in this particular column which is this month year which happens to be the month in a year it's so that doesn't do anything but let's do something a little more interesting right so let's say that I want to find out the top political events in this data set and and that's why doctor one name and I want it so I'm doing a selects you know that an account and I'm grouping by and and doing an order by account right so that this is a bit more interesting so I'm going to do that you know it returns to help results I'm not sure why the top one is blank but I guess there's a lot of events that don't have a description but the US and Russia are some of the top events that show up let's say that I want you know so the table is nice but let's say I want to graph this right so this I'm just doing the same query but instead of just as plain to result I'm gonna actually do a bunch of transformations in Scala to get into a format that the notebook can understand and now if this if my the Mac will cooperate with me yay so now I can actually get a nice chart right which is so this is very fast and interactive and so I can even do machine learning basically I can this is using the ML Lib statistics library I don't know how many of you know but ml dub actually has the statistics library so let's say that I want to a correlation between two columns one is the number of mentions and in the and the second one is the number of articles and they both have similar meanings I forgot what the exact meaning is but basically this is something that you would expect to have a very high correlation and basically I'm gonna run this what this is doing is first I'm gonna do the I'm gonna select I'm gonna select one column out of data frame then I'm gonna map it into an RDD of double because the statistics library will works on rdd's and not data frames I suspect that they're going to rectify this soon then I'm gonna run a peer in correlation and this has already come out there's ninety-eight percent as we suspected it's fairly high but this shows the flexibility of something like father-to-be when you connected to spark that I can run the machine learning library and do you know all kinds of interesting things and this comes right out of box cool so yeah thanks for coming and we bite well I'll just mention again like we're open to you know helping folks with development projects and big data and be very happy to say questions about you know father would be other things what is the usage overhead of filing to be compared to standard Cassandra haven't really profiled it I think on the read side it compared to standard Cassandra and SPARC it's fairly similar my memory might be a little higher but definitely the CPU and resource is a lot lower because it runs a lot faster on the ingestion it's going to use up more memory than Cassandra just because in order to convert to corner chunks we need more memory to have basically the idea the more memory you can use in the mem table the more efficient the writing will be because it will write in bigger chunks so it's kind of a trade-off what's kind of compression libraries do we support was that a question okay so if I ODB itself doesn't do compression on the data however we use Cassandra itself has a lot of compression options so the standard one is lz4 and it can also do snappy and I think a few other ones so what I've seen on the GW set is using the built in LZ for compression it will compress even though the common trunks are already very efficient compared to the native Cassandra storage format the lz4 gives it an additional about like it compresses the data down to a third of what it would be is this what I see so yeah so it's definitely I think it's definitely worth it yeah yes it's both so the data is versioned because the data is actually written to different physical versions but the schema is version as well in the sense that let's say that I have columns ABCD I can always add new columns to my current version but I can't change for example like if I want to change the type of a column which you typically don't do in Cassandra or Yoshiko anyway but let's say that let's say for some reason I wanted to change the version of column 8 from in string or something so you can't do that on on the current version but I can do that mixed version and then that allows me to separate out the data and I can do my ETL from version one to version two so it's both and deleting a version is actually just marking a version at marking a sorry deleting a column is the same as marking a column as deleted effective version end so if I was to query version n minus one then I would still see it a column and if I create version end and that column disappears so that's so so yeah so as well I would I mean pretty much I you know I would run said of Cassandra and AWS and smart and Coleco locate them there are different ways to do that one of the neatest new ways of doing it that I've seen is that mesosphere has ad costs that you might have heard about it's called the data center operating system and with that you can pretty much it will you give it your credit it works credentials and you can do ad cost package install spark and Cassandra then it will set things up for you but yeah there's that that's so that I mean that's not specific to follow DB but you know there's different ways of doing that too like you can you connects what you mean to have done for example and so yeah sure a little bit about akka I know you want to say a few words so basically akka is an actor framework I don't even heard of Erlang right but you know the telecoms invented this thing new language back in the 70s what akka let's see what's the best way to describe actors it's basically like people talking right instead of the idea behind actor is instead of sharing let's see have one box and multiple threads you share a beautiful state that's kind of like traditional Java application architecture actors are very different actors this basically every entity is like a separate person and the only way you talk to each other is by sending messages you don't share state and because you should and everything is so one thing is you share messages and another thing is that people watch out for each other so if if there's a Fault in one actor then somebody else supervises it and can restart it and can responsible for it so that pattern allows you to easily distribute applications it's just like having a distributed organization your people talking to each other but instead of you know seeing each other in person I can still exchange messages through email right or this lack or whatever it is so actor based programming gives you a really easy way to distribute applications and in a case of streaming I would say actors are a great way to do we don't have an event processing with very low latency because you can easily exchange like millions of messages a second and you can operate one message at a time so that's where it fits in like one way you might use it as I can use it to read data and feed it to Kafka and I've used in their past to process messages out of Kafka and as like independent processes and for two other things all right well thanks very much I'll be available afterwards for questions you know and stuff yep you