BDSBTB 2015: Evan Chan, FiloDB: A Reactive OLAP Database using Scala, Akka, Cassandra, and Spark
hey guys um so maybe we'll get started I I don't know if someone was supposed to introduce or whatever but it's like 11:10 so probably want to like keep the schedule like I don't know somewhat at least somewhat on track so how's everyone doing like good um so good morning everyone welcome to Oakland is everyone having a good time here yeah yeah I'm uh as you can tell I'm a I'm a I'm an oldland ace fan you know and I'm also a Raiders fan so I'm a big you know local person um I actually no longer that's right that's right um yeah I don't think we'll make any any enemies but you know I don't know if if a Giants fan for we to leave or whatever no I'm just kidding um no but it's it's really good to be here here um and uh let's see if we make sure my display is up and running ah there we go all right awesome so uh yeah we'll get we'll get started um and talk a bit about um you know really really interesting stuff that I've been working on to to bring um you know I think almost everyone here knows about um actually we'll just get a really quick show of hands like I think you guys everyone knows about spark who's who's um I guess I would say like um you know raise your hands if if you're just you know learning about spark um or and raise your hand if you if you're using it in production okay cool that's it's it's hard to tell like sometimes I see like the same hands shoot up you know um and what how about Cassandra how many of you guys are using Cassandra all right awesome so yeah this this talk would be great for you um I think that uh oh okay right anyway um so um just a little bit about myself I've been using um Cassandra and Spark for a very long time Cassandra since you know really early days the 0.6 I any of you guys remember that um and uh spark since uh .8 and I've been uh contributing to spark uh for a while um there's one file at the root of the spark repo that that was contributed by me that you you find which is pretty cool um but uh I also maintained the spark drop server and today we'll talk about olap on spark and Cassandra and some things you can do uh to um to get uh really good olab performance as well as uh some new and exciting open source projects um little bit about my sponsor um I'm I'm a part of uh Tuple jump most of you have probably not heard of them uh so uh what what we are is uh we are a u Big Data technology leader that specialized in spark and Cassandra we have a bunch of uh pretty interesting open source projects one of them is um is a is a spark and cander connector um another one is a um is a Lucine indexer for spark and another one is an htfs um uh plugin for spark basically you can run run that um so some interesting projects well um and and um I don't there there's some of you I don't know if in case you've attended the it's I gave some similar talks at kigner Summit last year uh this this talk is has a similar title but it's actually most of the content is actually very different so so here's a problem space um let's say that um you know I have a lot of data and I want to run a fast analytical queries on uh olab queries on structured data and what I mean by uh by by olab just to clear up that term because sometimes there's a lot of confusion is that I'm interested in analytical like say you know mathematical queries like let's say I want to find out what are the top um uh items in in this kind of category right or I want to run certain kind of um statistical things like what is the variance uh or a correlation between two things um and um and and in this problem space uh and let's say that I'm I'm really interested in flexibility like I don't I don't want to be limited to just a few like pre-aggregated uh items um I I really want to give people the power to run at hoc or at least have very flexible kinds of queries unstructured data um what's more I have data that is uh constantly coming and which is almost always true that um that I really care about um running queries over fresh data so um how do you solve this problem all right um and an example is um give an example from uh video analytics um let's say you have um you know many billions of new events every day and that every time let's say someone watches watches some video on consumer website you get some new data when you when when you are U playing a video uh when you fast forward or typically what would happen is that um every like sa minute or every end seconds uh through as you're watching this you get some feedback so so that people so that you can know like if someone skips ahead or if there's some ad that people don't like or you know some some something that you can figure out um and typically um Publishers uh want um analytics on the latest stuff and the sooner you can get it to them the better um now um what some products uh people have build in the space that you can have some dashboards and dashboards are fine but you can only get at really the top level um kinds of information like I find out uh you know what are the top countries but if I really but what people really want to do is to be able to drill down on specific uh anwers such as let's say one top countries but it's filtered by a certain kind of device type you know or Os right or I can want to drill in to a particular let's say for Android devices you know what are the top countries like as opposed to iOS right um so so the more insightful questions that you want to ask then the more flexible kind of C you want and and the problem is that um um as as you go down more and more it gets um the amount of preaggregation you would need to do to answer all these queries uh becomes like too big of a combination right and so some requirements that we might have for a solution is that we want uh something like this to be uh first of all it has to scale to a lot of data you're getting billions of events every day um and um so so we can simply throw an rdbms at it um and we want to be able to easily ingest and update new data so I'm not talking about tra traditional say you know olap cubes that are generated say once a day I mean that might that might work for certain kinds of queries but let's say that we wanted something fresher than that right and um want to support ex uh you know flexible queres and you know hopefully something open source um so why not use paret uh paret is um you know universally used it has excellent support in the industry um the only thing is that par is really um read optimized um it's it's a file format that um when you want to write a lot of data um one problem that you might have for example is that um you want your pipeline to be uh resilient that means that if you have an error you want to be able to retry and and typically that requires you to have an idempotent uh data store or or or the other pattern is that um you might end up writing um the same data twice uh and up pending to a file um then what you have to do later is that you end up having to run uh a d d duplication job which takes your original paret files and generates a new set that has a data D duplicated um and that works but you know that causes you know that's another step you have to manage and that causes a delay and when the data is available for processing right um it's also optimized for writing uh very large chunks very large and narrow chunks if you think if you think about the shape of the data that's coming in um P works very uh the thing is because H it's based on hdfs it has to generate very large blocks of data typical block size in ex is like 64 Megs so that means that I have to in order to fill 64 Megs I actually need a lot more memory than that just to um do the column the conversion and you know do a whole bunch of stuff that's necessary so it works really well when I have you know I'm writing a few files at a time um but let's say to have a very you know wide write pattern uh with um where I want to segregate data let's say for I um then then that you know pattern doesn't work as well and what I would say is that um what people would ideally like but don't have right now is that you would really like a database abstraction for uh writing data into into something that you can also read out very fast so something that is optimized but that um can later be read optimized um it turns out this problem has been you know as most of the problems and architectures that we talk about today you know they they've all a most of them have been solved before and this problem has been solved for many decades actually um with um MPP databases so-called uh like vertica and there there was a new article that you know uh that uh even Facebook is has now has the world's largest installation which is you know pretty interesting that considering the Facebook created Cassandra and a lot of other Technologies right and the typical architecture for MP databases is that you have a uh you have a writable store and a and a read optimized store and a way to convert data so what this allows you to do is it combines something that is easy to write data into and ID ident but um optimizes for read and a very famous paper by Michael scker out of MIT is the SE store paper that describes this in detail with later on share the slides you could click on the link um of course the problem with traditional MPV databases is that most of them are closed source and um you know cost a lot of money once you start to scale out into multiple nodes and with a lot of data although I think that some them are making overtures to have Community additions that and trying to expand the limits of how much data you could use for free um so so that brings us to Cassandra right um Cassandra wonderful database it's horizontally skiable has um very flexible data modeling like has a lot of support for different types of data uh the types sets um and it's also very easy to operate and um that's something I probably don't need to go into detail but it's a peer-to-peer architecture right that means that every note is the same which which is which is pretty awesome all right and so so that makes that makes it um that's one of the reasons why it's being so popular for ingesting streaming data and um other things because it can take it it can do very high-speed ingestion for all shapes of data whether it's narrow kind of data or is very wide kind of data um it can do both right and it also has a huge Community has um you know proven storage technology you know multicenter replication a lot of these things um the only thing is that um it it really supports um the native interface only supports simple queres like basically you can get um I mean this is getting better all the time but um you can pull out you know designed to pull out simple like you know key values and you can pull out a lot of them and you can pull out ranges but it's not designed for like you know massive scans right and that brings us to spark spark is the computation framework um every time I have this slide I always feel like you know increasingly I don't really need to talk about it because all of you guys are now very familiar with with spark and Martin just gave you know 30 minutes on what spark can do right um but um you know the fact that you can do everything in one platform you know right very appealing so what if we combine like both of these Tech Technologies right um you have an excellent storage Engine with you know an excellent the you know the computation engine of today that is super flexible right so so what I would say is that spark really provides the uh the missing you know fast and deep analytics piece um to Cassandra right it allows you to to get a lot of insights out of your storage and run all sorts of built-in algorithms so um have a look at how you can do how how you can how we can actually do this with spark and Cassandra um the first principle I'll bring out is that um compared to so in a traditional database you think about um you have everything one box you have the storage and you have a query layer an SQL query layer and it's all in one box um what you want to think about is separating the storage and query layers um and and this is actually I think a pretty powerful concept because then you can combine a best of breed storage solution like Cassandra what the best of breed uh computation platform like spark and you can take advantage of both right and the um and you can do interesting things uh such as uh let's say that I want to scale out uh my cray capability traditionally what you would need to do is you would need to replicate the data storage uh to multiple class and to to enable that sort of thing but especially with a memory becoming cheaper something that you can think about for example is uh let's say that I have multiple um spark clusters that can you know cach data from Cassandra then then I can scale out my query ability without necessarily doing you know a very expensive um um replicating storage and all the things that you need to do right so let's look at this concept in more deta spark is Cassandra's cache and you have data in Cassandra you want to bring it up into each spark worker to to Cache um so spark SQL is something that appeared with 1.0 it used to be called the shark project um that uh rayot Shin worked on um and they they decided uh to to rename it and bring it make it an integral part of spark um and then one 1.3 you know it's the feature is called Data frames um it has an in memory uh colum uh data store which is pretty interesting it stores data in a different format um than um your native rdds and has a huge number of connectors which is just growing every day right um it has two apis which are very interesting you can use you can still use SQL but it now has a um more um it is more of a scholar DSL which which uh makes it um basically it it it saves you from uh needing to compile a SQL and it's slightly more typ safe slightly um one thing that uh spark SQL um has or doesn't have depending on how you look at it is that it doesn't have a built-in um index into the data but the way that um the indexing works is that um the U there's a data source API that can read from different data sources and that has the capability to it has hooks um for doing um pruning and filtering so if uh the um essentially like the wear Clause is passed on to the engine so in the case of Cassandra for example um if you have a wear clause in in in your SQL then they get passed down to the data source and in that can take advantage of any secondary indexes you've defined in Cassandra which lets you limit the um the amount of data that you read and one really cool thing about spark SQL is that you can write um custom udfs in Scala this is actually very easy to define a function then you call it you know register UDF I don't actually remember the exact syntax but um it's very easy to use how many people here have used hi before um so so if you have you've had the experience of um working with hyp udfs or developing them then like this this is this is like a huge God sand right um and it integrates very well another thing that is extremely powerful is that uh spark SQL integrates very well with um the rest of spark so we can usually take the output of a a SQL computation and feed it into say MLB for doing modeling um and other things which uh it's it's very easy and integrated um so we'll look at uh really quickly at um uh getting connecting spark con is very easy you can get started uh with uh say you're using spark show um you can use the packages uh option which and give it a maven coordinate and in this case this is the um the spark C under connector and this will actually go and download all of the dependencies that it needs so this is super convenient um the only problem with this approach is that um it it you know it pulls it it has to pull a lot of dependencies when this the shell starts up um the other really important piece is is that you want to pass the you want to configure uh where your cassander host is um and that's something that you want to do like when you start um in starting in um the latest cender connectors you uh there is support for uh data frames and so this is what it it looks like uh first you well the SQL context this this is kind of code is something that you would have in your own app but if you're running spark showell you don't need a first line because the SQL context is already created for you um but um this is the one uh spark 1.4 syntax basically you do a SQL contact. read and and then you basically tell it what kind of data source to read from by default it might read from par or something um in this case we pass it the class name of the input source which is the spark. sq. Cassandra and the options are what tells the connector what uh where to read from the the keyspace table name and there's some other options um and then you do a load and um at that point it it doesn't load um the load doesn't mean that it's actually loading the um the data from disk uh the the load it's a little bit of a misnomer it just means that it's you know it's going to make sure that the table is there and figure out the schema that kind of thing so it should be a very fast operation and and and I I'll show you guys what this actually looks like later and there will be a demo at the end um the next line a register temp table this is something that you need to do only if you're running SQL um because that gives it a table name um you can actually do um using the Scala DSL you could um do SQL like things using just the DF object um and the cache table we'll go into a little bit later about why you might want to uh cache table um and there's an example of running uh running the SQL from from that table uh one really important thing to note is that um spark doesn't do any caching by default so um if if if I leave out the cach table line basically uh spark is always going to be reading from uh Cassandra every single time uh so um this is how uh table caching works and so and so the um what what the um what the what the uh Cassandra uh table cache is is it's a very high performance uh it stores all of your data in a colum format right and um the way it works is that so in cassander let's say you have these logical rows right and when the SMART cassander connector works it will read that into memory as an rdd uh of um of of cander row is it this okay with everyone so far okay now when what happens when you do a cache um when you do a cache table it actually doesn't cache it right away but it means that the n in the NY read um spark will then take what you read in as an rdd which are in the green R boxes the cassander rdd and it will take that and actually converted into a columnar format so instead of being row by row by row there would now be binary blobs in memory which are compressed and accessible like one column at a time and this uh tends to save a lot of memory compared to um the rdd uh but the the caching the difference is um so so again normally when you do a read it will read from dis every time so it looks like it's populating the entire Heap but that's not what it's going on it's actually an iterator and it pulls it from dis as it goes along um the the caching will that will save the entire rdd in memory but in a format that is um more read optimized yes uh okay so so the uh the question was is this way of storing or caching data native to spark SQL uh right as opposed to trying to do it yourself uh yes so it is native to spark SQL it is actually specific to spark SQL in data frames that if you do cache it it stores it in Obsession format and um it's different than regular caching of rdds which which is an important Point um oh and it's possible that it might disappear in the future but um that's not official uh question um well maybe we'll take that question offline I think the the quick answer the the question was well um can you get Cass um spark to read from local cassander node I think actually um by default um and you know we we'll take this later um if you have um Cass spk collocated um the um the connector is actually smart enough to figure out what and and try to read um local data it actually reads using token ranges so so it can try to figure out what is local but we'll yeah we can talk about that later um oops so cache tables are really really fast uh I did a test on my laptop and you'll be seeing this name a lot G out how many people heard of the gout data set okay nobody um since we'll talking about a lot I'll give it an introduction jout is a really neat uh data set it's the um but it's a public open data set that has a lot of basically history of all the major political events in in the world since 1970s um and so the the caching basically speeds It Up by about 1,000x you know it's it's a couple orders of magnitude and so it's pretty dramatic but it makes sense if you think about that um when you read it from disk you know there's a lot of work that happens cassander has to read it from s table it has to go through cql it has go through a network and it has to translate the data into u a spark native format the a cassander row um whereas the caching is is in memory and is already in a read optimized format right um and if you use caching you can actually get um almost interactive speeds on a huge amount of data I did um study on half of the gll data set which is about 120 million rows um and I don't remember how many gigs of that's probably you know 100 or 150 gigs of data on an eight8 node cluster and for for a really simple uh queries you can actually you know they actually do like one or two seconds which is really really impressive um one really quick guide to um to tuning the connector uh I'll share like this talk is not really about tuning but um the the cassandra. input split size is very important that basically tells um the connector how many rows to pull out for one spark partition um the reason why this important is that the number of partitions controls the the uh your parallelism in spark but you want to have at least one partition per node otherwise you're not getting a of parallelism you are out of spark but you don't want too much because if you have way too many partitions um it won't speed up the job but what it will do is it will flood cassander with requests and you will find that it will completely starve out any other cassander request it will make them super slow or timeout right uh because you're using aot connections so the first lesson is take advantage of caching if you can um but as soon as I say that I'm going to give you a whole bunch of reasons why you might not want to do caching as well um even though it's really fast um the among the reasons is is that um caching is not xay because the data only lives on in the Heap of the spark executors this will this will probably change in spark 1.5 by the way because there's a um there's an effort to move some this this off Heap um that doesn't really solve the xray problem but it might um it will get rid of the GC problem um and um even though it's it's really fast after you cach it it still takes a really long time to read the data from Cassandra um and you also need extra memory to do the conversion uh from uh a cassander row into the cache format um the other thing is that because the cache format is special it it is a special kind of rdd it doesn't really work well with new data so let's say that I have cached everything from the last 30 days but now I want to add some data into spark um and and there's something called a union in in spark you can load new data and Union a new rdd with an old rdd but this doesn't really work well with cache tables um it it kind of It kind of you can kind of make it work but then it it slows down so so it's it's not really a use case that you know they've really thought of right so you know can we do so the question is really can we do better um and and as a side note um if you don't have enough memory to fit a cache table Sparkle actually is smart enough to save it to dis and um this seems kind of redundant because now we're reading from dis again but it turns out that this is actually still uh much faster than reading directly from Cassandra um but even though it's safe to this uh partly saved dis it doesn't make it any more HJ because the uh the metadata for this t uh the cache table it still lives with one spark context so it only lives for a lifetime of that context does anyone know what I'm talking about when it say spark context yes mostly okay yeah just one spark application and as I said before doing an rdd do cache is not the same as caching a table using a spark SQL um because that saves in a special format um the rd. cache what it does is it takes each object in an rdd which is a cander row object in this case and it serializes using whatever serialization Define like such as uh Java serialization or Creo or something so it's a different format than the columnar one and it takes up more um takes up more space um a quick note about secondary indexing um secondary indexing uh does this work with in so uh depends on what you mean um I would say for most for running ad hoc queries not really because uh what what the caching does is that it's it caches the r the rdd that you load from disk if you pass a cassander a wear clause in your first query and that wear Clause restricts to say 10% of your data then 10% of of your data will end up being cash in memory which might not be what you want quick question IND I'm sorry can you say that okay the question is I thought that secondary index were performance reducing H interesting because I IND um I I'll just give a really quick answer and then we can talk about later or I might refer you to my my friend with some data Stacks um but yes I think you right that the secondary index mechanism requires um a lookup of every node um so it I think it really depends on the ratio of um the data that you want filtered um versus how many nodes you have do do you know what I mean like it it's like the cost of going to every Noe might still be very low compared to reading out um um just a very small chunk of data uh versus having to scan an entire table um a really quick mention about tachon how many people have heard of tachon uh awesome um maybe like a third right so tachon uh if you look at this diagram it's like a layer that sits between a spark and another data source and what uh whoops what uh t is is an in the memory cache uh is really originally designed for hdfs and it works best for files um but it keeps data off Heap and then it has a um a kind of demon so that multiple jvm processes can share that data which is uh really really neat so this tachon solves the XJ problem instead of keeping data in one spark tied to one spark context and um in a heap or off Heap it keeps the data you know off Heap but makes it such that other process can can share it so this is great how but can we take advantage of this right now um the short answer is not really um what what do you cash in tachon like if I go back to this diagram um the diagram is not quite right because uh if I read data out of Cassandra um using regular cql uh it will come back it's really designed for applications right cql so it'll come back as I don't know let's say your your case class uh of person with you know first name last name whatever now if I wanted to cach it in techon I would have to serialize it again into some binary format right um or um maybe there's some way to insert it between I think the way to really use tachon is you would want to read the raw data from Cassandra and use ton as a r cache uh Cassandra is his own row cache but if you um you know that's something that's not designed for spark right so if you had tachon um in front of Cassandra reading the raw data um then uh you can actually share that amongst multiple uh spark nodes but this is something that would require um a a a large amount of of work um so I love this quote from from from Linus that um good programmers worry about the DAT data structures so what I would challenge everyone to think about um today is are we um are we thinking holistically about data moding and caching and like how do we make the whole system work because sometimes it feels like um every every component designs a system by itself right um and like we're trying to fit a lot of things together that don't necessarily fit together and and no one's thought about like the whole picture right so so how can we do better than um either of these two architectures so um efficient Colum storage and and first I'll quickly um clear clear up what I mean by columnar um so how does Center store your SQL tables let's say that you had this cql uh table which has um a first name last name age an employe ID um a department and maybe we want to Key by department and employee ID and and the two-part primary key um I think everyone probably knows this if you know Cassandra but the two two- part means that the first part of the primary key is the partition key that determines on what node uh your record goes and the second part of the key is called a clustering key that um basically uh tells spark uh sorry sander to sort the records um by employee ID order so the way that um cassander 2.x uh actually stores your data is this um if I go back really quick um you see that the employee ID is um is the um clustering key so if I look at um this is the physical layout of a cazer table not The Logical layout and you'll see that the the partition key um is the uh department so that's like sales or engineering um and then all the other records are laid out um in a physical row but this is the interesting part is that um what cassander actually do is that um so every cell has a column name as well and it will actually prepend the uh your clustering key which in this case is the employee ID so let's say that record uh logical record one is employee ID 01 is Bob Jones and what Cassandra would do is that it will store Bob in in one column it'll store Jones in another column and the way that it this ambigu between those two is that it will it will uh append the the column name in this case first or last and an H so so this is the actual physical layout uh what does allows cander to do is that if I was to insert only um the last name for example let's say that I wanted to update Jones to I don't know Jones's or something right then then that allows Cassandra to um address only Jones and update only that um column and the other thing you notice is that um all of the uh items in record two comes after record one but that each row is stored contiguously meaning that all the fields appear in one cell after another and then you get the next row and you get one cell after another um if let's say that you wanted to run an little query and you wanted to analyze the age field only you let's say you want to get the average age or distribution of Ages um what cassander has to do is that it still has to read um you can say I'm going to read only the age cells um but because of the way that is laid on the dis essentially cander still has to read the entire physical Row in order to return a subset dated to you so basically a cassander is really a you know role-based LP oriented you know data store um you know if if you uh unless you know how to take advantage of it um and uh Michael Stonebreaker said U the traditional world-based data storage approach is you know is dead what uh I'll just go over this really quickly um but what the colum of storage um way of doing things is that um I would store items from my column together so uh for example uh let's say you have name and age again I would put the um ages from subsequent records uh together in one place and a name in another place what it might look like in Cassandra is that um this is one example right is that the the row key might actually be um the column name and say's name and then I would store and that way if I want to pull out all the ages together um I would U just pull out the ages and it would only need to do IO for age um and one thing that colum form brings you if you're familiar with paret is that it it it gives you these huge savings in terms of space you might know that uh part you know one of the reasons people like par is that it actually is uh compresses data down much smaller and a lot of like let's say we just dumped the text file um and one of the reasons why it's able to do that is because it's able to look at a set of let's say let's say all the ages together and it can figure out you know I don't only need to store like this in so many bits right because all the numbers are within this range so it can do a lot of smart kinds of compression by looking at um data um as a whole um so why why isn't uh if if columnar storage is so promising why isn't everyone doing this um I think the main reason is that most of our infrastructure is still roow based uh spark for example is is you know is roow based and um the uh colum format is you know there isn't anything that really works with um no SQL Stores um so I'll share share with you you guys can I'll I'll skip over you know starting run out of time so I'm going to skip over the um uh you know some of the stuff so you can find the links later but um I did a study of um of the of the um gal data set right and um with several different layouts and um with a uh with several different layouts with one layout with cql another layout with cql narrow and wide versus a column layout and and what we found was that um using a column layout um dramatically decreases both the ingest and um the um read speeds um and it also uses up less space about like uh 5 to 10x less space um the let me skip over um some really quickly so um to share with you um the um project I'm working on now um this project is called uh phb and what you can think of it is is it's it's a modern columnar data store built on Cassandra and Spark and um it is it is cumar in the paret sense that it uses a storage layout where it can minimize IO for olab queries right it is I'm proud to say that it's 100% reactive that it uses the um types saave stack it uses AKA Scala uh Futures the Phantom cassander Library which is a um a future-based uh Cassandra Library um and it uses Spark SQL as your query engine so so it's very easy to use um as well as data frames for input and output um how it compares to um Park a for example is that um it will give you comparable read performance like if I um look at it um on uh say a local host you'll see that you know what you see this in a sec that performs about the same um but it has a lower memory requirement because it's based on smaller block sizes which also makes it more suitable for um uh more like iot applications where um you have wider types of data um and as of right now it supports like much fewer types but that's something that hope to go out um and where it fits in is that um you would still want to use Cassandra for um key value lookups that you know pre-aggregation that kind of things but then you can use that existing infrastructure uh with Cassandra and Spark for doing uh ad hoc and more complex uh types of an queries and and this would allow you to simplify your um architecture from um let's say it's simp a typical Lambda architecture where I'm writing into two pipelines where incremental views might be going to something like Cassandra and the pre pre-computer views has an entire Hadoop stack as well allows you to simplify your architecture to U something like this where I am using spark streaming and writing the Cassandra and it can use it for both uh I mean if you compare this picture to this picture you can like just count a number of boxes right and see how many fewer boxes you have to deal with right that you're looking at using castander for both ump as well as olab um and you can do exactly one ingestion from kavka because of the ident potency that you can write values um again and again and it won't uh and it will replace the existing content um so I'm going to go uh really quickly into a um let me see into a um into a demo that uh that shows you basically uh let's see I hope you I think you guys can see this uh this all right so basically um I have on the top box I have a a FB connected to the Cassandra uh with a data frame and this is a four million records of the jetal data set HS it locally all 60 columns and I'm going to do a really simple um query which is just to count all the items in in month year right and and this will run in um about half a second right and if I do the same thing with uh with paret uh which is df2 which is off of a local parket file um it does a whole bunch of stuff it takes about one second you know kind of like the same ballpark right and in the bottom window I have a spark Cassandra connector which is loaded using um something um okay I don't quite I won't pull this up because um but you can find me if you're interested but basically this will pull up um this reads from regular Cassandra cql table and we're going to do the exact same query um and you'll find that this will take a couple minutes to run um yep and that's it thank you very much I don't know if we have time for questions or we'll just take it offline Okay so we'll take questions offline thanks