sfspark.org: Russell Spitzer Interview
Recording: sfspark.org: Russell Spitzer Interview
[Music] hello everybody I'm Alexa crabber the organizer of SF spark and SF Hadoop and here were at uber engineering for the very first time we had this first joined meetup of fascist park and SF Hadoop and we also have Cassandra SF Cassandra meetup which actually initiated this great gathering and we have a loss Ross Spitzer who talked about spark asunder connector welcome Russ so I think sparta super hot cassandra is very often used in this smack stack combination which we discussed earlier and i just want to find out you know how like how how did you come to do this spark connector and and what role plays in your life right like what's are the most interesting things about it yeah so i have been working at datastax for about four years now and we all kind of got introduced to spark around spark 09 which was the dark ages of spark one of my co-workers the team lead on the datastax analytics teams piata he he basically turned out a spark Cassandra connector prototype in about two weeks and he brought it to all of us and we were like Wade all of our hive integration of our Hadoop integration all of our pig integration looks terrible compared to this this is so much easier to explain so much easier to work with and it started solving a ton of problems that we had in in Cassandra you know at the time one at one of the issues is how do I move a whole lot of data from one Cassandra table to another I have different different partition keys that i changed my schema I need to do some big change and all of a sudden we had this really easy syntax for moving a whole lot of data around and having all of the hard stuff like splitting up your data battering together stuff on writing all of that happening efficiently and automatically and with this data locality I mean it was great it's a great experience so uh I think this is the first time we have Cassandra in Kansas passive spark meet up right on so generally like Vodafone familiar with Cassandra so obviously one of the questions with the supercilious data locality alright so far basically now like like multiple beasts it distributed in there all the same machine so you have basically spark sitting on the same cluster Cassandra sitting on the same classes all talking to each other using their own protocols can you kind of briefly outline how do they interact and do they take advantage of the fact that some data in some RDG or partition is calculated with some know that Cassandra is supposed to have this an item yeah so that was one of the key things that we liked about spark when we were looking at it originally is that you're allowed to specify with every section of data that gets read out of some system exactly where you think it is and because of that you can give priority to certain machines for reading that section of data so basically we take the token range that has all the Cassandra data and we ask Cassandra where are these tokens owned what ranges exist on what machines and then we tie that together with the spark executors IDs and try to match them up so when you actually say i'd like to read this whole table out of cassandra into my spark rdd it's going to try its best make sure that it reads the portions of data that exists on the same machine that spark is rerunning on this is of course just one way of doing things you know if you want to run it on a completely separate cluster it'll still work we actually use the Cassandra Java driver underneath to actually communicate all the data back and forth so all those same benefits you have with query routing and all that kind of stuff comes into play you just don't get the data locality if you're not running in the same cluster mm-hmm super cool super cool so in terms of divorce right so basically have these big clusters how did people typically manage you know both Cassandra and spark in practice or do you see um that's a bit of a hard one I mean so there are three main resource managers for spark right now there's yarn mace us and the standalone manager and I think I I didn't check the latest data brick survey but i'm pretty sure standalone is still the most popular of those three and with the standalone cluster manager well with each of those resource managers you basically have a different Avenue of monitoring you have to add monitor in a different way within our product within data sex Enterprise we have that all synced up with our op Center client which is our universe monitoring tool okay if you work with something else you basically have to hook in your own stuff but spark comes built built in with hooks for Agrafena and well basically any kind of code Hal metrics accepting sink that you got yes so so basically people do a whole variety of things our customers are using mainly using ops ops center and they stand alone spark manager contexts that it automatically makes its own website when you start it up so people use that but outside of that I think a lot of people are hooking into the various metric systems of Griffin and things like that do you see a lot of people using mesosphere because I've seen like it basically you can launch cassandra and spark on it yeah I mean so I don't get to interact with those people as much as I'd like to but there are a lot of mesas users out there mm-hm to be easier to work with them the Hadoop crowd because they tend to have a well I dupas java application we end up having lots of class past issues message tends to be a little bit cleaner and that regards but I don't know too much about what those people are using cool and it's I mean Cassandra is interest like we've seen examples in a start-up right using a kind of dating app and kind of speak time so kind of kind of briefly explain like what are the strong cases for Cassandra like what what among your use cases where do you see like even more succeed with it and we show these people use spark like what's the typical good strong connection so you know uh i guess i can start out most of our our base users are people who have some kind of big data problems something where they know their questions their question is asked their repeatedly very often but over a small set of their data at random times so basically the oltp dream right I've got a very easy query but I have to ask it a ton of times and I need it never to have a lowly high latency it always needs to respond quickly so we see a lot of things where you have a tracking of customer events things like that where you have tons and tons of incoming things and every once in a while you need to get like the last five right are you need to analyze all the information for just one customer mm-hmm where spark starts coming in is where we start having people say well I've been running my regular application for all this time I have this great customer facing thing but now internally I want to make some Disney's decisions based on all of my data in aggregate and that's where spark comes into play Cassandra doesn't actually have a lot of great tools for saying I'd like to do this bulk operation on everything that I've learned so far spark basically says well that's all I'm built for I'm built for looking at everything in conjunction doing these large shuffles being able to join all of your data together I mean it's one of the things that people come to spark fours like well I've got all this Cassandra data mm-hmm these two giant tables and I like to join them and you know doing a several terabytes not a problem for spark yes park and just do that yes so you start getting their and then the newest thing which is actually really interesting is a lot of people are bringing streaming into it so people will have some kind of messaging Buster message and Q and they want to have that go into Cassandra but also compute aggregates and have those aggregates stored and the same time in Cassandra so basically they'll have their incoming data go into this system compute a bunch of sub aggregates have those stored to Cassandra and then serve those sub aggregates out of Cassandra so they're reading of their aggregates is very very quick but the aggregate processing can be done with a more bulky machinery right all right just kind of get like another interesting take on love architecture because you can use mark and then loaded the Cassandra and then serve it from the center back yep yeah I mean isn't everyone's goal right everyone wants everything you want to be able to handle your high latency your low latency your high volume your low volume on the same system and have it all kind of behave the right way yes it's still very difficult but we're getting a lot closer I think now that we were I don't know five years ago so you mentioned oil to be case international education reminds me you know our common friend alan chan is doing Philo DB yeah which seems like it's seven both case I just wonder like what do you think of this oh I mean I love Philo to be and I of course love Evan it's a really cool idea it it's basically you know the idea of fitting using all the cool Cassandra elements that exist and fitting a slightly higher latency query system onto it where you basically are able to access huge amounts of data that have been kind of pre or ganized into a very quick to access fash cooked access format mm-hm so it's really cool idea it's really nice and I really hope we can get some more customers using it I only know a few people who are trying to use it in production right now but I hope that increases yeah we could ever know speak at scala by the bay and a from his i think current vantage point i think we'll see much more user should be so somebody looking forward to kind of that variation of spark Cassandra attraction ah cool so in watts what's next for the spark connector well oh man um what's next for the spark connector so I mean one of our big issues is that the spark project itself changes api's on what seems like a daily basis so we're almost always playing catch-up with the spark project but it's it's pretty clear right now that the future of spark is structured data sets yes the data set API you have a schema on everything that goes in and you have the ability to optimize and deal with things in memory that have been uncie realized it's really great so we're trying to look at the spark standard connector and a lot of our interfaces right now don't have direct translations into the data frames api so we're trying to figure out how we can make that work so so for example there's a there's something i'm really interested in which is basically a direct joint where you take an RDD of partition keys and you pull them all out of cassandra at once so inside of our parks and rec connector we have an API for this call join with the Sandra table mm-hmm unfortunately the current implement design of catalyst doesn't have that concept basically it would have to read the entire table into memory and then do a join within spark so we're trying to figure out ways to kind of get those kind of ideas to spark proper he's obviously that's not something that just applies to Cassandra right you can imagine tons of even our DBMS is where you would benefit from only taking the subset of items that you want out of the database yes rather than pulling the entire database in and then chopping looks like Park it right like the Parque like you're basically wants to to make spark data sets and play nicely with other distributed systems yeah so I mean parque has already got a lot of this stuff nicely baked into spark because within parque we can basically say within this file values between this and this exists here are some summary statistics yes the problem with with us and with other hash partition based stores like react and things like that is that we don't know exactly what values are there we only know what range of hashes of values are there so the mapping is indirect yes so we need a we needed a better interface that lets us express the spark like we do in the rd DS actually and tell it that there's an underlying partitioning here it's not monotonic on the values of the keys but it exists you can take advantage of that ok things like that you just saying interesting so yeah so that seems like a nice for like many players in the space junk elaborate and this makes I could have done the better for this I hope so alright well thank you very much for coming and it is a great talk and you know will surely will have it back you know and all these meetups alright you [Music]