Scale By The Bay 2019: Russell Spitzer, To Spark or Not to Spark

Scale By The Bay 2019: Russell Spitzer, To Spark or Not to Spark

Recording: Scale By The Bay 2019: Russell Spitzer, To Spark or Not to Spark

[Music] so hi everybody welcome to my talk I'm here to talk to you a little bit about Apache spark so battery spark is a leading technology in analytics machine learning it's faster and easier to write than dupe you've got machine learning at your fingertips and does seem like everybody's using it right now it even has an SQL compatible API and you don't have to learn functional programming although I know no one here would ever pass upon a chance to use functional programming so it can make all these awesome things possible and especially if you're working with the database like Cassandra or another distributed database it starts enabling operations that otherwise would be impossible for example you can start doing joins on data in a system which does not support joins but the question is is it the right system for you to use for your particular organization because there's cost associated with every technical choice mink and today I'd like to really analyze that and go over what spark can do what it can't do well and what other options might you other options you might have so I'm talking about this because I Spit software engineer have been working with spark and Cassandra for quite a while now I think it's about five years I've been working on Sparks and spark 100 came out and a Cassandra since Cassandra 300 came out and I work on one of the teams that works on the spark Cassandra connector which is the tool that allows you to take data inside of Cassandra and have it interrupt with all of your different spark analytics and things like that and I've learned a lot of things over this time and I've seen a lot of use cases that work really well for the two of these technologies together in a lot of use cases which don't work so well and I'd like to encourage people to use other technologies in those places so to start let's talk about what are the really big use cases for spark so in my mind spark has these three major use case classes there's analytics that's asking really big questions about your data it's asking what did every user do last year what are the patterns in my data over a year over a year for every single user and this is opposed to small questions questions like what's in this users shopping basket right now what did this user do yesterday at 5 o'clock so those are very small we call OLTP questions as opposed to the big questions which is what did he do all year last year which is more of a big data analytics sort of question then there's ETL that's taking data from one system and moving it to another that's a acronym for extract transform and load so for example taking your data from MySQL and moving it into Cassandra or taking your data from my from Sandra and moving into our key files somewhere and then finally a use case of streaming which is live data coming in and needing to be processed as it arrives and getting some kind of business insight from that data so data that comes in life for each of these use cases we'll go over the pros and cons and some alternatives so we'll start with talking about spark for SQL and analytics so why would we care about spark for SQL in the first place well lots of us use databases like Cassandra which do not support a lot of basic SQL semantics so for example as I said before Cassandra cannot do joins it's just not something that is intrinsically fast to do in the distributed system built like Cassandra so it's basically forbidden from the outset there's no way in the language to even do a request that joins between two tables but SPARC allows us to do that sort of operation without any added complexity basically once we have attached SPARC to our Cassandra data we can start doing joints although of course not at the same latency and response that we would expect from a normal Cassandra request in addition we can start doing all sorts of aggregates other sorts of things where we look over all of the data in the database at once which might require actually doing processing on several different machines because the data set is so large and in addition we might have business intelligence tools you might have something like tableau or even Excel which are able to connect to databases as long as they have a JDBC endpoint and Cassandra says it does not support NC SQL does not support a JDBC endpoint but spark and one of those for us and again of course any custom logic you might have so big questions or custom code maybe you have a Fortran library sitting in your basement and you need that to run on your Cassandra data well sparks a good way to get your data into a place where you can run your Fortran code on all of your Cassandra data at once so how can we use spark to do SQL and analytics well we have a lot of different options I'm going to start with what I think is the easiest option which is something that comes bundled with all spark installs which is called the spark thrift server I'm gonna do a quick shout-out that data stacks provides something called the always-on SQL server which is built off of this technology and basically is the same thing but in a aj environment and the thrift server basically provides a JDBC endpoint that's backed by spark so any sources that spark can talk to the spark thrift server can provide as a JDBC endpoint so basically if you connect your Cassandra data to the spark thrift server you now have a JDBC endpoint for all of your Cassandra data so it's a really useful tool if all you really need is an SQL API onto Cassandra data this works for of course all of the BI tools that I mentioned before now outside of that we have programmatic access to spark and that comes in a whole bunch of different api's and different flavors I've highlighted here the two main api's you probably are gonna care about our data frames and RTD if you are just picking up spark now you probably will never use an RTD and data frames is probably gonna be the most common thing it's available in a whole lot of different languages Python Scala Java R and within all of these different language bindings or the data frame API you actually look at the exact same performance so it's really excellent if you start out with SPARC you go into data frames in addition you can do interactive work and spark using shells they're shells available Scala and Python and R as well as notebook environments so you can actually test out code and run it in a wide notebook environment on all of the data that you can connect to spark so that's pretty cool so this code actually looks pretty interesting so I've got examples here of the data frames in our DD the data frame API is on the left there and what you can see is basically to do an operation and data frames you describe what you want the data to do now the top few lines are describing how to read the data basically saying that I'm reading data out of a Cassandra type source and then there's some specific options that our Cassandra specific a key space in the table name and then below that I describe the operation I want the data frame to do and in this case I just want it to abrogate a column called amount with a sum so basically just sum together all of the values in the amount column and right underneath that I've shown the SQL variant of that so if you do the top box or the box right below it they actually will do the same work under the hood and spark so it's really awesome that you can actually use a simple SQL API to do the same thing as you can do in a programmatic language and well you'll notice there is that in this API we don't actually describe what kind of objects were reading out of the database we don't describe how to actually do addition we don't describe how to shuffle the data together so that's arranged in the right places all we do is describe what we want and what we want is the sum of the column amount now on the other hand the RDD api is the first api that came out spark and it's a bit more low-level it's where you describe exactly how your objects are being manipulated and what operations are being done to them you'll notice that in order to get the amount column out of this cassandra table i actually have to say that it's an integer i have to use a getter i'm actually getting out a java integer in that function and in my reduce i actually have to explain how to add things together because this is raw code i basically have to say this is the implementation for taking two values and combining them so it allows you to have a much lower level operation but it also becomes unoptimized below the data frame api all the code that it will be generated by this under the hood by spark will be optimized and we'll probably not use any java objects you'll actually work on unci realized pipe data so it's much more efficient to use the data frame api but the RDD api still exists might see it sometimes so what are the benefits for using spark for this kind of work well both of those code samples I showed you before will automatically make use of distributed computation and distributed reading which means we don't actually have to define the fact that cassandra is a distributed database that exists with data on many different machines as its partitioned in this particular way and to get all the data you have to request it by token ranges there's all these implementation details that we just don't have to know spark is gonna automatically use the spark Cassandra connector in this particular case to know how that data is mapped out pull in the right chunks into the right machines and do all the work that we want if we ask for things like a reduce or we ask for things like an aggregation it's gonna automatically know these are the places where I need to shuffle the data from multiple machines into different places so I can do other operations so you don't actually have to think too much about how do you actually some data that exists on 20 different machines all you have to do is say I would like to sum the data that's in this column and it will be like oh yeah well the right way to do that is we'll take these two genes and we'll run through the sums on each of them and then we'll combine those sums and run through sums and combine and it'll figure that all out for you so you don't actually have to think about the implementation at that kind of distributed level in addition the state of frame API not the RTD I will take the expression that you wrote and try to optimize it into something that works a lot more efficiently as well as generally generating custom Java byte code that will hopefully do your operations much faster than you could ever write yourself and in addition it has this data source API which various vendors are able to implement their own connectors for so the reason that the SPARC Cassandra connector can be implementing me using this code in a single line is because of the data source API which just says if you provide a format like this like this string and all the implementation details any end-user can then use that source to connect to your particular database or queue or file system or whatever it is so it's got a lot of great benefits for interoperability and doing distributive work but there are some downsides to using spark now one of the main things I think that's downside in all use cases for spark is that this is a brand new application framework and a brand new technology that you have to introduce so that means that you actually have to get a lot of people up to speed on a rather large new framework that if you are managing in an in-house you also need to dedicate someone to managing all those resources a good way around that is using a cloud-based hosted spark platform something like data bricks is cloud or AWS is EMR or data product Google there's a lot of different services but you basically still need someone who knows how to set up that layer of connecting your programs to the cloud so you basically had on a bit of burden to actually get a lot of this up and running and the other problem is that all of these operations involve as code is optimized around distributed computing which means if you're doing something that does not require distributed computing can be done on a single machine this will almost always be slower than doing it on a single machine so SPARC is generally not the answer in that case so what other options do we have if we're not using spark to do big analytics questions so if we are working with lots of data I just listed there's a couple competitors to spark flink presto Hadoop all of these are different large big data frameworks that will let you read data from big distributed sources and do really large operations on them if you have any of these already in-house I would always suggest going with the technology that you already have expertise in rather than introducing a new technology just because it's fun although that it is fun so you might want to do that anyway but you know your mileage may vary there and if you're using Cassandra one of the things that you can think about and this is also applicable to a lot of other databases is sometimes you can remodel the way your database holds your data in such a way that your big questions are now small questions so for example you might take running aggregates as your data is being adjusted so that you can actually look up small sub aggregates rather than reloading the entire data set to answer the same question so for example in Cassandra you can do queries on a single Cassandra partition that involved aggregates so as long as I data for your aggregate exists in the same Kassandra partition you don't actually have to introduce a new framework to do an aggregate over it if I you have a small amount of data I can't help but but emphasize how important it is to use single machine libraries and single machine systems when dealing with small amounts of data it's almost always faster it's much easier to repeat and your overhead for working with it is much lower so things like our Python or even awk scripts I spent a lot of time during my PhD doing processing of data files there were a few hundred megabytes and it almost always makes more sense to write a quick hoc script to process a file than it does to spend the time to actually like think about how I would distribute this to a lot of different machines and all that 600 megabytes was a lot more more memory when I was in grad school than it is now so try to keep that in mind anyway so those options look a little bit like this this is just an example of doing an aggregate in Cassandra using a Cassandra aggregation function as long as my race points are within the same Cassandra partition I can do an aggregation over them there's an article here by Adam Drake which I really like called command line tools are 235 cluster which is a little bit of a tongue-in-cheek example but he basically shows that he can do a small I believe it's a Python script that does the same a Grenada file that's like several gigabytes large that takes a hadoop implementation like 10 minutes just to get started because you know there's a lot of overhead in setting up tasks and distributing data over the network and getting the data back in writing it to HDFS and back and forth so it definitely pays to think about how big is the task you're working with and what do you need to do with it and of course for SQL flink and press he'll also have SQL API so if you already have those in-house you don't need to switch to spark to get that API so consider those as well so now that we've got a bit of the pros and cons should we actually be using spark for SQL analytics for our particular use case with Cassandra or other databases I have a couple questions here and if you answer yes to a few of these that problem sparks gonna be helpful for your use case so for example do you have a large amount of data does your data span multiple machines and do you need to work with all of it to answer a particular question that's important to do you need a JDBC endpoint for BI this can be a really big selling point for a lot of users if you're providing your database as a service to a set of non-technical users you just want to you know connect to tableau and they're very good at to blow they're very good at Excel and you just need a way to expose your data to them is spark already in your stack if it's already in your stack you have big questions it's it's almost a no-brainer that you should be using it as well for working with Cassandra and using it with your other distributed systems and finally do you have multiple sources or sinks for your and M analysis are you combining data from our key files or log files with data from Cassandra with data from MySQL with data from Oracle because if you if you are doing that SPARC is extremely good at interoperating between different sources different types of data sources and data sinks which brings me to our next major use case which is using spark for ETL so ETL or extract transform load is basically the idea that we have data in a lot of different places and we would much rather would all be in one place because that's where we like to do our work so for example you might unfortunately be in the circumstance where you've got business data in Cassandra but you have records that someone else is processing and producing for you and our key files somewhere in HDFS and at the same time you've got a catalog in MySQL that you have to compare the results to and then maybe you need to actually publish this all into a Kafka queue at some point for some other service to consume the cool thing about SPARC is that spark provides an API that lets you connect all of these different things and that's as I mentioned before something called the data source API so this is where the spark Cassandra connector lists the set this is the tool that we work on at data stacks which connects Cassandra and spark but there are also data source readers for Park he files for CSV files for JSON files for MongoDB for MySQL if you can name it there is a connector out there for it that implements this API and the data source API unlike the RDD api is only available in data frames and you can use it in spark SQL or a Scala Java or Python inside of data frames and to show you how easy it is to switch between different sources or combine different sources I've got a little example here so we have two blocks of code here on the Left we have reading data out of Cassandra and writing to a park a file that's stored in s3 on the right we have reading from a CSV file and writing to Cassandra so we have two extremely different operations but if you'll notice both of them basically are the exact same code pattern we say read we say format and our format says what kind of source we're using we might pass some source specific options but then we say load and then to do the right we basically do right format we can do options if we have them and then save so basically to change this code from doing one from reading from a distributed database to writing to a s3 back distributed file to reading from a CSV in a India say if it is DCFS which is our local HDFS variant and to a Cassandra database really just can really just means changing the contents of these function calls we changed the format's around we changed the options around and everything works together we don't have to talk about the fact that while Cassandra's got data in a tabular format that uses specific Cassandra types and park' obviously doesn't use those exact same types uses slightly different types and it has a much different written format SPARC knows all of that for us and including and this also will all work in a distributed context all the readers will be distributed all the writers will be distributed so we didn't have to think about any of that all we have to think about is where we're taking data from and where we're putting it to so what are the benefits of using spark for ETL one is that it is extremely popular like I said there are basically connectors data sources for almost any database or streaming service or a file format that you want to work with so you can take all of those different sources of data and combine them together and do analytics or whatever you like to do on them and move them between each other without a minimal of minimal amount of code like I showed you the code is pretty simple and switching between sources really just means switching named connectors in addition like I said everything will be distributed and this been this works in with the optimization system that I talked about before where code will be generated for you that runs much more efficiently and with fault tolerance that you don't actually have to ever worry about so what are the downsides of using spark for this one of the the main downsides like I said before is introducing a new technology requires maintaining a new framework maintaining new clusters and things like that and the kind of hidden thing here is that we are restricted to spark types once we've moved the data into spark which means if you're using a Sandra you might have a Cassandra specific type like a UUID which when represented in spark has to be represented as a string because there is no corresponding internal spark type so you do have a little bit of loss of value when you take certain database or vendor specific types and move them in to spark and again it's usually not that efficient for non distributed sources so the readers and things for things like MySQL tend to not be as well built or as efficient as the readers for things like Sandra or MongoDB which are natively already distributed so reading from multiple clients to multiple servers is already kind of implicit so there are a lot of other options if you're trying to move data around from Cassandra good data stacks provides a tool called the bulk loader which requires no additional frameworks or code it's a single machine library that lets you pull data out and put data into Cassandra there's a copy command called cql SH copy and of course you can always write your own custom single machine application although I don't ever recommend doing this because I you basically are gonna do a lot of work that a lot of people have already spent time on here's just some quick examples of how you use those do you see bulk is just a command line tool in the CQ less each copy command is just a line of cql that you type into CQ SH so for ETL should you be using spark again we have a couple of questions to think about do you need to move data between really disparate sources or you're moving things between databases that don't necessarily have an easy connection point are you moving lots of data is the data already distributed and of course which leading into our last use cases is your data coming from the messaging queue because one of the cool things about spark is it as a great integration with streaming data so for our last main use case we're gonna talk about spark for streaming should you use it matte kiss so why do we care about streaming well we like to deal with data as it comes in now it's becoming less and less acceptable to deal with data only from yesterday you want to deal with the data exactly as it's coming in and respond to that data as quickly as you can so spark can be used to stream with two different api's they kind of mirror mirror the data frame in our DB API so structured streaming is the new API which mirrors the data frame API while the old streaming API is called D streams and again you'll probably won't see that as much anymore at the moment both of those are micro batch streaming streaming techniques which means they group up many events as they come in and then process them into small groups but both of them both api's have fault tolerance and checkpointing methods and optimizations built into them now I again would recommend using structured streaming it's the newer API it's much more efficient and it ties into the optimization system that already exists inside of data frames and one of the really cool things about it is that the API looks almost exactly like the data frame API so here I've just shown for comparison reading from a park a file and reading from a Kafka queue so the only real difference between these is we change read to read stream and the rest of it is the same the elements that come out of this are still data frames and you can actually still do any data frame code on them the only problem is that there are certain operations that you can doing a streaming context for example you can't just count every element in a streaming context because that would never return you have to count by a certain window or something like that so what are the benefits of using spark for streaming we have this great data frame easy API it's got the same support for data sources that the data frame API had and it's got lots of useful great functions and things built-in D streams are pretty good if you're looking for something that's extremely flexible it has more flexibility than the structured streaming API and continuous execution mode will probably be coming as a production-ready feature of spark soon and that's the idea of instead of using micro batches doing a per couple based event processing like flanker storm does again the downsides of this are there's no SQL API for the streaming side of things so there's no way to access this USB well it is micro batch based again you have to worry about setting up a new cluster and a new service that's hosting your streaming operations and of course you still have to care about checkpointing it back up so what other options do you have if you're using Kafka and you want to connect it to Cassandra we have a Kafka connector so confluent has an idea of connectors which allow you to very simple making data from your Kafka queue go into a database and we provide one for Cassandra and if you're looking for per event per tupple based streaming right now take a look at flink or apex or storm those are all streaming first frameworks that started out with a tuple based model so you can get that low latency if that's really important for you here's just a quick example of the code you have to write if you want to use the Kafka connector basically instead of writing any applications or things like that you just write a little configuration file that describes where the data is coming from and where it's going so you don't actually have to write new code or write a new program of course it's limited to just taking data from one place and writing it to another it doesn't actually do any transformations or things like that so should you use spark for streaming again we have a couple questions that I like to think about are we doing really complicated transformation something more than just taking data from Kafka and putting it into a database are we joining with other database sources are we moving the data and then combining it with data the Ceridian Cassandra or MySQL or something like that if so sparks a good good choice because of the data source API are we doing other business logic or are we already using spark somewhere else in the organization again I really think it pays to take advantage of the expertise that already exists in your organization so if people already are using spark and are good at it I would definitely leverage them and continue using spark so just to wrap up to spark or not to spark my general Pro my general cons to using spark are that it's a new technology which means a new costs in terms of monitoring maintenance and training and it's best only at distributed work so you always have to think about the fact that you're taking on a burden by adding a new technology to your stack but on the other hand it's got a lot of applicability to a lot of different news cases and it's great for working with big data it's really optimized for working with a lot of data at one time and it can work with data from a lot of different sources all at the same time with a minimal amount of code so my answer to the question should use spark is just yes because I think it's really fun and I think more people working on spark the better but of course your own mileage may vary and I think it takes it it really makes sense to think about your alternatives when you're about to add spark to your project so just as the take away from this if you want to learn more about spark and you want to learn more about Cassandra we have a website called data stacks Academy it's completely free and it's got tons of videos and courses on the spark and Cassandra integration as well as workshops and classes and all this stuff all for no dollars and thank you so much for listening and I would be glad to answer any questions you have about spark or Cassandra or their connection [Applause] yep you have to see this you're trying to bring some spark and you have sort of different levels of how committed you are to spark different approaches one is sort of an ad hoc approach where you practice in a drunken state like a WFC to be a bar cluster x1 another is where you'll see you have Apache Cassandra non-metastatic Enterprise and you're going to try to have like any other Linux data center in Cassandra if you wanted installs part of yourself by the one of those machines to then I guess the other option would be web perhaps using PSE we did spark integration and then you have the analytics data center we have that with Sparta fate is BSD so you may be compared so the question is about the pluses and minuses of the different ways you can deploy spark and cassandra together so I think one of the the key things to do in all of those situations is that if you have an OLTP application you have some active Cassandra client facing application it says isolate that data center from wherever you're going to be doing your analytics work because your analytics reads are going to be scanning the entire table at once which is going to take a lot of resources and you probably want your worst case scenario to be that data center becomes responsive unresponsive rather than your OLTP data center becoming unresponsive so I would always suggest separating that out unless you are well aware of your limits and you already have everything throttled out appropriately outside of that running spark on top of Cassandra nodes makes things a little more simple if you're using DSC because that's all managed together if you're not using DSC I probably would recommend running spark in a separate cluster because the main reason that we bundle it together is it simplifies management as long as you know you have someone else doing all the management for you like all the installation and the plugins and the configuration is done for you but if you're doing all the configuration yourself you probably are gonna benefit more from being able to scale both sides separately so if you're already taking the burden on of managing both components on your own I would run your spark probably on separate nodes than your Cassandra data and if you're using EMR I would basically just have a separate data set for doing your scans off of our data center for doing scans off of so there's benefits in terms of read performance but they really depend on it really depends on how your data is set up so one of the key things is that if you've got machines that have that are already doing that are not IO limited right if your machines are no longer IO limited then you can you'll be fine running or your if your machine your spark on the same nodes and you're gonna save on network transit as long as they're co-located but if you're not IO limited if in your fact CPU limited then you want your spark running on a different cluster because that's just additional CPU burden instead of additional IO burden you probably are not as sensitive about the IO between the network so again you really have to try it out big difference between running on EMR versus on Prem especially if you're cassandra's on Prem and EMR is far away have latency differences and all that kind of stuff so you probably need to try it out and see what works best for you so we have one time for one more no more we have to wrap it up thanks so much for coming up I'm always ready for talks or questions [Applause] [Music]