sfspark.org: Building Your First Spark & Cassandra Application w/ Russell Spitzer
Recording: sfspark.org: Building Your First Spark & Cassandra Application w/ Russell Spitzer
[Music] so hi everybody you are welcome to my very first and hopefully not last live attempt at showing people how to code so this may go a little poorly may go pretty well let's be optimistic about it in case you're wondering who I am I'm Russell Spitzer I work at data stacks and one of the engineers on the analytics team and that means I spend almost all of my time thinking about how to have Cassandra interact with all of the Big Data tools that are out there right now obviously everyone's biggest interest at the moment is SPARC so our big focus on the is on the SPARC Cassandra connector which is an open source piece of technology that you are welcome to check out at any time and we are hopefully still going to continue working your Internet nope there we are so it's an open source project you are welcome to check it out help contribute learn all about it and all of that but today I'd like to go over just a first application example and to do that we're gonna start out by going to that link that I had on the on the Meetup event and that's a data stack SPARC build examples so if you check this out before it's actually a pretty interesting repository because it has within it these two folders Java and Scala and inside of each of these folders are three subfolders each one with a different build tool so whichever build tool you like the best we already have a pre-built preset up build file for your application and even better than that if you go into the directory we have one for data stacks enterprise which is our commercial offering as well as one for open source software so to start out today I've just got my own little repo folder here and the very first thing we're going to do is just clone this directory clone this project into our own directory so in case you've never done that before basically what we're doing is taking the repository and all that's links from the website and bringing down to our local local system within that we're going to then just pick out the build file the build projects that were interested in because we're not going to actually be using all of the various combinations at once we're just going to use one so let's just grab the SBT an open source example and I'll go over exactly what's different between DSC and in this it's basically let's just grab this whole thing and copy it over to a new directory call it example alright so we basically have everything we need to start working on this project if we start up IntelliJ which is my preferred editor we can now import this so we've got a live demo and example and the reason you want to do this instead of trying to import the entire package is that the the full project has all of those different build examples and your build system will probably just get confused about which one you're actually trying to use so click on that use auto import make sure you've got 1.8 set and hit finish and you will begin the pretty short process of IntelliJ loading up everything in this SBT file and getting you set up with your own brand new little project alright so there we go we've got our nice little project here and while it's busy indexing because right now the very first thing that IntelliJ is gonna do is run our SBT file and compile our code while to doing that it's gonna pull down all of the different resources you can see down at the bottom there's this pot thing popping up with lots and lots of different dependencies basically it's going through our build file and grabbing everything so this is gonna be a little different let's refresh that there we go okay so this is the build file and while we're indexing all the time to go over what this means what do all these lines do so right up at the top we have some pretty self-explanatory things we have the name of our project what version were using what version of Scala we're using and then two little personal variables that we're going to use here this particular build example comes already set up with connector 1.60 and spark 12.6 or 16.2 and you'll see down here in our library dependencies we have four different things but they're all marked as provided so provided in this context means that when we build this jar we happen to build a fat jar or what they you know a jar that has all of the dependencies included within it all of these lines will be excluded the reason we want to do that is that when we're running our project we do need to have a spark core spark SQL the connector all need to be compiled against our code but they're going to be provided on the class path by various spark tools so there's no reason for us to include it within our own jar now all of this is actually going to be put together using the assembly plugin so we have a little file here in project called assembly SBT you could also do a plugins file but basically this just says that we're going to use the assembly tool which will if we you know uncomment it any of these lines when allow us to conclude other dependencies in with our application jar and have them include it in the fat jar so once this whole thing is ready we should be able to go straight to our application and everything should be highlighted correctly and we have this beautiful little little thing here okay so I'll go through this code in a second but before we do that let's talk about how we actually build this code so the first thing you're going to do is go to your your directory here you now have you know these two IntelliJ metadata things and we're gonna call SBT assembly so SVG assembly is going to call on that assembly plug-in to build us the fat jar for our application along with all the dependencies that aren't provided currently all of our dependencies are actually provided so this is actually not going to merge anything it'll just build our code and make a jar with it inside and you'll see now we've got this nice little right assembly 0 1 dot jar so some of you might have tried to just run this jar and unfortunately spark doesn't make it that easy we can't just run our jars and expect our class paths to be set up correctly instead we have to use something called spark submit so to set up our our little local testing environment one of the first things we're going to do is download and install Cassandra and spark so I'm not going to do those myself right now but the links are [Music] links are here Cassandra Apache org we're not going to download 3.9 if you aren't aware of how tik-tok works 3.0 dot 9 is currently the most stable branch so if you're gonna download Cassandra I would suggest grabbing this 3.0 9 and then for spark since we already have our build file set to spark 162 we're just going to download that one so we'll pretend that I downloaded those and I basically just downloaded them and extracted them to my two favorite directories one is Cassandra installs which has all of these great versions of Cassandra and I can just choose the one that we're going to run this time which is going to be 309 and I'm gonna just start it up so to start up Cassandra on your laptop right after downloading and extracting you really don't have to do anything if we're just going to be testing things out running our code making sure everything hooks together correctly so we just run binned Cassandra it's going to spew a lot of text out its standard out and when it's done we will have Cassandra running on our machine if you want to test that everything is working correctly you can always just do been a cql SH and we can see up everything's here we'll select everything from [Music] the system dot peers and we see we have no peers because my laptop is sitting all alone by itself on this podium all right so we have Casandra running the next thing we need to do is start up a little spark cluster in case you're wondering we're not going to be doing the integration testing today because I decided that the way we currently have to do it is a little too complicated I'm gonna fix that in the next few days and probably do another talk on that later but let's grab our spark I guess I have a slightly different version that I said but once you do that to start up your spark cluster I just go to s pin and then you can run start all so to make sure that your spark cluster started up correctly what you're going to do is check the spark master UI which is going to be at your localhost 8080 and you'll see now I've got a lovely spark cluster started up we're gonna take note that there's a master URL right here that's gonna be a little important in a bit but let's start out just trying to run our application now that we have both of these things running so one thing that we're going to be using a lot is spark submit so that is in the bin folder inside of your spark distribution and you'll see we have spark up sorry I've CD at the front there that's not then spark submit fun little tool it's got a lot of different options but basically what you need to know is that sparks submit will do a couple important things for you the most important thing it's going to do is it's going to put spark on your class path it's going to put it on the class path of your driving or driver applications going to put it on the applica class path evolve your executors that are starting up the other important thing that it's going to do is it's going to actually distribute your application jar to all of your remote executors so if you're not using spark CIMMYT and use notice that you end up having a lot of class path errors whenever you have remote code running it's probably because spark submit was doing a lot of things behind the scene that you didn't quite notice the other really cool thing about spark submit is that it reads the spark defaults file so you'll see that in this install since we're just running out of at our just we have a file in cuff called spark defaults and this file will actually let you set a whole bunch of different options that will be set inside of your spark configuration automatically when you run things so this is basically a way around having to hard-code different configurations into your spark conf and instead just have them automatically kind of read out of the environment you can also do them on the command line which is what we're going to do now that we're going to launch our little application so let's go to our little live demo we have our example we had our file I'm just gonna quickly build it again just so I get that link to where it is and I'm going to show you two quick ways of launching it so the first time we're gonna launch it we're gonna launch it we're gonna launch it using the local master now local doesn't actually mean that it's looking for something on this machine local when you use that in the master argument actually means that what it's going to do is run everything inside of one particular JVM so that means that there is no remote processes everything is happening within the same JVM in different Red's there's no issues with serialization nothing really gets moved anywhere so it's really good for debugging it's really good for setting up but it also means that things that you see when you have the master set to local will not necessarily be the same as when you actually submit your application to remote cluster so we're gonna do spark submit we're going to set our Cassandra hosts spark Cassandra connection host so that we don't have to have it hardwired into our code and we are going to add the packages for the spark Cassandra connector so sorry that was me otto completing that but basically the spark packages right here data stacks Parker Sandra connector 1 6 0 s to 10 sets that runtime availability of the spark Cassandra connector and all of its dependencies on the driver and on all of our executors so if we were running this in a distributed environment to use the the master set to the address on this web page instead of setting it to local we would see we would see correctly distribute everything so when we run this without saying - - master what actually is going to happen is we're gonna run this all within the same JVM so if I look over here on my master application you'll see it doesn't even say anything's running because we haven't actually communicated with the Master we're running everything within the application JVM and you'll see it did some fun stuff we can go over the code in that application a little bit but you'll see we've successfully run we read some data via the spark context and we wrote some by the hive context and it was very exciting so that wasn't distributed though or at least it wasn't even in different JVMs if we want to do it in different JVMs we need to go back into our very long command line and add master and add a master address so well I happen to have a master URL right here now if you're using maysa this is the point at which you would use the maysa address but we're using just a tiny little standalone cluster and now when we run this application we will see that it shows up on our master UI very nice this is going to do the exact same code so we'll be very less than exciting but there we go all right so we've run our first bit of little distributed code let's take a look at what's actually happening here and how it works and mess up a few things just to show how you can basically troubleshoot some things and work with the system okay the very first thing we're doing here in our in our Scala code is we're extending app so for those of you familiar with Scala this basically says that everything that happens within this definition is going to be treated as the main method it's basically a shortcut of saying I'm too lazy to make a main method just realize that everything I'm about to write is main this is pretty nice let's it's pretty quickly get into the to the meat of things so the very first thing we're going to do is start a spark conf when you make a new spark pump like this it's going to inherit everything from your spark defaults from any command-line arguments from any environment variables so basically all of that is going to get fed in to this particular invocation you're also free at any time to continue to set whatever you like you can set spark dot whatever to whatever you like and now I'll get carried through but in most cases anything that you think might change at some point in the future you probably want to do on the command line or on the spark defaults a good example of this is your cluster management system so say you're using yarn and tomorrow you'll want to run your application on a mesas cluster well it would be pretty unfortunate if you hardwired your master into your SPARC comp because that means you have to recompile your application just to run it against the different resource manager and that's kind of annoying so I like to keep that kind of sort of stuff out of my SPARC comp the only thing you actually need to set is your application name everything else you can basically pass through on the SPARC command line and have set correctly that this next little bit of code is our first little bit of communicating with Cassandra so starting with this Cassandra connector what this is going to do is use the spark pump and look for any of the spark Cassandra connector connection variables so you'll notice that when I started this I used - - pump and from - - Kampf I used spark Cassandra connection host well if you go to the documentation here we have a ton of stuff in connecting but I'm going to go straight down to the reference file here and the reference you can see we have a whole set of connection properties so for example the one that we just set connection host is the contact point the initial contact point for Cassandra cluster so again this is another thing that probably makes more sense to pass through on the command line than just actually set programmatically because you might be changing your contact points you might add additional IP addresses you might do all sorts of different things and the other big thing to note here let's see here is our most common issue with streaming applications as this parameter here connection keep alive in milliseconds so the code that we're looking at over here in Cassandra connector come when it actually makes its connection to Cassandra it's actually using a pooled cache of connections it's going to compare the configuration that you've passed in against any existing connections and if any do exist it will return that rather than establishing a new connection to Cassandra so this means that when I call this Cassandra connection or in comm code I'm actually getting something out of a cache I can actually call this inside of a map inside of some kind of parallel operation and continually get references to this Cassandra connector object and will always be the same session object so I can actually use this thread safe remote safe doing all kinds of cool stuff and not have to worry about it but it's also good for this kind of mundane stuff like just setting up a key space if it doesn't exist and running code with it so this little width session do basically says give me that session out of the cache I'm going to execute two little pieces of Cassandra code if I was a little bit nicer I'd probably check for errors here and handle them gracefully but we don't have time for that all right once we've done that we move right into actually setting up our spark application these two lines here's the new spark context and new hive context this is basically how we talk to the resource manager for the very first time so this spark context line basically says we need to talk to the resource manager we need to get a certain amount of resources we need to set that all up and I'll be waiting until that's ready so if you take a look inside of a lot of the spark configuration options and I don't really want to go into all of them basically what's going to happen is it's going to request a number of course and a number of RAM or an amount of RAM for the resource manager it's basically gonna say I need these things set up start up these executors JVMs which are separate JVMs and have them contact me and when I say me I mean the driver application it's really important to know this when you're working with spark that basically your application is managing all of the resources after it's reserved them from the resource manager after that point the resource manager basically out of the picture so at this point we've gone our set of executors we've got this nice hive context which is how we can do SQL this is a little bit different in spark 2 out at this point it's in spark - oh it's basically a session object but it's basically all the same code alright so we have that and the very next thing we do is we do this little SC parallel wise parallelized is our way of taking an array that exists somewhere on our local machine and moving it out to our distributed machines so this paralyzed one-to-ten basically says i've got the numbers 1 through 10 I'd like them to be split up into pieces and placed on all of my executors remotely so each of them has a little bit of that range to work on and then our next little bit here merely Maps that single number into the number a tuple a two element tuple where the number is repeated twice so that as set is actually good to save into our Cassandra so basically this will then trigger all of this work to be done and a bunch of rows that are just 1 1 two 2 three 3 four 4 five 5 all to be saved into Cassandra and you'll see our table really just kV so that's why we we mapped there for more complicated mappings I would check out our documentation we have a lot of information how you can map to more complicated row patterns using you DTS and things like that below here I'm showing exactly how we can read that data right back out so the same to Cassandra is how you take your data from an RTD which is what this SC parallel lives does that distribute it's a resilient distributed data set and it takes that RTD and puts it into a cassandra table the cassandra table a call does the exact opposite it takes a cassandra table pull the data out of that and gives it back to and gives it back as an RTD our collect action here basically says the moment we've gotten that data out of Cassandra it's going to be remotely pulled on all of our different executors a little bit at a time bring it back as an array so I can take a look at it this is almost always not the right thing to do but in this demo this is just an example of what what you want to do in almost all cases you never want to collect your data after you've pulled it out of Cassandra unless you're absolutely done with it unless it's the results that you actually want to do something with everything else should probably be treated on the the server so for example let's say I wanted to take this data and instead of doing this collect let's read it in as a tuple of two intz and this is allowed because one of the cool things that Sparco sandra connector can do for you is it will automatically take the data that's in your table and map it to tuples if that makes sense and since we have a kv table of just into names we can do that so let's say we want to take that and let's filter out all of the elements where they are the same so we have this cool little bit of logic here actually you know I'm gonna just skip the hive contest as I know we're ready running a little bit slowly and I'd like to actually show how little of this works so we've got this little piece of data here we do this we can then collect it and let's just print that out all right so if we want to quickly run our code again we have to rebuild it and like I said we actually don't have any extra dependencies here so we don't actually have to use the assembly command if we really wanted to we could just use the package command which will build the jar as well and just use that but since I've already got this command line already very long and written out let's just quickly run it and you'll see we're gonna run this fun little little piece of code in case you aren't familiar with that little display that just kind of popped up is actually sorry that was not as exciting because this is let's just do that little little progress bar is actually showing us how many tasks are left to do and how many have been completed so in spark all of your work within RTD is split into partitions and as every partition has worked on that work is called one task and basically a task will last from the beginning of reading a data into the point at which you have to shuffle it once you have to shuffle your data you basically have to stop collect all of your things together perform this large shuffle operation and then you come out of it with a new stage which has its own set of tasks so we do our little assembly here let's run our jar I should probably switch back to local mode because it'll be a little bit faster than this because every time we run this we have to actually start up a whole nother set of JVMs after we start up our very first JVM to actually run our information and we can see that our data is empty cray so we successfully did some distributed filtering on a trivial data set it's not too exciting but let's get into some of the gotchas here say I was really confused about why this was happening I'm sure something that a lot of people would really start to do just right off the bat is they would take this code and you know if you're like me I should say maybe not a lot of people but this is what I would do is I would just put a nice little print line right in the middle of my my code because I I don't actually trust computers so maybe it doesn't really know that x and y are the same but I know they are so I'm gonna just run that example again and surprise surprise I'll spoil the lead here nothing is gonna get printed out and that's really disappointing because I put a print line there and that's like 99% of how I do bug is they put print lines places that I like to see what prints out nothing printed out very disappointing so the reason for that is that when we look at our code here inside of this filter operation all of that is actually being distributed out to this other JVM this executors JVM which means this implicit system dot out print line is the system dot out of the executors JVM and not system dot out of the application I actually just hit Enter Rann so how do we fix that well the very first thing you can do is you can actually take a look at what your executors have in their standard out and their standard in and I think the easiest way to do that is let's go right to where our spark directory is so this is going to be dependent on how you've set up spark but basically the same thing applies in almost all the cases we will have a work directory and obviously I've been practicing something and every application every spark context that you create is going to have one of these directories this application directory if you go inside of it you will get one this fun little I'll show you this first we get a zero and that is the executor number if I have multiple machines each executors JVM number is going to appear here and then within that I'm gonna get all of the jars that got moved out so this is like I said you didn't see me actually having to move all of these jars or say them at any point but when I said - - packages it basically says make sure you get all those things that I said I actually needed in that dependency so all of this stuff is getting brought in because the - - packages but down here we see these two great little things here standard air and standard out so let's let's take a look at what happened in standard out that was not exciting I think I have the wrong application ID so I'm gonna just wipe that out and see live demos are a lot of fun it's just why our work directory oh yeah that would also do it I'm a pro guys don't worry about it so we rerun our assembly because sometimes we forget to do that when we run it once more we are not surprised that nothing prints out to our standard out here because we're running with a remote master which means that the code is actually be run in a different JVM so we don't see printouts all over the place something else is very upset at us but it shouldn't be a problem and now let's take a look at our directory and there we go one of the cool things you should notice here is that we get things done pretty much random order and the reason that we're getting things in pretty much random order is because cassandra is returning them in pretty much random order if you know how cassandra works our partition keys are not actually organized by their value but organized by the hash of their values so we're going to get stuff in a kind of strange order and that's basically what i wanted to show you there so you might be asking well I really want to debug in a faster way I'm only using one machine I want to quickly see my printouts all of that stuff well what we can do is remove this little hook on to our standalone spark master and it will default to local mode now in local mode not only is it going to start up a lot faster because we don't actually have to wait for remote other JVM to start we're gonna go right into everything in the same JVM and the standardout is going to be correct so you'll see now all that great stuff just printed out right there on my standard out okay so that's a little bit about how you can work with with code work with spark Sandra connector some very basics on how you set everything up and run tests I would be remiss if I didn't do a quick a quick overview of the difference between doing this and doing it on DSC so you'll notice the big difference in the DSC build file is that there's only one dependency and that's this and that's comm dot data stacks DSC DC spark dependencies and then whatever a version of DAC you're using that's going to just be a catch-all for all of the things that DSC is automatically going to put on your class path so if you do DSC sparks submit we actually will make sure that all of those jars will be on all the executors and on the main machine so basically this is going to cut out having to actually call - - packages instead you just call spark submit DSC spark submit and it will automatically do all that for you the other cool things that we'll do in shortening our command line significantly is it will actually make it unimportant for us to specify the Cassandra connection host and because that's automatically going to be determined using the DSC machine that you're running on although you can still set it to something else and it's going to automatically set the spark master to whatever the DSC standalone spark master that's running is so all that stuff will kind of get set automatically so I think that's probably a good place to stop this should be enough to just kind of get your own copy of this started up start debugging things and start figuring out why different pieces of code work the way they are and I think we'd be fine to take just a few questions before we move on to the next speaker yeah sounds good yeah in the back yeah so the question is about this looks a lot like a reactive kind of pathway where we're moving around elements immutable elements and we actually are we have our D DS which are immutable and we're moving them around this particular piece of code works on our d DS and if you look into Scala are looking to spark the reactive ish equivalent is something called spark streaming which is something called D streams which have the exact same API but basically everything within spark is I have this large immutable object this large immutable data set called an RTD or a data frame or a data set there's a lot of different words it gets a little confusing but they're all basically the same thing we have a huge set of some kind of element those elements are going to be treated with some set of operators and at the end of the day I'm gonna do something with this set of operators now they don't actually get treated in a streaming fashion unless you're using using D streams if you're using rdd's they're gonna be treated as basically a large block of data getting you think of a large block of these things being moved on to an execution platform iterated over completely and then another block taken yeah so the way spark streaming works is a it's a micro batching approach so it's basically doing this in a mini mini version so there's a lot of talk on the mailing list right now about moving towards a more purely event based model something like the way apex or flink works on things but currently what it's going to do is take the data that's incoming build these small distributed data sets and basically treat those little chunks you know as a little iterator operate on the whole thing take the next one operate on the whole thing for con like that so any other questions way down there so safety Sandra is picking up the arguments here so the question is how do you save to multiple tables so save took Sandra table is actually taking two arguments here but let's have IntelliJ help us out here there we go first is key space name second is a table name so my key space is just called KS my table name is called K V because I'm very boring and then at this point we have our column selector so the column selector is actually where we can start doing mappings and saying like well the element you're about to read is going to be called foo bar but we actually want it to be bar foo so do that mapping and you do that within this selecting object any other questions yeah so the question is is there a difference between the community version of Cassandra and the data sex Enterprise coupled version of SPARC and Cassandra and locality for locality there's really no no difference here though all those features would be the same as if you set up and configured your own cluster with sparking Cassandra running on the same machines the big games you're gonna get from DSC are a few of the things we have around cluster management we have automatic failover of our spark master things like automatically telling you where the Cassandra connection points are and making sure that the application is always up to date with that so a lot of maintenance tools and enhancements around there but performance wise the two are basically equivalent at the time at the moment yeah what was the question I'm sorry I don't understand the Community Edition in the enterprise edition of data sex I mean one so the question is what's the difference what was I talking about the difference of performance then that's between the the community version of Cassandra and the data stacks Enterprise version of Cassandra currently the two are very very similar the things you've got to think about when you're setting up your your own spark and Cassandra integration is are you gonna run spark on the exact same machines that you're running Cassandra on or aren't you with DFC it's basically going to automatically enable you to set up Cassandra and SPARC with just a flag switch if you're not using DSC you basically have to make sure that all of your processes are co-located make sure that they all are reporting the right IP addresses because the way that spark provides information about data locality is through the bound address of the executor so you have to make sure that when you're binding your your executor JVMs when they're binding addresses they need to bind the same addresses that cassandra is gonna report because otherwise there's no way of the two being matched up and you don't get any locality yeah in the back yeah so like I showed at the beginning though the link we have here has a starting spark build examples at data stacks we have Gradle maven and SBT and we have the all of these examples in Java as well if you're interested in that I make a strong statement right here and on the record do not use Java if you have a choice it'll be much easier to work with Scala by far hmm groovy to use groovy and SPARC I don't think anyone's done a lot of that recently it works in scholar EPEL so SPARC actually comes with its own repple so let's say I wanted to play around with SPARC Cassandra connector in the SPARC repple actually I don't want to delete any of that that was silly basically all we would change is we would delete all this stuff I'll do a local example just because that'll make it really easy we're gonna keep this packages line we're going to keep our connection to Cassandra line and we're going to just change sparks emit to spark shell and that's going to just drop us automatically into a spark shell where the spark Cassandra connectors are ready on the class path of all of our machines so the only things that you have to do at that point are do the import for all of the data stack stuff and once you've done that it will automatically attach the Cassandra table method and the save to cassandra table method and I can do chaos okay V do a quick action like that and there you go pull all the data in a fun rapidly way one of the interesting things to know about spark and this is just another quick little detail say I run this without the collect oh it returned instantly does that mean that my code is really super fast as something cache the secret is that spark is a very evaluated language when I do this I can do that I can I can map my thing too I don't I don't know length I don't know if that's a member of that okay it is so you know I can keep chaining operations on to this but until I actually call an action so an action is something like save to Cassandra okay thanks autocomplete something like saved to Cassandra or something like count nothing actually happens so that's something really important to know if you're trying to benchmark various things in spark nothing will actually get done until you perform an action so let me quickly reload that fun there we are and let's say we just want to do a count of this which obviously is pointless but you'll see that this count is what actually caused work to be done not actually typing the line before then yeah in the back there are various terminal operators they all end a single chain of operations though so the questions haven't worth where's are there multiple terminal operators and yes there are common ones are collect take save as file save as something in the new API which is the data frame data set API pretty much all of the sources are something called session dot read and all of the syncs our session dot right yeah it that one more question I said don't use Java but the question is can you should use Python um Python is really great as long as you stay within the data frames API so this is something I didn't talk too much about so far but basically the way that that Python works and just just for your information the way you would start the Python shell is very similar to this you would you know just switch outspark shell for been PI SPARC and I just basically did the exact same thing but the way that Python works inside of SPARC is that you write your code you might write a Python lambda you write that lambda as Python code it gets shipped over to a JVM which takes incest I don't know how to do anything with this so it actually starts its own Python interpreter that it then serializes the code back to Python so now pythons got its own lambda it can run stuff but your data just got read in a JVM so it's actually JVM objects so now all of those have to get converted as well so you serialize all those to Python objects so your Python code can work on them but then you've got to give them back to spark cuz let's say you have to do a shuffle all that machineries in Java well you got to serialize it back into a JVM object if you're using data frames none of that applies if you're using data frames you basically are constricted to a bunch of operations that can all be handled within the JVM without having to basically go out to python there's a bunch of efforts going on right now to try to make better python lambda serialization but at the moment just use only data frames yep and I should wrap up thanks a lot everybody [Applause] [Music]