sfspark.org: Holden Karau, PySpark Beyond Shuffling -- Why it isn't Magic
Recording: sfspark.org: Holden Karau, PySpark Beyond Shuffling -- Why it isn't Magic
thank you yeah so I'm going to be talking actually not about Scala stuff today I'm going to be talking about Python and spark but it's it's all running on top of Scala code at the end of the day so we can we can just squint climp attend and yeah if you want to follow me on Twitter it's Holden Caro I do occasionally tweet about computer stuff right now mostly really depressing stuff um so maybe don't follow me for three and a half years Oh sticky device oh damn it no clicky device for home okay I will yeah okay so um I normally have boo as my co-presenter she she comes with me to almost all of the conferences but we gave her a bath this morning and she wasn't quite finished drying off so she's at home don't worry though she's she's doing fine and she will be with me the next the next talk that I give right so yeah my name is Holden my preferred pronouns are she or her I work here this was the least far I've ever traveled to G but meetup talk so thank you all for coming to me I really appreciate that I'm very lazy and yeah I'm a spark emitter and I've worked at a whole bunch of places including Foursquare which has not yet gone bankrupt I checked this morning and their applications still work so it's exciting and I'm a co-author of a few books I'll try an get you to buy them too later and this is just not related to software so much but after the sadness of last year I added this slide just in case you don't happen to know people like me I'm trans I'm clear I'm Canadian I'm here on a work visa one of the ones which people are talking about changing which is just not at all stressful and I also consider myself a part of the leather community and I write the same really bad spark jobs that everyone writes I I get out of memory exceptions from my code - there's no secret like queer Java garbage collector that we've been hiding from you we're nice people if we had a better garbage sector we would share it with you so you know we kind your colleagues especially the ones that need hugs yeah rockin VGA already do this slide um if he still here he is dammit I can't make these jokes but you may have noticed our very green lobby as you were walking in and I like to think that this is why we make really good software it's not the people that work here it's that we have this green lobby and this should convince you that we make only the finest Python and Scala code so I'm hoping you're all nice people I'm really curious how many people know about Apache spark like okay awesome now how many people have no idea like this is their first time they've heard someone talk about spark it's okay don't be shy okay there's like one person - okay and a few more hiding behind the camera man okay lovely so I'm going to be talking about the magic behind the PI spark and it turns out the magic behind PI spark is 300 small gnomes hiding in a box and if this is your introduction to spark this might be kind of a scary talk please don't be scared away all distributed systems are terrible we're just talking about how spark is terrible with the don't worry everything else is bad - um hopefully you're interested in making sparks jobs work better and by finding out how spark is terrible we can avoid the parts that are going to like steal our cookies and our lunch money so for the people that are new to spark its general purpose distributed computing it's awesome it's magical it totally fixes all of your problems by a support contract from IBM today I can totally even have products I opened product no you're shaking your head okay um whatever if an IBM salesperson comes to your office you want to buy whatever it is they are selling I kind of failed the marketing training class but they made us take a few times um but yeah so this work isn't a touchy projects not nice but we contribute to it we've got a lot of other people here working on it and there's some people down the road of data birds working on it and there's some people up the road at Cloudera working on it and there's some people who like work all the way down in the South Bay very very exotic location and Hardin works working on it and people in other countries too so most people come to spark because they're like Map Reduce that is so last season I need something new to convince my CTO to give me a bunch of money I'm going to use spark the other reason people come to spark is they're like I'm getting a lot of out of memory exceptions how hard could it be to do distributed computing and then it turns out here kind of but yeah let's just go on to the next slide yeah right so this is this is the cat who's coming to spark because they're like well my MapReduce job is going to take 16 hours if I drink a lot of coffee I can probably learn spark in those 16 hours and you can with one of my handy new books um and the other one yeah and this is the other cat this cat has found out that pandas runs out of memory when you run in on a MacBook Pro and they're like I know I'm going to use this yorgi computing to solve my problem and the other reason that people use it is there's a magical cat and 600 gnomes inside of it it's not a guarantee okay so what is the magic of spark right like we all know spark is amazing and it's in all of these fancy magazines but like presumably it does something different than Map Reduce and all of these other distributed systems but so what is the magic I think the key magic is this thing called the directed acyclic graph which if you spend a lot of time in grad school sounds like a very reasonable thing to name something and if you you know finished undergrad like me sounds like someone spent a lot of time in grad school but we can just think of it as like a query plan but like less fancy we can think of it as being that's where the 300 gnomes are finding out they're going to hide out inside of our directed acyclic graph and then our optimizer is going to be able to look at our really fancy query plan and do some really cool things the other part which is really important is spark has a different resiliency models and other distributed systems traditionally distributed systems often depend on doing things like on multiple machines and then writing the results out to a bunch of different machines so that one goes up crap um so that one thing is cut damn it so that one things go wrong it can recover by just going in and reading in the missing data but in spark it's approaches like now whatever that's cool if something goes wrong I like have this query plan I'm just going to go back to my query plan figure out what it is I was trying to do and I'll just recompute that in the same data and it turns out recomputing some missing data is often a lot faster than writing to three disks over a network writing those three disks over a network is kind of slow um the other thing is like we got some cool magic stuff so let's just move on um words yeah we can just get this one we've got mostly smart people right this is our magic yeah um and the other really important thing that I think we don't talk about a lot is that rdd's don't really exist at the end of the day an RDD is actually really just a query plan and it's kind of like a 16 year old you ask it to take out the garbage and it says yeah I took out the garbage but it's only one to actually go and look to see if the garbage got taken out that it gets around to actually taking out the garbage right and if you if you have to do a bunch of different things it'll put off doing all of that work until the very last minute unlike a 16 year old it'll actually probably do the work better as a result because it's no like oh I have these five different things I need to do I can combined a bunch of these steps together for you I'm this is really cool I think it's I think it's awesome and so here's a picture of what the dag looks like we can see like we've got essentially some data coming in here I was reading in text files because I'm lazy and parquet is too much effort and then I was doing a bunch of sort of functional programming stuff on top of it and so that's pretty cool right but spark has figured out that it can combine a whole lot of these steps together and it can actually even figure out that it didn't even have to do this step because it had this data cache for me but here it was like oh yeah you asked me to like do a bunch of things on the same piece of data but I can just push all these things together into one operation any type of spark and this is really important because iterating over our data is kind of expensive right like if everything fits in memory that's great but if I have to like pull stuff in from disk and like do that stuff doing multiple passes over my data it gets really expensive so being able to combine to this awesome the other thing is there's a thing called Cori plans for data frames how many people are using data frames that's not everyone I'm so upset just take up that take up set query plans are awesome data frames are really cool they allow spark to sort of see a whole bunch of things that are going on so here spark really just sees like I was doing some cool functional programming business like cool fancy but here sparks he's like hey you are doing some aggregations I can optimize aggregations and do really fancy things and so there's some extra pieces of information inside of the query plan right and our traditional word count example you still be familiar with this as you've noticed it is in every big data talk ever except for Fred's talk which was right before mine and he will probably be hearing from the licensing bureau like I did the last time I left it out that's why this has to be in all of our talks if we leave out word count they take away our license and as a foreigner I really don't want that happening now is not the time to lose my Big Data license so I was already counting Python and so I've got to input data I do some work and then I saved my so next slide and this is the slide where our 16 year old has to take out the garbage so once we do this action right we sort of force spark to do its work but it can it can combine a lot of these previous steps together and do less passes over the data and it can do some fun stuff as a result now this is really awesome I mean we've talked about how it's cool and we can do this pipelining but that's really funny but people don't think super excited so we'll move along and it turns out that there's this pipelining stuff it's like it's really cool but the magic only goes so far right it's like we went to Hogwarts but then we didn't really pay tuition and we got kind of kicked out um that's that one needs a bit of work okay um so it doesn't have a whole program view and this is the biggest problem with spark from the point of view of like an optimizer right support can see what we're asking it to do right like when it got to that part where we asked it to take out the garbage like spark was like oh I know all of the things I need to do to complete this step what it is no idea what is coming next right if we're do a second thing if I take those word counts and I put them into my awesome totally real business logic then spark is going to recompute all of those word counts for me and that's going to be kind of sad and so this sort of limits what can go on the other part is it can't see inside of our map functions right so when I take data and I tokenize it like like we did on the previous slide support doesn't really know that I'm tokenizing the data it knows I have this function but it's running but it doesn't really have any idea of what's happening inside of my functions but there's there's some there's some more magic and we'll talk about sad things later umm what our data is magically distributed yeah fine um so okay the other really important thing is that our data is magically distributed so where are didi splits the data up onto all of these different workers for us and at the same time it splits up the work that has to happen on the day - to the workers that have the data very keeping it that way it does shuffles for us so if we have to join data together it like figures out where everything goes and does really awesome stuff I'm will skip the bottom part that says yes right so when we say distributed what we mean is 300 gnomes um and each one of these gnomes has a little piece of our of our dictionary or whatever it is that we're building and then we go to each gnome when we ask it to compute whatever piece of data we need the only problem is when we do something where we have to like sum up our results from each other all of the gnomes have to talk to each other and that goes kind of bad now thankfully computers don't drink as much as gnome but like this can be a really bad and hard thing and this is this is 40q comes to the anti rescue um I need a better word for that um but me villain I don't know Keith Q could be a villain zip codes no okay I'll figure out a good magical cake for that but so keys aren't evenly distributed and this is a really big problem with real data with fake data it's great I can use the random RTD generator and I can make some really nicely evenly distributed stuff and pretend that I don't have Keith Q but if I work with humans they have this inconvenient property of clustering together in cities and it's just if they were just normally distributed across America it will be so much easier to do all of these statistics and there's all of these other clustering like by age it turns out humans learn evenly distributed across the range of 2 to the 32 like clue so when I group my things together things go kind of poorly and so what happens is instead of splitting up our work evenly amongst those 300 gnomes one of those gnomes becomes responsible for New York City and that home gets really depressed I've known just don't do a good job and if you're lucky you maybe get an answer from that gnome like at the end of the week but if you're unlucky that gnome just gives up and drinks and then your your job fails and you don't know what thinking no yeah and so here we can watch one of the cases where gnomes are going to fail oh I love this example okay ah yes okay sorry um I so this example in which which zip code do you live in oh damn it okay close enough I'm going to pretend you live in 94110 but whatever um so this is my example for what I want to do after I get out of software industry because this is kind of god damn it okay yeah um so I've got a bunch of information about where people with handlebar mustaches live and I've decided to open an artisanal moustache wax shop um and I'm really excited about this but I need to figure out where to put my artisanal moustache wax shop if I put it in the wrong neighborhood in San Francisco I'll go bankrupt and I'll have to get a job in software again and that's going to be sad yeah and so here if I took my data and I tried to sort it to figure out where I should put my art is no mustache workshop we can see one of my gnomes has far too much data and it's the gnome who ended it up responsible for the Mission District because there's a lot of handlebar mustaches out there and that gnome just gives up and goes and days drinks in the park just like the Mission District um but so we we can see that this goes poorly and we don't get an answer because there's too much data but it's okay we can just like help spark out and when we have Keith Q we can allow spark to do splits within the same key and we do this by kind of doing bad things to our key and we we we add some letters to it and you might be going holding like why doesn't spark do it you know spark self like it's supposed to be magical and fix all my problems I read that in info week um I don't actually know if they ever said that but it turns out that the problem is we have deterministic partition errs so because spark when we do the shuffle all the data with the same key has to go the same gnome or note no note and so then and then it's sad but now we can make it so that we've got different keys and everything will work out just fine and clearly right we we don't have to have all of our data on the same machine when we're reading it in this rule only applies when we're shuffling our data and so after we've done a shuffle we can go ahead and turn these back into regular keys and everything will be fine and we can do a processing it turns out that we can combine this sadness with another kind of sadness to get extra sadness um I actually really like known but in this example they're just not going to have a good time so there's an even worse way to do word count let's look at that yeah okay so we can tell the cat is getting covered in water so this is going to go poorly for at least a cat and probably whoever was silly enough to take that picture so we've read in our data we tokenize it things look fine so far we group it together by key and then we sum the values and then we save it out to a file called Boop because I like poop okay and so I give this on kilobytes of data so that the job would succeed and we could actually see what's happening but we can see here we have an input of 385 kilobytes genuine big data please buy an IBM support contract we have a shuffle grade of 424 kilobytes once again genuine big data in a shuffle three to forty eight kilobytes and next slide how damn it there was supposed to be a slide where those numbers were lower um moving on um but if we think back to the previous slide we note that this number here was bigger than this number here and that's sad when your data gets bigger that's not normally happy there's there's the saying the cross product is big data and big data is an out of memory exam and so what do we do instead instead of group by key we can use reduce and this is because when we could tell start to group things together by key it's it's forced to make a list of all of the things for that same key and if we do reduce by key spart can actually see what's happening if we go back a slide oh two slides sorry right so this this black box thing that I'm talking about is that spark can't see inside of this lambda expression it has no idea that what we're doing is just this really simple sum if it could see inside of this lambda expression it could move that up for us and do really happy things but because it can't introspect Python or Java code it can only sort of see the fifth part right it can feed math values and group by key the optimizer is sort of limited now we go forward to spikes yeah three slides oh this is the one where the numbers are lower yay and so we can see here that we have a number then we have a lower number and then we have a much lower number and so this is when we use reduced by key what happens is spark reads in the data it applies the reduction on each of the individual machines and then it shuffles the reduced version and so each Nome only has to deal with the information about how many handlebar mustaches are in each zip code rather than having to deal with a list of every one handlebar mustaches per zip code the alternative is we can use data frames yeah yeah I got yeah okay data frames raise the roof um-hmm oh damn it there's a video camera well whatever I found stupider things on video umm so data frames are really awesome we can escape back to our TVs if we if we get in trouble but the cool thing is data frames gives spark optimizer a chance to understand more about what we're asking it to do it does limit what we can what we can express some degree but there's so much cool that it's worth it let's keep going let's look at some cool um so we can do a functional and relational word count together in Python and so here we you know we load our data once again I tokenize it with the very high quality space tokenizer i construct these row objects and then i group it together by key and then I take the sum and this this group is actually safe it's a safe group by its friendly it doesn't stab us and take our lunch money I like this one okay let's go back um so why is this a so this is safe because it's on a data frame and so what's happening here is instead of giving it a lambda expression afterwards we're calling it with like this named sum and there's there's a whole bunch of different kinds of aggregations that we can do but essentially because it knows that we're doing an aggregation and we express which aggregation we're doing it can pipeline that aggregation intelligently for us we don't have to manually do it with writing reduce by key and not to nog reduce by key like it's pretty okay but like if I want to keep track of like six different aggregates I have to like keep track of like X 1 through X probably 12 look I'm really bad at naming variables like I'm going to get them wrong and then I'll report the wrong average sale to whoever it is that cares and I have to find a new job again and like I only want to do that every 18 months not every 6 joking joking I love my employer they are perfect in every way um right so we can see here group by key on rdd's is pretty bad if we do reduce by key it performs pretty okay and the Susan Scala but if we do it in data frames it performs even better and that's because the data frame optimizer gets a lot more information it knows what we're asking it to do and it gets to put extra sprinkles of magic on our code and it gets to use really efficient representations of everything yeah our final bit of magic is how Python and Scala work together to be friends or at least frenemies um and this part is sad this is the part that might scare you away I'm sorry but it's okay hi spark is essentially an answer to the question of how expensive is inter-process communication and the answer is kind of expensive but with an IBM support contract you just buy twice as many nodes and pay twice as much for support and your problems are definitely solved yes that was it um so there's a large e tore into the internals of PI spark um this is what a playground would look like in my home country and not in yours I don't think my understanding is that you have layers let's go on um we have lawyers in Canada too they're just busy suing over Tim Horton um so spark is in Scala how do we make this work right besides just showing pictures of playgrounds I need to make the IPC work so far has this concept of a program called the driver program and we can think of our driver program as sort of the coordinator it's the one that tells all of the other gnomes what it is they need to do it asks all of these gnomes to do work for it and on the driver program we use this thing called PI for J and Piper J is this really interesting library for calling Java from Python and it kind of works pretty much almost some of the time um but that part is okay because the driver program is well it's the part that you work with the most it's also kind of the least important part cuz there's one of it right if something is slow on the driver program it's only slow for one mill but when the things are slow on the executors all 300 gnomes do something slowly and that's where things get add like if one gnome wants to drink that's not my problem but if all 300 gnomes are drunk I have a problem I used to be funny okay exciting so what does this architecture diagram look like um so besides illustrating the fact that fred has access to the designers and I have yet to trick them into working on my slides this architecture diagram indicates that each worker has a Java process that communicates over UNIX pipes to Python um and I think the really awesome thing about this architecture is that it somehow works I like super impressed um so what happens is the Java process will go out and read some data from HDFS or s3 what's the IBM version called or IBM whatever it is objects or okay object store um and then load it into memory and then it'll get copied over to Python and Python will do its fun like it'll do numpy and Skippy or pandas and all those happy things and some little about Java and they talk to each other and then you know the data gets copied back and forth a lot so how does this break right like this wouldn't be a talk from me if it wasn't about how life is terrible and so it turns out the main thing that sucks is that we have to serialize our data a whole lot every time we copy our data from Python to Java and back and forth we have to serialize it and we serialize it using a thing called cloud pickle um which sounds nice clouds are fluffy clouds are good you can buy them some companies right and they're the future but it turns out cloud pickle is really just pickle and someone called it cloud pickle because that sounded a lot nicer than pickling my data we out deckle it okay yeah the other parts are all like these are all things that are bad they're far less important than this one though at the time we'll just look at that one piece of sadness um so what do our nose look like so I said that we actually have 300 gnomes and this is how caching is implemented in pice Park it is well I can read this because I spend far too much time reading this code but it turns out that like almost everything in pice Park is just really simply asking Java to do all of the work for it and that's really cool because it lets us just take all of the work that Java people did and just like copy it but without even having to rewrite it into Python code because like I don't know how to rewrite a factory Factory in a Python that's just going to make me sad I can just call the factory factory and everything will be fine right um and this is even super more I should have put the site elsewhere but if we think back a while when I was talking about why you should use data frames this is super true for Python users so this like double serialization cost means that reduced by key like it works right like it succeeds it's still twice as fast as doing group I key but like doing it in data frames is so much more awesome because when we do it in data frames pi spark is able to take what we're asking it to do and compile it down to JVM Jade code for us and it's just amazing I can like write my Python code and have that stuff runs out of the JVM for me and I can even pretend it doesn't exist until I get an exception and then I give up and you know I drink like a gnome but that's future Hold'em um and there's a bunch of future things happening I think they're really cool um there's this thing called Apache arrow I think it's really awesome it gives us the possibility to do interchange between the JVM and Python without using cloud pickle all the time and this we can manage to trick and convince the spark community into using Apache arrow I'm going to have a little party and you're all welcome to come to my party it's BYOB and downstairs in front of the place until the security guard kicked us out but it's pretty fun and it will give us like a really good 2x speed-up the other thing is if you work with PI spark I really want you to share your UDF's with me please share your the things that you're doing inside of your lens expressions with me I keep trying to trick other people into doing things that will make spark faster but they keep coming back to me and saying like I don't believe your benchmarks and that's just word count um I'm like I can I can make more fake benchmarks but if you can like give me things from companies which aren't mine I can try and convince people that the changes I'm proposing are useful for like at least three people because right now I've got them convinced I think it's a funny idea and the last time we merge that code it went really bad so like I I need a new strategy right um there's a bunch of really cool things you should by Pavlos introduction to spark video if you an expense account but I have a new thing that I'm trying to sell you so we're going to skip this one there's a bunch of spark books don't buy this one I'm really sorry it's out of date I really wish we'd stopped selling it I mean if you want to give me money IBM compliance training said I can't take $20 bills in brown paper envelopes so you could just buy a lot of copies of this book and like send me a nudge nudge wink wink I really cut down at that Germans are in the back um there's learning PI spark which is from some of my friends IO one of the co-authors of this book of beer and if you buy it please let them know that I don't own the beer anymore his name is Denny he's lovely I'm sure it's a very good book but the best book is the one that I make money on so let's get to that book and yay you can buy it a normally I have to try and sell people a book that doesn't exist but it exists um so you can buy it from O'Reilly today and I get the best royalties when you buy it from O'Reilly directly so if you have a corporate expense account this is the gift of the season um if you have cats or dogs just please buy several copies I like drinking coffee and drinking coffee in San Francisco is expensive if you want to buy it from Amazon because you hate me and you don't want me to get as much money from royalties it'll be available from them in a few weeks but you can buy it from O'Reilly today oh yeah if you want an excuse to come to Berlin Porto Vancouver or somewhere in Australia I should really figure out where someone told me that's kind of a big country um you should come join me at one of these conferences I'm sure they will all be excellent does anyone want to go to Berlin come join me in Berlin tell your boss it's super important I don't remember what I'm talking about but I'm sure it's good okay so this looks pretty much it for me this is another one of my stuffed animals so yeah okay does anyone have questions or do I just like run away to the sub dammit oh wait yay thank you for asking that question um so the question was do data frames offer any improvement in portability between the JVM and Python and the answer is yes so underneath the hood rdd's in Python are rdd's of pickled strings and that's just depressing if you want to try really really hard it turns out you can parse pickled data in Java just no one wants to and you shouldn't but data frames actually represent all their stuff inside of the JVM heap in a format that Java can understand and we know how to transfer it to Python so if you use data frames you can you can actually call in to Scala libraries that take data frames and use them from Python you don't have to wait for Python libraries admittedly doing that is one of the ways that I have accidentally someone Cthulhu but I will I convince them that my soul had already IBM owned it at this point so it was time to go back to the underworld but you know it's it's doable it is a little what's that expression you can do it if it breaks I'll blame you unless you bought a really good support contract like that's like that's like the support contract where you can like make an engineer show up I don't know how much those cost but like yeah but please do that and make your startups code depend on out startup make your company you're you're like publicly traded um make your company's code depend on this really complicated feature and then like I can make money wait no that's not why I do this software yay okay oh I'm sorry all were there any other questions or oh yeah there's a second question [Music] yeah here let's do it one at a time I'll answer your first question I'm going to answer your second one otherwise I'll forget your question part of the way through I've done this so the first question is is it my lambda expression that's low or is it the mapping part that's slow and the answer is like yeah your lambda expression in Python is a little slower than the same logic expressed in the JVM but the expensive part is really just copying the data from the JVM to the Python executor to interpret it's not that your lambda is slower it's really just the copy and I actually have this really interesting proof of concept using JSON where you avoid the data copy and and you still do Python evaluation on the data and it runs within 5% of Scala native code performance the only downside is it adds 100 Meg jars as a spark dependency and when I tried to get them to take that pull request they just laughed with me so that's not happening but arrow arrow is going to fix the copying part and things will probably get a lot better and like spark 2.4 and the second part of your question I cut you off do you still have it ok maybe later yeah all done next speaker yeah you [Applause] you [Music]