SBTB 2015: Dean Wampler, Data Science at Scale with Spark
Recording: SBTB 2015: Dean Wampler, Data Science at Scale with Spark
so I this is talk variants of talk I've been giving a few times now about you know really Scala's taken over the big data world that was the reference to the northeast scala symposium talk where what we're seeing a type-safe is that spark is really driving interest in Scala because it's just it's like the perfect storm it's the ideal language for getting things done quickly and efficiently I think know like API or framework better demonstrates the best aspects of Scala than spark and maybe hides you know this sort of gnarly bits that everyone likes to complain about a little better than other system so this is a little bit about what it's like to do data science at large scale with with spark now I put this slide up your not because they want to push my books but because I you know I think this is a developer talk really and I think that every developer talk has to have some XML so there you go so I before he joined typesafe around 2011 I started doing Hadoop consulting I decided I wanted to get into the big data space actually what was attractive to me about it was well it was kind of new and exciting had you know hard problems you know big scale and all that but also they were using a lot of the math and machine learning that I had done in physics and so that was kind of an attraction and I'm actually haven't used any of that since I've started this but nevertheless i started doing Hadoop consulting then and quickly got really upset at the quality of the tools and the experience so this is kind of my LinkedIn profile as of like 2012 where I just started trolling the Hadoop community because I thought it sucked and you know I thought we could do a lot better so in fact it the Northeast Scala symposium is an earlier talk three years ago I said you know big data really needs to be functional we need to be doing functional programming in this stuff and so this last year's session was more about we actually achieve this you know this is three years ago or so and well so if we're going to do data science and scale there's no better platform in the JVM I mean you can argue about its strengths and weaknesses all you want but you know it's obviously a proven scalable system people know how to run it in production it's just a great platform to start with and that's why most people you know including the Hadoop community started with it and oh and I didn't mention these all these photos are from the north cascades national park in Washington state in their Canada but anyway this is one of the creeks in that park or rivers and you know the foundation is the JVM for us and we've had all these great tools that just keep getting better and you know more expansive you know languages like Scala you know just a throw a bone here foreclosure Eclipse you know the IntelliJ a whole bunch of libraries like algebra inspire that are reusable tools whether you're running on Hadoop and MapReduce or spark or whatever and we've got it just an incredible load of systems that have been built many of which were created here in Silicon Valley and look well not all of them but a lot of these are written in scholars as you may know obviously spark scalding was a predecessor in a sense at Twitter that was my you know when I was grousing about Hadoop in 2012 it was about the same time I discovered scalding and suddenly a you know I saw some light at the end of the tunnel summing bird is another Twitter thing probably many BR using Kafka Cassandra as a database and on and on I won't read this whole list um but you know so Hadoop is kind of what people think of when they think of big data and just you're very roughly speaking this is what a Hadoop cluster looks like I write a job in something like mapreduce or some abstraction layer on top or now in spark submitted to this you know resource manager that then splits it up into JVM test running across the cluster so you know how do we do it volume you know basically doing everything in parallel it doesn't matter how sucky MapReduce is as long as we're running a lot of them we can hopefully sort of make it up in volume as I said and that's sort of what the really the argument was and then there's this distributed file system that handles things like durability and scaling up and all that we don't need to get into this most of you actually probably know this already it's not exactly new anymore but the core compute engine is supposed to storage was this MapReduce API that was really MapReduce the names come from these sort of functional concepts were used to although we're all experts here so actually this is flat map reduce it was never really one to one mapping it was always like ingest something and spew stuff out the other end and then reduce what came out the other end so it was really more like flat map reduced but I guess they didn't look good on a slide or something so we ended up with MapReduce but it had a lot of problems it turned out it was really hard to implement algorithms in this sort of narrow narrowly focused idea of mapping and reducing especially if you had to do something that couldn't be done and like that one step of map reducing or maybe I had to sequence things together and the Hadoop API was particularly awful it was really like almost like an assembly language it was so terrible so for example I thought I would go through an example of MapReduce and I want you to you know pay attention carefully as I go through this so first we set up some structure then we create a map step and then a reduced step and anyway there's just a whole lot of boilerplate here and it you know I mean this is a what am I doing here I think I'm doing inverted index i forgot what i said but it's actually not a complex algorithm but there's just a whole lot of bit twiddling and you know sacrificing the chickens and you know all sorts of things that you have to do to make this ecosystem happy and you end up doing a software engineering problem it takes a couple of hours you make mistakes you debug I'm actually testing these things is a another royal pain in the ass pardon my french so it just wasn't a good way to work and but it was a simple problem and simple things ought to be easy right and hard things ought to be tractable so we needed a better way to do that another problem was that sort of a sea change in big data was we need to actually get answers faster so you know 15 20 years ago when the Internet's started it was you know people like the googles and the yahoos and those of the world all they you know they were they were just um you know living in the sea of data that you forced them to invent new ways of handling the scale the best they could do is just try to capture it somehow and then go back later and run jobs to analyze it so sort of batch mode with how the world lived but actually my favorite example of this is like inverted index or more generally like a search engine you know if you think about what Google and those guys do they have these web crawlers that are constantly walking the interwebs you know and they're fighting all the new content and then they they capture like say the URL and the content and then they would run this batch job every few hours or whatever that would tokenize those words are those texts into words or phrases and then build up an inverted index of you search for something here's the list of documents that have that use that word or phrase and also the count of them is important because you don't want it to find the documents that mention Hadoop in passing you want to find the documents that are obsessing about to do because that's probably the content you want but so they worked ok you know 15 20 years ago but then you know these days you hear about a breaking news story you go to Google and start searching for information you don't want to have to wait 24 hours for Google to update its index with you know the whatever breaking information or misinformation is on the CNN site right you want to see it immediately so you'd really like to you know shrink that time from data arriving to value being extracted from it and so streaming is now become very important even for problems that don't necessarily require streaming as opposed to problems that do like avionics where you know if I detect that some ailerons out of position slightly I want to correct it right away before the airplane tumbles out of control or whatever actually some of the latest you may be noticed the latest fighters actually fly in an unstable configuration because it makes them more responsive but they become they have to have computers constantly adjusting if a human could not actually respond fast enough to keep them stable okay and you know I sort of alluded a minute to go to the fact that MapReduce is actually very inefficient in terms of all the sort of micro details but they made it up in volume by running a lot of parallel stuff but even that event he catches up to you you know people can are concerned about power concerned about the size of data centers and so they want better performance so all these things were sort of adding up to and we have Cloudera to thank for this actually you know in late 2013 they said you know what Matt produces had a great run it's been about five years but we've really got to embrace something new sort of the next generation compute engine and spark had started around two thousand nine ironically about the time that MapReduce or Hadoop in general went mainstream but it had been evolving first as a graduate research project at Berkeley by you know guy named Matassa haria but it had gotten better and better more performant more flexible more more durable and resilient and all that stuff and so they embrace they've been Cloudera embrace spark is sort of the next generation compute engine so let's see how it does on these requirements that we have to do better than MapReduce well it developer productivity is obviously extremely important and as well as happiness that I find the productivity and Happiness tend to go hand in hand for me you know if I'm just sort of slogging through something I'm not not a happy camper and I start trolling on the interwebs anyway so we've got these very concise functional api's there's spends a little debate on twitter lately if this is really a functional api no it's not perfect but it is actually a lazy you know sequencing of Combinator's kind of a pil show an example shortly that evaluates lazily the actual pipeline so you're basically defining data flows normally except maybe in the sequel query ish kind of alternative and then you're evaluating that pipeline at some point down the road but the cool thing is for you know the broader community of people doing everything from ETL to joining data sets to data science on the other end of things you can actually use really for actually it's more like five api's which is you know our favorites Scala and Java but also Python and our which of course are very popular in the data science world and even just writes sequel queries which is one of the things I love about the sort of the life history of sequel is how it sort of went through this phase of being like oh you know that's like old technology we're onto no sequel right so we don't need that stuff anymore but then here it comes roaring back because you just can't keep a good man down I mean it's just such a powerful tool that we it just keeps coming back and continuing to be useful but anyway you can get to choose all you know all five of these options if you want for whatever problem or your you know preferences you have I think all of us know especially if you came from Java to scala it's a revelation when you suddenly get this you know interpreted evaluation loop the repple but you can just you interactively play with things and experiment and learn and even do real work that way and you get this with all but the Java API so I should have added sequel to this I guess I did that sequel to this okay forgot but we also need a lot more flexibility i mentioned that MapReduce is very restrictive with a you know a rich sweet of functional Combinator's we can think about the problem that you're the most logical transformation step two step is we're going from A to B and then you know actually just implement what makes the most rational sense and that's a very liberating capability rather than just have to do this context switch oh I've got this idea of how to do this now I have to figure out how to force fit it into MapReduce and this sort of layer of Combinator's and some of the other things that will get to about spark also makes it relatively easy to add stuff on top of it like machine learning algorithms iterative algorithms like training machine learning and also graph traversal kind of algorithms so these things are just kind of natural extensions when you have both the performance and the flexibility to work with them actually one of the things i'll mention i don't think i really put in the slides but it's something that I think is underappreciated about tools like spark is the fact that they scaled down as well as up that kind of goes back to this idea I can run it in a ripple either locally it turns out I can also run it on a massive cluster in a rep if I want to but scaling down is really crucial for when I want to just experiment or I don't really truly have a you know big data problem i have a small but crazy data problem i don't want to necessarily have to spin up a thousand node cluster you know just to you know add one plus one or whatever so it scales down beautifully effectively under the hood the reason it works fairly efficiently is it's building up a directed acyclic graph lazily of whatever pipeline you're defining and it does things like combining you know sequencing steps together in two stages that you don't necessarily require like a new JVM process to be started every you know I go from map to flat map to filter I don't need three JVM processes I can mash that together into one JVM process and you get efficiencies that way you have capabilities for doing caching of intermediate data so you don't keep reevaluating the pipeline when you're going to start at a state you get to a point and then from there you know go off in different branches you could do that kind of stuff yesterday I had the privilege of visiting the IBM Silicon Valley lab which is down still in farm country if you've ever been that far south 101 and I gave it mark nerd risky and i were there and we talked about scala and reactive streams and all this stuff and i didn't realize it when i was up here yakking away about reactive streams and spark that too IBM fellows were setting in the front one of whom invented row-level locking in db2 and the other guy had been doing essentially back pressure and streaming data for about 20 years and I'm you know talking to these guys you know like well there's this thing called a back pressure you know and and then afterwards they come up and talk to me and they and what they were going on to telling me about all of and when you do you know back press you've got to be careful about this you know scenario if you've got a bunch of consumers you know how do you figure out the logic of getting it back and if you don't you know stream everything together in two stages then you're going to get huge bottlenecks and anyway it's just kind of amazing that we keep reinventing the wheels over and over again these guys have been doing it for 20 years but okay back to the topic the other thing that's really neat is there's this new data frame API it's built on a sequel query planner that they implemented called catalyst that actually gives us the same performance whether you're writing in Scala Python Java or are or sequel which is really the first time in the big data sort of space that we've actually been at a point where you could actually write and deploy Python code and have it run at the same speed as Java code it used to be with like mapreduce your data scientists would do their nice model in our Python and then they throw it over some cube farm wall to the you know the Java hackers who would hopefully get it right converting it to Java to get the performance those days are pretty much gone now and then I think this is the last bit on this is that it has it's interesting spark was actually implemented as a batch mode system just like mapreduce but because of its relatively good performance they came up with a clever hack to do streaming which is basically a mini batch model it's not designed for individual event processing or subsequent latency like avionics or whatever but if you're okay with capturing data in like one to one second to say few minute intervals and then processing those as a mini batch which is great for a lot of problems for which we use streaming then it works really well and it actually it has another interesting benefit especially for those of you doing the lambda architecture where you often have like a streaming pipeline and a batch pipeline a classic problem with that architecture is that you end up writing the same logic twice and in like your streaming API system say storm or whatever and your batch system say MapReduce with spark streaming you basically write the code once and then you just repurpose it for batch or in many batch streaming so it's an interesting capability there at the core of spark is this notion of a resilient distributed data set the way I think of it it's like a sharded collection or partition collection over your your cluster and it's resilient in the sense that if one of the partitions is lost because it's built up this directed acyclic graph it can actually go back and and reconstruct the lost partition and then they layered on top of that this notion of a discretized stream so if I'm capturing these many batches then each of those will be an RDD and you have like a sequence of these are dd's over which I can do window functions if I want like moving averages and things like that okay well this is this is a scala conference so this slide is maybe a little less relevant but there are some data scientists like Vitaly Gordon who have embraced using Scala for data science as opposed to python it's and obviously that's the how I want to do this to that this is a snippet of the inverted index actually implemented and spark for times like I'm not going through the whole thing but what I love about this code and I think all of you have experienced this writing code with Scala collections or spark or scalding is that you have these Combinator's in red that you just sequence together and you you know a pass in these functions to do the actual work my favorite line is the one next to the word powerful we're basically I take a tuple with a nested temple and move the parenthesis you know in like you know 20 characters to restructure the the data for the next stage and the pipeline it's my absolute favorite line in this whole program because it's something that would you know if I did this in job I'd have to have like you know a factory tupple master factory something but but here I just move parenthesis and I'm done right but we can also support our larger ecosystem besides this kind of core you know functional API we can write things like sequel in spark and have it run at the scale we need and they give us basically three ways ways to do this and I'll briefly talk about the first and the last we can integrate with it was sort of the traditional standard sequel tool for hadoo which is high invented at Facebook and it's one of the things that became one of the most important tools and to do for raising the abstraction level above MapReduce spark has its own sequel dialect that's slowly getting to be you know like a superset of hiyo sequel and then there's also this data frame api that i mentioned i'll show you an example of that but if I just want to write sequel queries this is all it takes you know I basically do an import statement and create a spark context and wrap that in a hive context those are like the sort of the entry points to the system and then I can just write sequel queries like I would before now the show method is sort of like basically just dumps the first 20 values in the resulting expression that basically you get a data set back and it gives you the first 20 or so now those of you that you know like strong typing which most of us hopefully do will say well this looks this doesn't look so great this is strongly typed programming right I thought that was a bad thing and it is a bad thing I wouldn't do this in production but I would do this in the repple all the time because there's nothing like writing a group by statement or a joint statement and sequel it's always faster to do it that way than to write it in a an API because sequel is so concise and if you ask if sequels alien language you know then you've got a tool here that you can actually use all right so I guess I said this already we're going to just write sequel queries now if you prefer Python it turns out for this example all you have to do is make a few changes just replace the import statement with the equivalent in Python and delete the Val keywords and it basically is the same code the Python API is dramatically similar to the Scala API in general there's more to it if you did the general API but it's really easy to go back and forth but if you only want to write sequel there's actually a rep illusion that just takes you into a sequel prompt for hi basically and you just write sequel queries as if you're in any interactive repple for a date of database that you've ever used so those same queries you could just write like this now so strongly typed programming that's bad right and turns out you can do all of this with a more typesafe api called the dataframe api if it's inspired by similar api's in Python and are for those of you that have ever played with those so and this is basically what it look would look like I find it easier to use the sequel personally because i know it i don't quite really know this api well yet but nonetheless i would use this in production jobs because i can at least you know do type type checking but basically the same sort of thing we set it up with a sequel context load some data in parque is a really popular column-oriented format in Hadoop now the example I'm pretending i'm loading word count data which would be like you know I tokenized a bunch of documents and counted the words in them so the the data schema will be like you know a word and then it's count so two columns schema and then I can reference stuff like your dollar count this is a basically an interpolated string that they provide it may be the spark API and it lets me reference the count field in this data set and I can order by that if I want to I can show the first twenty character 20 rows I can cache the data because I'm going to use it subsequently for other queries and so forth I can do things like filter for all of the words that are greater than 20 characters and length rights the sort of stuff that you would typically write with sequel but in a type-safe way and then I can save the results back to my file system if I want in Parque format let's say so that's more of how you would use this you know in like a production job I think the last apil talk about there's several add-ons on top of spark but one of the more fun ones is the machine learning library called ml lib and we'll talk it i'll actually walk you through an example of streaming camis so what this means is i'm going to this will be a streaming context right data coming in k means is one of the simplest algorithms in so-called unsupervised learning where I don't know much about the data but I think it clusters in some way you know if it's like actual Geographic data like actually a favorite example i played with recently is there so you know a bike sharing service in Chicago called divvy now you can download their data sets and you could do clustering on that to see where most of the trips happened you know they happen to be you know clustered around like the loop area of chicago or up in the you know the restaurant areas and that sort of thing but you could do this in streaming so as events are happening so you're actually updating your model of the world in real time so to speak or you can do it statically if you want but this one I think nicely combines a bunch of these pieces running on the core the machine learning part the streaming part all you're running on core of you know some good ideas resilient distribute data sets and so forth so you know it starts in the usual way with a bunch of import statements that I wanted to show for completeness and you know everything I've shown you basically your scripts or scala scripts I could have done this on the repple if I wanted or just the source to script this time what happened to my thing here well I did something wrong here let's try that again like I hit the wrong key there we go this time I'm going to wrap my spark contacts with a streaming context so that gives me the ability to say I want to capture in this case every 10 seconds of data as its streaming in I want to capture it I'm going to capture actually two streams of data one of which will be the data that I'm going to use for training and I'm going to parse these into so-called vectors these are these aren't this gutless usual Scala vectors these are sort of a sort of a machine learning notion of a vector which is basically like a list of features it could be like latitude longitude it might be housing price number of rooms location and you know that kind of stuff and then also test data which is already labeled to be in a certain cluster and so that's what a labeled point is it adds a basically a label to a vector and then I can train a k-means clustering is create an instance of this object and give it some properties like how many clusters do I think there are let's say there's five and that might be what K clusters is decay factor is how much do I want to use the data that I saw previously or do I want to start clean with each iteration and then you can also say how to initialize basically what k-means does is you guess five locations and then you find like for all those five locations you know which points are closest to each of those centroids and then you move the centroids to be the true center of those new clusters recalculate which clusters are assigned to which centroids and iterate that way and it turns out very quickly you iterate to a static point where you've got things clustered to its nearest centroid and that's all this is going to do but it's going to do it every 10 seconds is I run so I'm building up a pipeline here I need a function that will take a labeled point and tear apart the label and features into at uppal because I'm going to pass that to these guys so one of those datasets it's going to be streaming in I'm going to train on it every 10 seconds the other data set i'm going to use to predict what cluster this data belongs to based on the training data I've got my data flow all set up i'm going to start it and just let it run forever and wait for it to finish which you know usually means control C unless I give it a timeout well anyway whether you understand k means you're streaming isn't really the key thing what I really want to get across is just the concision that's available and the ability to when you have a good foundation you can build on top of it a bunch of different tools like streaming like machine learning and then run them at large scale so that's it thank weird Alexi go hook