1st Spark: Alexy Khrabrov opens the very first Apache Spark meetup in the world with Matei Zaharia
You can embed it. >> All right, guys. Let's get started. >> Uh, my name is Alexi Kro. I'm a research engineer here at cloud. This is the first Spark user group meetup. It's huge. Uh, we passed the mark of 100 members in 10 days, which is a record like it's it's very quick by Meetup standards
We got a kudos email from headquarters saying you do something right. Uh so this is all of course the achievement of day and his group which made spark so interesting. uh question uh who among the president was here for the first meet up devoted to Sparkish we held in December uh at scalful startups a few okay so this is basically a new group right for most of just to sort of get some stat like how many of you guys actually try spark cool all right so basically just a few sort of logistic uh points Uh so this is the uh inaugural meeting. We have it here at cloud. We're very interested in using spark. We for a bunch of people uh hundreds of millions of users uh basically for uh influence online. It's called descendant influence. We're very interested making this process faster
So that's how we actually you know uh found spark and found it interesting. Uh so uh this is a sparker group. We actually other things. So I was going to suggest to mate that we may come on the future with science part to other frameworks such as scubby scrunch scalding right so there are other interesting uh big data frameworks based on scala also there's cases log based on closure I sort of put them all in the same league but nothing actually exists comparable to spark in terms of memory right so it's it's a memory caching uh system which makes it all uh very fast and fascinating. So that's I think what we're all here for. So question uh and welcome again. [applause] So let me do this. Yeah, it's on
Right. Okay. So thanks a lot for coming to the first Spark user meetup. It's really exciting to see this many people here. when we set up the meeting, we thought we would get, you know, maybe 20 people, 30 people, something like that. And it's really cool to see all you guys. And uh thanks to clout for hosting this um and providing, you know, uh drinks and food. Um so actually before we get started, I was just curious if everyone would would like to introduce themselves just so we kind of know who we are and where we're coming from
Uh so I'm mate, I'm a grad student at Berkeley. If you just want to say your name and maybe company that that would be great. I'm Mark Hamstra. Some of you may have seen my presentation on Spark a couple of months ago for the power enthusiast. >> I'm from Deadlines. >> Um a general request. So we don't have the speaker mic at this point. Uh so please stand up and speak louder >> and you know face the rest of the audience and try to speak as loud as possible
>> Thanks Michael your power. How are you? >> David, I work here at Matt. I work at home. Name's Josh. Tony work at Viso. I'm George. >> I'm Ariel, a PhD student at Berkeley. >> I'm Andy, also PhD student at Berkeley
I'm Ben. I got to micron >> I'm a grad student in Berkeley. I am one of the developers. >> I'm Ron and I just I'm a master student at Berkeley. >> I'm Antonio. I'm a master student. >> I'm a second year student under work. >> [laughter] >> Looking at the grad students
I'm Dave. I'm a master student at Berkeley. >> Patrick. >> Okay. Well, Okay, thanks. Yeah, so just wanted to get an idea of of who is here. Um, so what are we going to do at this meetup? So we saw a lot of interest in Spark and uh we have a bunch of people that are starting to use it already and we wanted to make it easier for people uh to use and to hear about what you can do with it. Um, so I'm going to cover a little bit here about uh the history
I just have a couple of slides and then I'm going to to dive into our tutorial which is really when it gets down to it how do you use spark what kind of things can you do with it u and I'll be showing you some stuff running on EC2 as well doing that um and then after that uh we have a second talk from quantify about how they are using spark for predictive analytics and I'm looking forward to hearing that too um so just to give you some background about who we are here why we're doing this stuff um we're part of this lab at Berkeley called the AMP lab. It stands for algorithms, machines, and people. It's a fairly new lab about a year and a half old that's all about attacking big data and it's by scaling up these things, algorithms, crowd sourcing, uh, cloud computing and so on. Um, and we're building a software stack that includes a few layers and we've actually worked in the past on other things also like Hadoop. So the the reason we are trying to do this and to open source this is because we've had some success getting our algorithms and our code into Hadoop. Um so one of the things we're building is MSOS uh which is the cluster management system that makes it easy to write um and deploy distributed apps and actually this is a thing that's already starting to to make the transfer uh to be widely used. So Twitter is a major user of this and also a major contributor which is really cool. Um and Spark is is our computing framework
So basically the the goals of Spark are to give us this general and efficient uh programming model um that will let us do a bunch of applications on top of it. And one of the things that makes it efficient which you've probably heard about is the support for in-memory computing. So whenever you have an algorithm that needs to share data across multiple steps, you can often make that happen 20 30 times faster using Spark. And essentially it's you know for some algorithms you'd be crazy to use anything else if because it would be going so much slower than than your real machine can. Um the other things we provide are this highle API in the Scala language makes it easy to program and we're using Spark as a substrate to build upper level tools. So we have for example a SQL on Spark. We have a part of Apache Hive called Shark that that compiles down to Spark and we have an implementation of uh Google's paragle model as well. So this is kind of what the stack looks like today
Um so the messos layer is what provides our cluster management and um it it actually lets you run either over Amazon EC2 or over private clusters and on top of that we have spark which is the big box here because we're talking about it but we can also run other things like Hadoop and MPI. Some of the things we're working on are building on top of the spark layer. So we have bagel which is a implementation of bagel the graph computing model on spark and this is actually a very small amount of code you have to do to get that API on spark and ankor who's an undergrad wrote it and you know it would be great to say he's a typical undergrad but I don't think he is but it's still you know it's a it's still something we were able to do. Um something um else we're building that's more involved is Shark the port of Apache Hive on Spark. And this is a thing that's actually more or less um working at this point. Uh we're probably going to release it around the end of the month. That's the plan. And I think at the next user meetup we'll actually talk about this too
But this is kind of cool because it's completely compatible with Apache Hive. It's just the same code but instead of compiling down to map produce it compiles down to spark and uh other things we're doing uh one of the things we're working on next is also adding stream processing here so a few of the grad students who are here are working on that and in addition to that we also want to do things like machine learning libraries matrix algebra stuff like that all these things will compile down to Spark and benefit from the the scheduling and the efficient um runtime that it provides um and finally Apart from this stuff that's above it, we're doing some improvements in Spark itself, one that maybe will be interesting to users is a bunch of debugging tools that were we're developing that will make it really easy to look at what happened in your Spark job after and uh sort of rebuild any of the intermediate data and and explore it. Okay, so that's kind of by the way if there are any questions about this stuff, feel free to to ask me. Um okay so that's kind of where we're going and u on the way there u we're quite fortunate to have a bunch of people uh working with us to to actually improve spark and to use it as well. So here are some of the companies I know about u and companies and universities. Uh there may be other people as well that I don't know about because I get questions on the mailing list or even pull requests from people that I haven't met. But it's really awesome uh to see all these guys using it and you'll hopefully hear from some of them um in in these meetups. Okay, so let's get just down to how to actually use Spark and I want to make this, you know, kind of a walkthrough and something you could you could do after
Um so this is how you would download Spark. I'm just going to show that happening. And I also want to add um if you want to follow along with some of the examples, we'll be using some Wikipedia data, you can download it here. This is a 2.3 megabyte file, so it's not huge. Uh but I figured that having a lot of people download this over Wi-Fi in one location may not work. Um and this these are the slides for the talk. So if you actually want to see what was on the slides, you can grab them over there. Um okay
Yeah. So, so just to install Spark, it's on GitHub. You have to clone it and then we use the SBT build system to build it. Um, so that's, you know, it's probably kind of an easy thing to show. Actually, there's probably no point in me showing that. Um, you can believe that it works. Um, but yeah, this is what you do. Um, so, so one thing I should add about Spark is if you want to run it locally on your laptop, this is all you need
You don't need to install any of the cluster stuff, uh, like the Mesa stuff. you you just need to do this and you need a JVM basically. Okay. Um so on my um so next thing we can do is run some jobs. Um so there are some example jobs that come with Spark and there's a script called run that will run them. There's also the interactive interpreter where you can use Spark from the console that's called Spark shell. Um so what I'm going to do here is log into a EC2 machine where I've set up Spark and just show these things working. Um, okay
So, I'm going to make this a little bit bigger. Can you guys see that? Maybe I'll move it up a bit. Let me know if this is too small or anything like that. Um, so what I'm doing here, um, Spark has these scripts that make it easy to launch on EC2 via MSOS. These are provided in the MSOS project. So, that's what I did there. I already had set up a cluster called demo and I decided to log into the cluster. Um, and uh, let's just open a screen here
So, I have a bunch of stuff here. Uh, if you see what's what's installed in my um, home directory here, I have MSOS installed. I have actually HDFS that uses the ephemeral storage on the Amazon discs. There's an HDFS that also stays on the machines if you stop them and restart them later. And the Scola and the Spark, of course. Okay, so here this is the the sort of it's the kind of thing you'd get if you just get clone spark. Um and um let's just run first one of the examples. Okay, so this is one that computes pi
All the examples have this usage stuff. So you can do for example local and this is it's um so this is you know very poor approximation of pi but this is how you can run the examples. If you want to see them, they're in source and um they're in examples in the examples folder here. Um as a more interesting thing, let's do the the Spark shell. Um so actually let's do so I'm going to launch the the Spark shell and I'm going to connect to this uh Mesos cluster that I have set up on EC2. So this is how you do it. you have to set the master environment variable and then you get this askart and eventually um you get into the shell. Okay, so there it is
Um so this is this is one of the easiest ways to actually learn Spark or play with it is to use this thing. Um so what what kind of things can you do in here? You can type in any Scola stuff you want. um any Scala expression and you also have this variable SC that's the spark context that lets you run stuff on the cluster. Um so just going back to the slides here um there's this I have a slide here on how to launch this on EC2. Um I think you can look at this after the talk also but basically the idea is we have this script where you just tell it how many machines you want and you give your cluster a name and you can launch it. Um and later you can do stuff like pause the cluster with stop or just destroy it or all that kind of stuff. Um let's look a little bit at um the kind of programming concepts. Um and then we'll go back and actually show some of them
Okay. So in Spark there are sort of three things you want to um think about. Um one of them is the Spark context object I just showed. This is the main thing you use to launch um functions to to just do stuff in parallel on the cluster. And I'll show how you can create that in the shell. You just get one created for you. Um then the main thing you work with are called resilient distributed data sets or RDDs. That's kind of a mouthful
We just wanted some unique name, but you can think of them as just parallel collections. Same as your lists and arrays and things like that in your normal Scala program. Okay? And these things are distributed. So each one different slices of it are in different machines. And you can also choose to to cache them in memory. So if you cache them in memory, then next time you try to use them, you'll just have Java objects sitting in memory. It'll be very fast. If you don't cache them in memory, they actually get recomputed each time you use them
So it does a bunch of work again. Um, so so that's how that works. And they're also immutable. You once you create one, you can't go and change stuff in the middle. Okay. And then finally you can do these operations on them. And there are two types of operations. There are transformations which give you back a new data set by transforming an old one
And there are actions which give you um which actually output a result. Um so they may either return the result to your main program. For example, you count the data set and you get back an integer. Okay? Or they may write it to an external um system like you save it to HDFS. Um, and the the reason these are different is because transformations all happen lazily and basically as soon as you do one of the actions, Spark will figure out a way to plan all the existing transformations to to compute that thing. Okay, so um yeah, so here's how you would create the Spark context yourself. Um, if you saw in the shell, I had one already. If you want to create your own, you basically provide these two things
um you provide a string for the the master of the cluster you're running on. Um so this can be either local if you're running in local mode or it can be this kind of MSOS URL if you want to launch on a cluster managed by MSOS including the one that you get on EC2. Okay. And you have to give a job name. This is mostly so that the name shows up in the web UI for MSOS. So when you look at the cluster you see which jobs are running. Okay. Um and then here are different ways of of creating these data sets um and then transforming them
Um so there's really data sets can come from two places. Um one of them is you can take an existing scholar collection and turn it into a parallel data set and the other one is you can read files from any storage system that um is supported by Hadoop. Basically we use the same input um routines as Hadoop. So you can read from the local file system, Hadoop file system, S3, hyper table, HB, all these kinds of things. Um, so the easiest one of these to use is text file. You just you read a file and you get it back as a bunch of lines. Um, if you want to use your own Hadoop input format class, you can use this more complicated version down here. Okay
Um, so so let's go and do that in here. Um and then I'll show some some of the operations you can do on it too. Um so the easiest way to play with this stuff is to just use parallelize to make uh to make an existing collection parallel. So let's say I had this list of 1 2 3 uh and then I want to make it parallel. I can do this and then I can do stuff like for example uh I'm going to map each number to you know um say the square of that number. Okay. Uh okay. So what you see here is this defined a new RDD containing integers
It hasn't actually computed it yet. So as I said all these things are lazy. And then if we want to to compute it we'll have to do one of these actions on it. So the easiest uh action is called collect. This takes the elements of the distributed data set and gives them back to you in a local array. Um so you can see this is this is actually yeah there we go. This actually ran this on the cluster which is kind of silly for a three element list but but that's what we got. >> Yeah
>> Can you take multiple sources and treat them as one? >> Yeah, you can you can actually so you can do uh you can do union on these data sets. Um so for example if if I had another data set like uh uh let's say this uh okay I can do like this and this will do yeah this takes a little because it actually launched some new JVMs on the cluster but you can get this yeah >> if you run collect again in the first variable >> recalculate it, right? >> Well, it it it will recalculate it unless you ask the system to cache it. >> So, we haven't done that yet. So, now it's recalculating it. >> Question. >> Yeah. >> If you if you launch a job, does it keep the JVMs alive afterwards once it's launched once or >> Yeah, it it keeps them around. Yeah
The reason you saw it taking a while is because this cluster has like 20 nodes in it and probably only one or two nodes got used. Um, but yeah, normally I don't know, normally it it happens a little bit faster. Yeah, there you go. And in local mode, it's even faster. >> What happens if one of the nodes dies while you're running the job? >> Yeah. So, Spark will tolerate that. It will recover from it. It will give you the same result as if no node had died
And it's actually designed so that even if the data doesn't fit into memory or whatever, you'll still get the same result. Um, it'll just be slower. >> Yeah. So does it do checkpointing for that or >> no it actually um it knows how to recomp compute the lost data. Um so I mean for stuff like say you did a map and you lost an output of that map you can do that operation again on part of that file. >> Yeah I can explain that more uh later also. >> Question what does it use to I guess in the code there were like multiple