1st Spark: Matei Zaharia answers questions about Spark at the first Apache Spark meetup
Recording: 1st Spark: Matei Zaharia answers questions about Spark at the first Apache Spark meetup
protocols like guitar and so forth. >> Yeah. Which means doesn't use by default. >> Okay. So the broadcast is kind of um the broadcast is is like a slightly more advanced thing. It's useful when you have a job where um you need say you have like a large parameter vector you're learning and you need to send it to all the nodes. Turns out just copying it from the master to each node is very slow. Um and so we have a bit torenbased broadcast and other things like that
Yeah, I wasn't going to talk about that, but um that's a actually that's something we might talk about at another meeting. That's one of the things we did to make this fast for machine learning. Yeah, >> cool. Okay, so let's let's just do a more interesting file also. Uh so I have here a file in um in HDFS on this cluster. This is a Wikipedia text file. Um and um I think some people may have seen this kind of demo before but but this is this is the text file we'll be using um in this job. Um so this is a file sitting in HDFS and it's about 50 GB uh large
Um so how do you work with a bigger file? Um you can do even if the file is big there are still some things you can do in Spark to explore it interactively. Um, and that's that's one of the things that makes it cool for actually working with this data. So, for example, I'm going to call first here. First returns the first element of a data set. And you saw that was really fast. So, first in this case just opens the file and reads one line. Uh, it doesn't try to do anything fancier than that. Um so this this file here um is a is a tab separated values file and um you can see this is the line that we read
Um and you can see it has sort of five fields. It has an article ID. It has the title of the article. This is maybe the alphabetically first thing. U it has the date when it was modified. It has the XML and you don't see the last field but there's also a plain text field. Actually, if you want, we can do that. Uh um yeah, let's split it and let's look at the last piece
So, this is actionpacked was a funk pack. Okay. So, so this is so you can use this to to sort of play with data interactively. And of course, we can also play with the whole data set by by doing stuff on it in parallel. Um so, let's just show some of the stuff we can do with this. So let's do the first thing um is let's just count how many lines there are or how many articles. Okay. Oops
Yeah, it's not called file. It's called wiki file. Okay. Um so so this is now running on the cluster. Um and this is not using the inmemory data. So it's just um this on disk data. >> Okay. saying managed >> lib managed drivtory
>> Oh. Oh, interesting. I I think um I've seen sometimes that problem if you have uh it helps to delete like your M2 and IV2 um which I sometimes I think there's one library we depend on that the old version in in Maven was wrong and it confuses things but I don't know I haven't maybe it's not the same thing. Yeah. Yeah. Okay. Um yeah. Okay
So here so this was this was working with the on disk data and we did a count and we see you know there were this many articles in this dump of Wikipedia. So >> did it do it on 20 nodes? >> It did it on 20 nodes. >> Yeah it would have been slower on on just one node. Um yeah so so this is so you can work with data this way. Working with data when it's on disk is is slow though. So, one of the cool things you can do in Spark is of course load it into memory. Um, and I'll just do that before going back to show you some of the other things. Um, so let's actually do um so what what we're going to do also as you saw uh as you saw looking at this thing, it's it's not really easy to work with a string like this
So, we're actually also going to define a class um to represent these articles and uh and we'll have the title and text of the article only. And let's do a map function to pull those guys out of the file. Um so, this is going to be this actually going to be a couple of functions here. Uh so, first we're going to take each line of the file and split it by tabs. This is the syntax. you can do for that in in Scala. So this is just shorthand for the for the function like x tox.split. You can also write it that way
Um but if you want to, you know, to write it a bit more concisely, you can use underscore instead of x. Okay. And what we're going to do here, um some of the articles don't actually have the plain text field. Um so we're going to filter to the ones that have exactly five elements. And now we have these arrays of five elements and we're going to do a new article and we'll take field one and field four. Okay, so this now we have a data set of articles. Um the other thing I'm going to do here is tell the system to cache these ones in memory. So the first time it computes them is going to take a while but after that they'll be in memory
And now we can do stuff like say articles that count. uh this is going to be, you know, the ones that didn't have uh whatever the that had exactly five fields. So it'll be a little different. Um so let's just do that. And so you can see now as it's running, it's adding these cache entries. Um I should also show you what this cluster looks like in the meantime. Um so this is kind of Oh, that's taking Yeah. So So this is the Mesos cluster we're on
And you can see I'm running the Spark shell and it has, you know, this much memory. It's using this many CPUs and I have a bunch of nodes. Okay. Yeah. >> What happens when they can't all fit into memory? >> Yeah. When they can't So, what happens when they can't all fit into memory? In that case, we'll keep only part of them in memory and the rest um either we compute them on the fly or we keep them on disk. You can choose which one >> and you can control which policy you start how which >> Yeah. Yeah
Yeah. Right now there's like less control about that. Uh there's one setting you can set once, but the goal is to to let you control how to do this. Yeah. Yeah. >> Do you compress the data in memory? >> No, we don't. But you can choose how you want to represent it. Sorry
The question was do we compress the data in memory? Okay. Um so let's so yeah now now we did this now it should be in memory. So if we do this count again this time you know you get a much faster um result. Um, and the cool thing is, you know, you don't just have to do counts. So, for example, uh, let's say we want to we want to filter them to to count how many articles have something specific. Uh, so yeah. Uh, so for example, we can do this and this was how many contain broke and say we want to see which articles those are. For example, >> do you see how much space it takes to memorize? >> No, right now you can't really say
I mean, you can log into a machine and look at Java, but there's no tool built in. Yeah, that's one of the debugging tools. And so here we're So here we're going to take the the titles of the articles. Oops. Yeah, we should do this. So, um, so these are some of the articles that contain Berkeley. Um, and my favorite one here is actually this history of the hippie movement. Yeah
Okay. So let's let's go back here. Um so this is this is how you run it in the shell. Um there are a whole bunch of other operations you can do in Spark. This is a table with just some of them. Um so basically the thing to take away here is that we want to give you high level operations for common data processing tasks. We don't want you to be stuck with map and reduce and then figuring out how to hack your you know joins and things like that into that. Um and we try to make these efficient
We're we're working on improving some of them. Um so some of the cool ones you will see for working with data interactively sample is pretty cool. Um it's a deterministic sample. So it means if your nodes fail and then you later have to recomputee the data you'll get the same sample and the same results. Um we have joins we have union crossroduct um all this kind of stuff. Um the ones I'll talk more about the ones that do reducing. So these ones are on key value pairs of objects and that's why they're called by key. So like count words by key
Okay. Um for the actions we also have a bunch of different things. So we have the the opposite of loading a text file or loading a Hadoop file. You can save stuff. Um you saw first take is you know take the first five items for example or however many. Um and there's other stuff like reduce is just aggregate all the items. Hey add them all up. Give me one answer
And then you also saw cache which is to keep it in memory. Okay. Um the other thing I wanted to talk about and show is how to run standalone jobs. So running the shell is is cool, you know, makes for uh nice demos and stuff, but you'll probably want to write a standalone job also. Okay. Um so so there are two ways to do that. It depends on whether you want to use Maven. Um if you want to use um if you don't want to use Maven, the easiest way is to package Spark into a jar and then you get this one jar with Spark and all its dependencies that you can add into your project and you can use it however uh you'd like
Um if you want to use Maven, you can use this sbt publish local command uh to publish it to the local Maven uh directory and then you can add a dependency on it. And I'll show I'll show how the standalone job works um in a little bit. Um the other thing you have to do for standalone jobs is the main you know the program main that will be running is the one in your program. So you have to tell Spark a little bit about um how it should find your code and its own code on the cluster. Okay. So you you need to set these two things. There's the Spark home. This is the location where you've installed it on all the nodes
And um there's the spark class path which is basically the code in your jar in your job that you're adding to the class path. Um one way you can use these is to set the environment variables here. The other way if you just want to do it in your program and not mess with environment variables is as extra arguments to this. Um so the nice thing about the second way also is if you package your job into a jar, this will ship the jar automatically to all the nodes. Whereas this way here assumes that the code is in the same path on every machine in the cluster. Okay. So, let's try to do that also. Um, so let's get out of uh this shell
Actually, do we want to do that? No, let's let's leave that open for now. Um, nah, let's just get out. Okay. Um, okay. Um, and uh, >> yeah. >> How do we do class loading? I think your class path comes at the end. Actually, it comes before our stuff. So, yeah
Um, and then there's some other stuff when you're using the shell. Class loading is a little different. If you want your own classes in the shell, you can do it, but it's a little it's a little trickier. Yeah. >> Okay. So, so let's just make one of these standalone jobs. So, here first I'm going to go here and I might have done this already, but let's just build this assembly um job. Okay
Yeah, maybe it did do it. No, I guess it's doing it again. Okay. So, you can see this is this basically takes a whole bunch of drives that we depend on and sticks them into this thing. Oops. Okay. And then we can copy this guy over here. So, now we have this guy
So, so now we can also create our own Scola file. Uh and start doing stuff. Wow. I don't know why it says that, but okay. Um, so the things you you need to do, you need to import the Spark context and import um stuff inside it. The stuff inside it is basically some um conversions that convert between Java types and and Spark types in a nice way. Okay. Um, this is a Scala thing
Uh, Let's just said that. Uh okay. Um so what what we're going to do here um let's just try to do one of the one of the things we did before first and then maybe we'll do a more interesting example. Um depending on how things look. Um so let's see let's say we're going to take two arguments here. Um the clust the master of the cluster is aug zero. Oops. And uh and uh let's say um the the file that we're going to be working on which is the wiki file
Okay. Uh so we can do this and uh again I could pass here. I could pass the spark home location and that stuff. I'm not going to do that. I'll just um set that through the environment instead. Okay. Um, let's actually call this file name. So, we can have this text file here
And for example, let's do uh really boring thing just this um and we have to do the system.exit zero at the end because there are some server threads running. That's actually something I want to eliminate. It's just um an artifact of the current Scola actors library. Okay. Um and now we can do we can just compile it the same way you normally do. Okay. And we got this guy. Okay
Um now in this this way that we're running it, we're going to um we didn't package our code into a jar. So we're going to copy it to all the machines. Um there's a cool thing in the in the Mesos EC2 cluster. Uh there's this copy d function that will copy your whatever directory you give it to all the machines. So that's kind of nice for deploying code. Um so that's what we're going to do here. >> And I'm just copying these guys. Okay
All right. Um, and now we can run uh we can run this guy. Yeah. So, so it's going to complain because we didn't give it arguments. Yeah, it wants argument zero. So, so to actually run it, remember I said you have to set those environment variables also. So, let's do that. Uh, and this cluster is installed in root spark
Oh, sorry. and the spark uh the spark class path will be um basically the current directory. So this is this is where our code will be. And remember I copied this to all the nodes. So now if we run this guy um let's run it with um for example let's actually just run it locally for now um on on itself. Say we want to count the lines in its own file. Okay. So now it did this and it says there are 14 lines in the file
And hopefully that's actually right. Yeah. Okay. And so if you wanted to do it on the cluster, we need that that cluster URL uh that's in here. And then we need the file name. And I'm going to have to get that from in here. So I need this guy, which is kind of long. Okay
So, we can run it in there also. Oh, what happened here? Probably missed something. Oh, yeah. Okay. So, this is actually another um Yeah, sorry. To run it locally, to run it on MSOS, we also need to tell it where it can find MSOS. Uh uh because it's a native library. Yeah
So this is just going to pass an option to the JVM. Okay. So now hopefully it will actually work. Um yeah, there you go. This stuff is running. Okay. And you know it's few nodes are taking a while to finish but hopefully it'll eventually get there. Yeah
Let's just give it a second. Yeah. Any questions about this stuff? How to deploy code? Stuff like that? Okay. How about >> for handling this situation? What do you do when you actually get a cluster that hangs up? >> Yeah. When we get So yeah, um the actually there's a few things you can do. So you can kill if you control C the job you it will kill all the tasks on all the nodes uh cleanly. Um we want to do some stuff in the scheduleuler to deal with this case. I think what's happening in this case is uh HDFS is is timing out um and hitting one file
We can actually see what these guys are doing too. Let's just Yeah, it's still doing that. Interesting. So, see, they're both on this machine. And my guess is this machine has some problem. Um, you can go another thing you can do. Okay, I guess it finished. You could also go and kill the tasks on that machine
That would also work fine. Yeah. So, we're looking to make this this aspect um a little bit nicer. Um, we just Yeah. >> You see progress or nonprogress? Right now you can you can only see this stuff that is being printed here. Yeah. But if you look on the machine you'll see it has a log so you can see what's being printed. Yeah
Actually let me show that also. Let me just turn this. So how do you uh debug it when it's running? Um this stuff is provided by MSOS basically. So we can pick a machine like say this one and we can pick this is the job name and we can look at its output. So this is one way you can debug it. Um you can also Yeah, it didn't pin much to stand it out. Um you can also find these log files on the cluster. >> Yeah, this time it was faster
I think it's just one flaky node in there. Okay. Um cool. So I think um I wanted to make sure we have time for quantify also not to keep you guys too long over here. So let me just um end this part by saying where you can go from here. Um so if you really want to learn Spark, we have a whole bunch of samples you can look at. Um you can look in this documentation page here to find the programming guide that covers what I just talked about. You can find some code examples here
We have uh different machine learning algorithms, page rank, all that kind of stuff. Um and um you can also look at the actual code if if you want to see all the operations. Some of them are sort of more obscure and they're not in any of the examples. So uh let's so I'll take any questions now while uh Cartik sets up from quantifying. >> Okay. >> Yeah. Is Shark already available? It's actually not yet. We're hoping to release the the alpha um at the end of the month
That's the plan. So, if you're cur if you're interested in in Shark, Cliff and Antonio who are back there are the two grad students working on it. Everyone can look at them and bother them or try to recruit them or whatever. Um yeah, >> you can say that we can use Scala on a regular. >> Yes. Yes. Yeah. I think so
That part of the goal was to make it as well the question was can you use Scola the same way you use pig and yeah part of the goal was to give you all those operations. >> Yeah. >> It looks very equivalent. >> Yeah. Yeah. Yeah. And the cool thing what's cool about Scola is you can use any Java code in there just natural. You can call into it and it's uh statically typed
So it means it's actually relatively fast