1st Spark: Matei Zaharia gives the first talk about Apache Spark, hosted by Alexy Khrabrov
ash and you know it's going along and it will eventually hopefully give the same result. Um >> how big is the Wikipedia? >> Oh yeah, I didn't say that. So it's it's about 50 gigabytes the the plain text file. Um and then we're pulling and 120 nodes. So it's 2.5.0. Yeah. Yeah. So this so so this took again about 20 seconds and it's similar performance to what you'd get with Hadoop
Um but now the data is in memory. So so now if we run this again on the in-memory data you get the answer back you know in less than a second. Um and this is again this is really the advantage of being within memory data. And now since the data is in memory we can ask all kinds of other questions about them as we want. So the question I always ask because I always know the answer is this one. So you can guess. So Berkeley was 13,000. How many will Stanford have? Oh, too bad
[laughter] And yeah, we can also do, you know, let's just be fair to East Coast brethren. Oh man. Okay. So this is Yeah, this is the place to be. Um and just just to finish this up, um let's just see. So all the articles about Berkeley must be about the amazing research that's going on there, right? Uh so let's just grab the titles of all the articles that mention Berkeley. So we can do that with the collect operation. Gives you back a data set as an array
And let's just look at some of the titles. Okay, so here they are. So where is some of the big data stuff? Um so we have exabytes. That's good. We have history of the hippie movement. uh we have list of free free software lawsuits. So and we have there actually I don't know if you can see it but there also a lot of people name Berkeley so it's not it's not all about the university but but this is some of the stuff you can do um using this. Okay
So so that's essentially it for uh my part of the talk. Last thing I want to do before we switch to someone else is just to say that this is open source. It's BSD licensed. Uh we actually we started putting out releases in May and we have a pretty active user uh list on on Google groups and we encourage you to try it out. Um and next you'll hear from Tim who's been using this for machine learning. YEAH. [applause] I don't know. >> So maybe we'll while he said we can answer some questions
Yeah. Exactly. Questions. >> Yeah. Okay, cool. Do you >> you don't need the dongle here. >> I may come back cuz any questions? Yeah, I'll just do this. >> Oh, you have this
>> You can put this. >> Yeah. >> This it can just go in the pocket anywhere. >> The question was >> you can put in the pocket. >> We'll have this on the next generation. So Hadoop next generation also has functionality to let you run multiple computing frameworks on the same cluster. And the way we're going to do that is that we'll actually grab some nodes from Hadoop nextG and launch MSOS on it and have Spark run in there. And there is a the technical reason for that is that Hadoop NextG isn't really designed for small tasks
In the demo I did, you had 300 tasks finishing in, you know, 08 seconds. So each task is a few hundred milliseconds. And it's just the way that it's architected, you can't really have tiny tests, but but you'll be able to run it on NextG. Yeah. >> Um, okay. May maybe one more question and we can take I can take more questions after because we just wanted to have all these folks present. Yeah. >> Just wonder just [clears throat] wondering if shark is expected
>> When shark is expected, right? So yeah, it's a so you know if you ask the professors who are very excited about it, they say like January or February, I want to be nice to the people that are, you know, working on it and say maybe a little later than that, but we do want to put out an alpha in the next few months. >> Yeah, the end of February. >> Yeah, maybe. Yeah, cool. >> Okay, so thanks Mday. >> Good evening everyone. My name is Timothy Hunter and I am a third year PhD student at Berkeley in machine learning. So I am using the um Can you hear me? Okay
>> This is actually for just for the camera. >> Yeah. >> All right. Okay. So and I I have been one of the early adopters and of spark and a big source of email of complaints and praise to to mate. So I'm going to present how we use uh spark for doing big scale machine learning. So I am working with a project in civil engineering called mobile millennium and this project aims at giving you what Google also wants you to to give you which is real-time traffic information across the the whole city. So for this we use some taxi cabs uh from certain cities
Can you tell me which city it is? Yeah, it's it's fairly easy to uh to to Yeah, this is San Francisco and this is from one of the taxi cab companion we're running in San Francisco. So, we get the positions of the taxi caps every minute and from this we want to infer eventually what the traffic looks like across the whole road network of San Francisco. So, not only the highways and not only the big the big streets. So one of the So that makes about half a million observations a day and we hope to move to one of magnitude more later in in the year maybe. So one of our big problem with this kind of observation is that it is extremely sparse and it doesn't give us the um the travel time on each of the links of the road network. We do not know the time on each street rather what we know the travel time between each of the observation that we do for these for these taxis. So we have to reconstruct somehow the uh travel time between each of the um on each road link taken by the by the taxi and there is one way to do this in machine learning. It's an algorithm called expectation maximizations
So in a nutshell it's assume that you already have an idea of the travel times on every road on every road of your road network and it aortions this time this one minute between every sample on every road link for example it will say 30 second on this road link then 20 second on this road link and 10 second on the next road link and then for every road link it will aggregate every every sample it has and compute the mean and variance and the nice proced the nice idea about this procedure is that it is iterative. So you can repeat the same cycle again and it will converge towards some uh some equilibrium distribution of travel time on your road network. So from a from a system perspective how does it work? Um so we have all these observations which are the big blue circles. So we have about 50 million to a billion of them. And for each of these we use um they are mapped somehow to the road network which are the the links the green circles down there. And for every observation we generate some travel time samples corresponding to the road links corresponding to each observation and we we aggregate them then on a link by basis. We use them to learn some new parameters on the road network and we send them back to the observations. So you can think of it as an iterative map reduce task and this is where spark comes into action
So this is the actual this is a snippet of code that we use to uh to write the algorithm. As you can see it's very concise. First we load all the observation and we cache them across the machines. Then we initialize our parameters which are travel time on every link across the road network and then we simply run a loop until convergence of the algorithm. Can you see what is written? It's so the the first step is to from is from each observation to generate some uh some samples and then you these samples regroup them by u by link ID by road link ID and then collect them when we when we compute some new parameters. So here you see really the all the functionality of the all the functionality of spark in action which is reading reading caching mapping and then doing grouping by key and collecting. So you when you when you see this algorithm you can imagine why it is so important to go to have a very good iterative procedure that works only in memory. So we ran these experiments on um on a cluster first in EC2 and also in um at the Lawrence Berkeley National Lab because they have a big dedicated cluster for running a big map reduce job and it was extremely nice to be able to use Spark in this context because we debugged it on a single machine with a small data set and then we run it on our large data set which is about 10 GB of data of row data But you have to imagine that when you generate some samples, you have 50 times to 100 times more data being generated in the process
And this data is discarded at the end of the iteration. So that makes up to a terabyte of intermediate data that that needs to be created, shuffled, and then discarded. So when we run it on um on four machines, it would test it was taking more than an hour and a half. And we simply changed the number of machines to 160 and it was running in um a bit more than 6 minutes at the end simply by changing the number of machines. So that we were very pleased in in the way we could use it. So there are a couple of uh aspect that mate did not mention uh and there are a couple of things you have to be aware of when you use it. Mostly it's it's a Java program. It runs on GVM
So you have to be to make sure that you have a very liberal policy towards memory usage. It will use a lot of memory. Um the the other some other aspects is when you use these closures to pass around your task. These closures will actually aggregate a lot of data um underneath and sometime you don't realize that they're becoming huge and they can become the bottleneck when spreading the computations. So fortunately we mate has implemented some uh some mechanism to cache and distribute some u well it's not caching it's really broadcasting some some variables that are common to all these closures and using this we can um so using this broadcasting mechanism which um you can actually go also much faster when you when you run the jobs because the closures that you send are also much smaller. So using this we can also cut the time the the processing time by more than uh by more than four. Okay. So as a conclusion um so running big machine learning on on the cloud was um was I think a hard task until a few years ago and we were very very happy to use a spark as a as a programming framework
It really helped us to go from our Java based framework to some u to to the cloud with really minimal minimal effort which is very nice to uh when you are in a research environment. Okay, thank you for your attention. I think the conviva people will also present their application. >> Okay. [applause] >> So maybe while uh they're sitting up you can answer any questions you guys have. >> Yes, please. What other what other alternatives are there running? >> So sorry, can you repeat your question? >> Yeah. What are your what are your options? I mean are there is there anything else out there that >> So we we could have we tried map reduce on with Python the with the Python binding at some point and it was running too slow more than three times slower than spark when we tried it
>> It's got to persist between So yeah, there's this whole problem of persisting the data in memory. Here it makes sense to actually persist the observation in memory to cut the loading time and saving time at each iteration and that was a big bottleneck in >> there aren't any other frameworks that can do that. So as far as I can tell no. Uh probably Mate has a more comprehensive answer but that was and also since we are part of the same lab one of the reason we were trying to uh to promote the use and also that was one of the that was one of the first use case that we could see for joining the the two groups. So we were we requested a number of features to spark and mate gave us some uh some ideas about what we could do with it. >> Yeah. Yeah. So really the other thing people use is MBI and MBI is very low but it's a lower level but it doesn't provide it depends you know if you want to get one thing I think it's very helpful so the group itself so you know if I'm running but it's a new system somebody will always ask I think you post a >> [clears throat] >> Okay
Okay. Hi everybody. Uh my name is Dip. Uh I'm a software engineer at Conviva and Spark has been like extremely useful to us in analyzing the large amounts of data we collect. So I'll be talking about how useful Spark has been. So first, oh oops. So just a word about what Conviva is. Uh, Conviva is an uh, Conviva monitors and analyzes and also optimizes online video for a lot of premium brands like HBO
If you go to HBO, we are on there. If you go to ABC, we are on there. If you go to ESPN, we're on there. So, we help all these premium brands analyze how their online video is doing. And if it's doing badly, we help it automatically optimize your delivery. So, the basic goal is to get the consumer as good an online video experience as possible. So, we see tens of millions of streams every day. We collect a lot of data and we use spark to analyze it
Before I talk about spark, let me give you an overall picture of what the uh conviva data processing infrastructure looks like at a very high level. So if you go to say abc.com and you watch a video, there is a bit of conviva code which sits in your video player and is sending back analytics data every a couple of times a minute. Now this data goes into our data center and it gets split into two. The first part goes into a live processing infrastructure which updates a dashboard every two seconds and this is from each and every client who's watching video. The other part of the data or actually a copy of the data goes into Hadoop for persistent storage. And then we use Hive which mate mentioned uh Hive provides a SQL like interface on top of data stored in Hadoop. So we use Hive to produce Excel and PDF reports about the data. So we do a lot of analytics about what the video is for, how people are watching it, what the quality are they're getting and a whole bunch of stuff
So we use we were using Hive to generate all these Excel and PDF reports and Hive takes a lot of time because it reads data from Hadoop each and every time and it takes a very long amount of time. That's when about maybe seven or eight months ago uh we got to use uh we got to use Spark and Mate came and showed us the same example. Stanford was with Berkeley and and we were all sold and we started using Spark since about like March March or April this year and so so one of the main things we use Spark or Hive for is to produce uh prepackaged reports in PDF or Excel format. We also use it for a lot of ad hoc analysis just like me showed on the Wikipedia data. We do the same kind of stuff on video data. Uh so so how can Viva uses Spark? As I mentioned, we are a video we collect a lot of video analytics. So the main query we do is we take a lot of video data and then we slice and dice it based on different dimensions. So for example, we slice it based on the country
How are people doing in in in USA versus Canada? How are people doing when they're watching different kinds of videos? So you're watching say lost versus the bachelorette. What are the characteristics of people who do that? So then we slice and dice based on the ASN, the CDN and a whole bunch of stuff. In all these cases, it's the same piece of data and we're just slicing it on different dimensions. Now when we're doing it from hive, it keeps scanning the same data again and again and again and it makes it very very slow. And the best part about spark is that it keeps all the data the data we care about in memory and we run the same queries over it and it comes back the answers come back very very quickly. So we had a report which used to take about 24 hours to run on a week of data and it was slowing down our cluster. Nobody else could get any work done when that job was running. And now we use Spark and it runs in about 45 minutes
And that means of course we can produce reports faster. Other engineers can you know do some useful work and it's been it's been really great. So right now about 30% of the reports we produce are based on Spark. The rest 70% are on Hive. And the reason they're on Hive is that we haven't gotten time to port them to Spark yet. That's the biggest reason. Uh so these were about prepackaged reports. And as I mentioned a little bit earlier, there are couple of other projects on the way
We plan to do some, you know, use more of the fancier features of Spark to do some machine learning, anomaly detection and things like that. So those are works in progress. Uh we also and want to build out a full-fledged interactive console to do like uh you know interactively analyze video quality issues. Often a customer calls up and says I'm having a problem with my video. That's that's what they say. Now it's up to us to find out what's going on. And so having something like spark where you can interactively you know dig deeper into various issues is a very big is a very very useful thing. The other thing we are planning to do is or we hoping to do in the near future is to use spark in a near realtime fashion and there is some work happening at Berkeley which is which will hopefully help us with that
Hopefully they'll develop the whole thing and just give it to us. That's the best part. So so we're planning to use Spark. is already almost real time like a couple of seconds or maybe tens of seconds for our kind of data and say when you're analyzing a week of data but we want to analyze it faster. We want to analyze 10 seconds of data so that we can take a decision based on any problems we see. So we want to do that kind of realtime analysis also hopefully using uh spark. So what are the benefits and challenges of using spark? So the benefit the obvious benefit is speed and it has really saved us a lot of time. It has enabled us to run a lot more reports just because we had Spark
Uh the side benefit was that we could write our reports in Scala. Previously we were writing reports. We used to write hive queries and package them inside uh Python code and often it happens that the hive query runs for a day and the end of it there will be a typo in the Python thing which completely crashes your report and it's a really frustrating feeling and we tied uh you know static checking tools but they didn't really work well because we were not using Python we were using Jyon and that didn't really work well. So it's really a great productivity improvement to write your reports in Scala can use autocomplete you can have type safety everything it's it's really useful some of the challenges uh we faced while using uh spark and misos is that we're always on the bleeding edge so you know if when we installed it the first time mate came and did the whole thing for us so it worked perfectly when I try to do it myself and we try to do it ourselves there's some dependencies to get working but that is pretty much the only challenge and once we have uh I think mate's He said he's going to release have more releases of Spark and that is hopefully going to you know the dependency problem will go away at that point. The other challenge I would say was learning Scala. We were first time Scala you know users at that point and we're just getting more and more familiar with the fancier constructs in Scala. So uh so as a conclude uh like Spark has been great for us as I mentioned 30% of our reports uh use Spark and we couldn't have done a lot of reports without Spark. So it's a great it works great
So you should try it out if you haven't already. Great. Thank you. [applause] Any questions? >> I'm sorry. >> What are the types of queries [snorts] you want to do? uh very very similar kind of queries like you you take a you take all the video streams which are coming in and you want to see if there's a problem in some some particular dimension say a country or a state or a CDN or whatever. So we want to do very very similar kinds of queries to you know make those kind of decisions. The thing is that our queries are pretty simple. We just need to do them over large amounts of data and very quickly
Now the metrics we calculate some of them are really really complicated and takes a lot of tweaking to you know we can calculate the data but and we can analy we can scan all scan over all the data but the question of what to calculate that is sometimes tricky but it's not it's not very very complicated but the general type of query is you take a piece of data slice it along different dimensions or multiple dimensions. And the idea is that someone would then be able to act and say, "Oh, all video is not working." >> Exactly. That is some of the kind of things we want to see if and a related project is anomaly detection like can we detect you know it's not working properly is I mean the question is how do you detect it's not working properly? Is it a threshold? Is it automatically you learn it? So all those questions are still open. >> Yeah. >> So this real time analysis how do you get the data in? Do you deposit it in HDFS like all new chunks of data right coming in? How do you get it?