1st Spark: Apache Spark talks at Scala for Startups Q&A
Recording: 1st Spark: Apache Spark talks at Scala for Startups Q&A
It's in the Spark. >> Yeah, the data goes into HDFS. So, we have I give a very high-level picture of our infrastructure. The data comes in and there are some components which aggregate the data and, you know, cop sends a copy to a live stack and the rest goes uh is written down into HDFS. And the data for Spark comes out of HDFS. Now, obviously, when we're doing the real-time thing, that may not be the that may not that won't be possible. So, we're thinking about various ways to, you know, you know, reuse some of our other code to do Spark in a live fashion. Yeah
>> How much data do you have going through there? >> Uh we have a few tens of millions of uh streams a day and it's I don't know it's a few like hundreds of gigabytes. Uh come compressed and uh you know, yeah. the sorry, this is this is sorry, this is processed data. It's not the raw data which is coming in from the clients. So, we summarize the data and summaries are about a few hundreds of gigabytes compressed. The raw data is like a few terabytes. >> Per day? >> Uh yeah. Any more questions? >> Yeah, and if you have questions for the rest of us, write them at the mic
>> Yeah, yeah. Right, yeah. So, so, when you call cache on something that doesn't fit in memory, um the system will only keep part of it in memory and you should get performance that's in between having everything in memory and having nothing. So, that's the ideal. Now, to actually get that, you have to tweak um some of the settings a little bit uh to make it actually do that. Um the thing that is memory hungry, like the thing that's probably most surprising to people is if you just use standard data structures in Java, many of them are not really memory efficient. So, if you use say a list, you know, and you put integers in the list, well, first they are java.lang.Integer, which is much bigger than 4 bytes, and then the list itself has these nodes and pointers and stuff like that. So, you can easily have an integer that's consuming 40 bytes or 60 bytes
Um, and that's that's the kind of thing that that people have to optimize to to deal with this. Yeah. Um, it is So, customizing the cache It's sort of possible if you write your own cache. Right now, we just do LRU, but you can write your own cache uh subclass. Yeah. >> How do you deal with resource sharing? >> Resource sharing, yes. So, right now, you can run um you can run Mesos on a cluster and have multiple people using Spark. They'll be running different instances of Spark, and they'll each have their own separate JVMs and their own separate memory pool
Um, so, we can't right now, we can't actually share the memory pool between two JVMs. Um, we want to do something more clever with that, where you can share data sets between users, but it's it's not that easy to do it. But, yeah. All right. How much configure? Yes, you can definitely customize GC settings. You have a config file where you can set Java options, JVM options. And you just have one JVM per machine. It's pretty easy to think about
So, yeah. Yeah. Uh-huh. Oh, I see. So, using Redis for in-memory representation. So, I I actually I don't know too much about Redis uh in particular. So I don't know if it has something like a map reduce or a query language. But do you mean using it as a storage system essentially? All right
Yeah. So using it as a storage system, the main disadvantage with using an external storage system is that you'll have the cost of talking to it. So you'll have, you know, at the very least a memory copy, but probably more than that that's required to get data in and out of it. And in some applications it's okay. In some applications that's actually more expensive than the query you're doing. So that's why that's why, for example, we don't use Memcached or stuff like that. But the advantage of using Redis would be that you can have you can easily share data sets among JVMs and stuff like that. So you you could, if you wanted, you could add support for Redis as a data store for the data sets in Spark
It's actually not very hard to do that. Yeah. Yes. Yeah, so there's a local mode. So if you just download Spark just by itself, you can run it on one machine maybe with multi-threading also just out of the box. And it just runs in what the one Java process. Yeah. Are there questions? Yes
All right. Yeah, that's a really good question. So the goal, especially from a research point of view, is to make it figure that out by itself and to do a cost-benefit analysis of which tables and which intermediate results I keep around. That's actually pretty hard to do. So in the first version, one of the tools we'll have is essentially like a create in-memory table where you can create your own table and tell it to keep it in memory. And then you can explicitly have that stuff there. Yeah. All right
>> So, I don't think I have a lot of experience with Spark 6 months ago. >> Uh-huh. >> So, when it actually started with the inception of Spark, how stable was it then? >> Uh when the project started. Yeah, so I think the project started maybe a year and a half ago and um it really started as sort of a demo that you can do something with Mesos that's a lot faster than Hadoop. And back then this Spark like to handle logistic regression example was about 800 lines of code for Spark itself. Uh now it's grown a bit more. Um but um yeah, and we we've had it on GitHub I think for about a year and we decided to start putting, you know, numbered releases and putting up a lot of documentation and stuff. Um in in May
Um and in terms of stability that it's actually reasonably stable at least across the people that have used it, you know, some issues do come up like the memory stuff or serialization, but we've mostly seen them. And so I hope it remains that way especially with the stable releases. Like if you get the 0.3 release now, the latest one, then it's it's okay. Yeah. >> If you tell us the scenario where you have some computation even though you have a lot of data. >> Yeah. Yeah. >> So, this is This is Monday morning in New York City
Can you start over this? >> Yes, you can, yeah. So, you can call you can call save. There's a save operation that saves a data set to a Hadoop file or to HDFS or whatever storage system. Yeah. >> Is there some kind of automatic mechanism to restart from >> Oh yeah, no. So, yeah, we we don't have that. So, the the thing So, we don't automatically checkpoint and we also don't automatically restore the state of your main program, the one where you have your for loop or whatever doing the iterations. So, you'd have to write that yourself um to have it save once in a while, yeah
Yeah. Okay. >> Is it possible So, yeah. To interface with R. Yeah, that's that's a really good question. So, I think you can So, R has Java bindings. So, I think if you use those, you could do some stuff with R. But, we don't have anything to actually compile R down to Scala and down to Spark
That that would be really cool. I know a lot of people have asked about that. But, we we don't have anything like that. Yeah. the road map. So, yeah, that that's a good question. So, I think um the main thing that will happen in the near future, probably in the next month, is I want to put out a new release because since the last release, there's been a new Scala version, uh 2.9.1, and uh Mesos that we're on has changed significantly in terms of API. So, I'm going to put that That will be more, you know, to to keep people up to date with with what's going on
Um So, some of the things that are going in There's at least There's a number of optimizations that I know um can be done and that I I'd like to do and other people are also uh helping out with. Um Another thing that's missing, but that we will hopefully get help on, is um uh basically operators that can spill out to disk if your data doesn't fit in memory. So, say you're doing something like a join or a group by. Right now, we just do it with a hash table. So, if that doesn't fit in memory, you know, that's that's kind of too bad. Um you can do an external join where you write stuff to disk. And uh there are some folks uh from Yahoo research actually that that may or may not help with that. So, no promises
But, if they do it, this like they they know how to implement this stuff pretty well. Um and uh so so some other things we're doing For example, Ankur is working on a debugging tool that will be pretty nice that will let you sort of see what happened in the job and and replay any portion of it. Um so so we're doing stuff like that. Yeah, I think a lot of it is going to be driven by what people ask for, too. >> [applause] >> So, just want to thank everybody for coming. Uh we're very happy to have you here at Cloud. If you have any questions about Cloud, uh feel free to ask me or any of our folks around. And uh uh send us uh suggestions for the next talks
And probably the next meet-up will be in January. So, uh we'll announce that and we'll also send announcement about the Spark User Group, which will physically gather, I think, also on a monthly schedule here at Cloud. >> Mhm. >> I