BDSBTB 2015: Mike Olson, Apache Spark: The Killer App for Scala
Recording: BDSBTB 2015: Mike Olson, Apache Spark: The Killer App for Scala
thank you Alexia and thank you guys for that intro of course it's bull the community and in particular some of the really early researchers in industry and academia pioneered this space in and and you know I should shout out to Martin order ski in the work he did on Scala to the team at Berkeley the work that they have done on spark and all the way back to the early days at Google Jeff Dean and the team they're designing systems aimed at big data problems that were previously just absolutely intractable give me a minute here to set up and I will kick off yep that looks right alright so I want to talk to you today about apache spark the most interesting processing an analytic platform we see in the Big Data ecosystem today and I call it here the killer app for Scala that may be a bit of a reach in fact Scala is used for a huge array of really interesting problems but spark is getting a ton of industry and academic attention right now a very large and vibrant community I want to talk a little bit about that project in the context of Scala why we think it's so exciting and what we think the future holds in order to do that I want to start out kind of orient the group so as you heard i'm one of the founders of cloudera I've been working in Big Data now 4 for 8 or 9 years when Google first invented the concepts behind hadu the two key components they invented with a Google file system really scale-out storage share a bunch of j bods attached to a bunch of 1u rack-mountable servers so large scale distributed robust fault tolerant storage that's a good thing and on top of that a processing framework that allowed you to push work out to where the data was so fragment the data store it in lots of different places when you've got a question to ask rather than suck all the data into one central location you send the work out to all those little frog that was a really powerful and innovative idea it had been tried before but we never seen it succeed at scale and the reason was you couldn't actually buy and assemble large enough clusters at low enough prices to make that work but in the early zeros it turned out that rack mounted storage and rack mounted compute we're getting affordably priced and you could take a whole bunch of older ideas and make them work in that context map produces a way to work with data and one of the early knocks on Hadoop was that MapReduce sucked in a bunch of particular so I'll get to that in a little bit but the failure of imagination at that time by people making that criticism wasn't believing that MapReduce was the way to work on data in parallel and app scale in fact if you think about it you got this big distributed storage system and a whole bunch of processors right nearby all of those disks you could imagine a whole bunch of different analytic and processing frameworks yeah absolutely MapReduce was a good one but there are other ideas that have been in the literature for a long time long time for example massively parallel database systems you could think of ways to paralyze document indexing and search to design parallel machine learning and other frameworks over time that's precisely what happened so when we talk about Hadoop it Cloudera today you know we talked about that unlimited storage layer of course it's an enterprise thing you want to secure you want to govern you want it manageable if the regulator's show up with a demand for information you got to be able to live up to that so you need the kind of data governance infrastructure that large enterprises require but most interestingly across the top there are lots of frameworks that you can now deploy in this world that allow you to work with your data and that's why we think spark is such a big deal a little bit of remedial computer science for those of you in the room though so what is MapReduce well key idea in this framework was as I said rather than bringing the data to the computation you send the computation to the data so you figure out where fragments of data live using the storage systems metadata and then when you've got any animal any analysis any computation any data processing job you want to do you bust it into pieces and you send them out to the servers that have that data locally the idea is that you map the tasks to the storage locations and you do your computation then you combine your results and you reduce those results into a single result set that you can then use as the answer to whatever question was asked and again this is a pretty powerful framework it had not been built and deployed successfully at scale these ideas had been in the research literature for a long time but nobody made it work reliably and at reasonable cost it took advances basically in the server manufacturing sector to make that possible so Google invented MapReduce and used it first of all Google used its massive ingest capability and that scale-out storage system to ingest the entire internet in 2000 it was doing that about twice a week now it doesn't continually of course indexing that data relied on this framework and by the way when you go to google and you run a search you ask a question you get a list results back when you click on the third result in the list you're actually telling Google something very interesting about results 1 and 2 Google would like to continually refine its search results based on that user feedback and so it wanted to automate analysis of its logs as well and they built that on this identical MapReduce framework so it turned out to be a very flexible very powerful way to attack data parallel problems and it turns out a whole bunch of algorithms could be recast in this data parallel way this was a big big deal this was magic that Google possessed that Yahoo and Facebook and the rest of the consumer internet actually feared they worried that Google an unanswerable advantage so when Google published its early papers some of the early web properties jumped on those ideas and created the open source Hadoop project powerful transformative as I said couple of drawbacks right one is a you know it's a new programming framework you got to learn a bunch of new programming abstractions you've got to design to run in this new parallel model oh hey by the way it really helps if you're a hardcore Java programmer because the whole system is written in Java and then there is map reduces dirty little secret its name should not be MapReduce its name should be map shuffle reduce that middle step right there where all the data moves around that is the parking brake on performance in the system the Google built that is forced synchronization nobody gets to go forward before everybody in the map stage has finished because the shuffle stage must reduce most basically shuffle all of the results before you can declare victory if you wanted to do computations that were pretty complex if you wanted to design sort of multi-step analyses every single step blocked until every processing engine was done that is why you hear about these terrible MapReduce latencies you hear about these really long delays these batch mode interactions that people have is because synchronization requires everybody to finish so powerful transformative but had a few drawbacks notwithstanding those drawbacks it turned out that it could be used to attack a huge number of problems over time hive a sequel parser that basically generated MapReduce jobs out the back end Pig a dataflow language machine learning framework called mahou crunch soul or a whole bunch of these new analytic and data exploration tools got built on top of that identical framework I wrote a haiku to tell you that it's a bit like haiku it is a really rigid framework it requires you to recast problems in an exactly MapReduce way that now was standing it was very diverse and very powerful people liked the diversity in the power people didn't like the latency the complexity the programmatic difficulty enter apache spark so this research project initiated at the University of California Berkeley was in some sense a response to MapReduce so we've learned why this framework is really great we've learned why this framework is not really great we've learned what its problems are how do we take those two large lessons and merge them to build a better follow-on system I'm a Berkeley grad so I'm always in favor of Berkeley technology we're a sponsor of the amp lab where this work happened and Matassa haria the student researcher now MIT faculty member and data bricks founder who did this work intern for us in twenty ten and twenty eleven we were paying attention timett a very very early we were excited about it and i'll come back to that story in a little bit we are very pleased to have seen this early research work bear fruit so the code complexity if you're building applications against spark versus MapReduce it's just easier you're like half the code twenty percent of the code required to do an equivalent amount of work and it is an order of magnitude actually that's not even a fair comparison it is just vastly faster than MapReduce remember I told you if you wanted to do MapReduce work you had to basically break your job down into a series of discrete steps and then every step had to run to completion before the next step went you forced synchronization everywhere in your execution graph well spark doesn't force you to do that you can design any sort of directed I sick a cyclic computation executed in spark and the developer who knows where synchronization is required to where it's not inserts Express synchronization points in the algorithm the result is very high performance streaming non-blocking computation that plus another innovation remember what Google built was great storage so good use of disk and great parallel processing so good use of CPUs Google didn't really pay any attention to great use of memory spark absolutely does that allows you to do your analyses using local memory intelligently that collection of innovations a simpler API general-purpose dag execution no forced synchronization points and wise use of memory that's why spark is totally taking the world by storm it is a very good design based on the lessons that we learned developers aren't forced to write only in Java against spark there is high performance there are high performance language bindings for sort of the next generation of very interesting language not now the system is built in spark so I'm sorry in Scala so of course Scala interfaces are easy but the skull interface is very well designed likewise data scientist popular languages Python in particular very well supported you can still build Java applications that talk to your spark infrastructure so you don't have to cash in all of the development that you have done in the past against MapReduce but you've got now a choice of languages for building those applications that did not exist previously and that's translated into interesting new use cases stuff you simply wouldn't have done against MapReduce big innovation is spark streaming and you just heard about the importance of real-time and continuous data ingest and model retraining in order to deliver meaningful results turns out with that non-blocking architecture with the ability to process data in memory at very high speed we see customers use this platform for continual data ingest and even alerting and complex event processing on the fly ETL watching for anomalous behavior doing on the fly model retraining reporting summary metrics as necessary without forcing the system to freeze and do that work so spark streaming takes advantage of the architecture that spark provides in order to run well Scala is the magic that makes that true so first of all very powerful very expressive language it was an excellent choice by the spark team for the framework that they developed inherent advantages for those developers building Scala applications against spark Scala is an interesting language and actually I had a very good dinner conversation last night about why this language is so popular among the most innovative developers that we see in the industry right now and I'm not just sucking up to you guys because you chose to come to this conference functional programming has been a really interesting idea for a long time back in the day in the early 80s I learned Lisp because that was the language that we had the idea of developing functional applications not relying on side effects was really intriguing the ability to take advantage now of parallel ISM in ways that we couldn't previously eliminates some of the performance penalty that we paid back in the day so it's now possible to develop really performance functional algorithms and run them well trick is of course that we need to train a generation or generations of programmers that are used to really procedural languages with variables and assignment and all the rest of that we have to turn that we have to teach them this new way of thinking those who have picked it up have fallen mat crazy mad in love with us so you can read the example here from Barclays these folks are doing very high-performance scalable aggregation of business data across their customers allowing the bank and the user to make much better decisions about card usage about investment decisions about fraud and risk scoring absolutely transformative so a little bit ago I showed you that multi framework picture right started out just Google file system and MapReduce and then over time these other engines crept in where do we see spark happening in this ecosystem first thing to note is when your google when you invented this platform some years ago you had that single execution engine running on top you didn't really need to worry about about resource allocation and about isolation and about managing competing engines that are trying to go after a different awesome memory and disk and so on no longer true when you've got impala for high performance equal paired with MapReduce and also search and mahou and now a variety of workloads running in spark you want to be able to allocate compute storage memory and so on reasonably so you need yarn the new resource management layer that's gotta underlie all of these engines we've seen spark take its place as a meaningful alternative to MapReduce and I'll tell you my belief is that the fraction of new workloads running on MapReduce is going to decrease dramatically now I want to be careful that doesn't mean that MapReduce is going to go away there are many many many very large applications running on MapReduce around the world today I think about mainframes think about COBOL systems like that don't die in fact systems like that are very well designed for a specific class of problems but the advantages I talked about before right performance lack of blocking very low latency and most of all the ease with which developers can use spark mean that it's a much more attractive option we're seeing work that would have gone into MapReduce applications now being done on spark point of fact I talked about a number of engines run on sparks a on MapReduce crunch and hive and pig the community is porting all of those now to run on spark instead so those higher-level languages running on that general purpose MapReduce framework are going to perform much better and deliver much better results because they're being ported to this new framework we've got lots of customers I'll talk in just a minute about where we see adoption happening embracing this hybrid infrastructure when we talk about Hadoop today this is actually what we mean not MapReduce in HDFS but a collection of processing framework search Impala MapReduce Park and friends running on a shared resource management framework with a few different storage alternatives underneath we buttress it with the kind of manageability and data governance that I talked about but this platform looks very little like what Yahoo was running in 2008 and that's not surprising big data matters to a whole bunch of different industries and customers and it has evolved to take on workloads that would previously have been impossible so let me give you a quick overview of how we progress through this through this pathway so I told you that we began paying attention to spark when matei was doing his early work still as a Berkeley doctoral student in 2013 we actually recognized the utility of spark in the context of MapReduce and I posted a pretty controversial blog post on the Clara blog basically asserting the spark would replace MapReduce as the general-purpose processing engine for new workloads in the Hadoop ecosystem at the time a lot of people screamed and yelled in facts now IBM i guess is announced that they're going to go hire a few thousand spark developers and train a million spark professionals data bricks has launched and gotten funded substantially Hortonworks map are have all embrace park as well III will claim we spotted this and announced this before the rest of the industry did but it was pretty obvious if you were paying attention to why big data matter what was going to happen so we shipped in 44 in 2014 we began doing some pretty substantial contribution integrating with the rest of the framework so it's not merely data analysis and processing that's interesting you have to think about the data on which you're working what constraints apply to that data who's allowed to see different values have you got a robust security infrastructure if the system goes down do you have demonstrated ability to failover to a remote site are you able to manage enterprise level availability guarantees so we've spent a lot of time working with the global spark community to make that happen and then promoting right so clutter a university we've got training we've been driving those kind of enterprise requirements promoting spark as an analytic platform with O'Reilly data science book series and more the folks at type-safe and data bricks ran a survey recently basically scouring the different companies in the industry and I'm super pleased with folks using the cloud era cdh distribution to manage their spark infrastructure so we've got a whole bunch of users doing that we've got more than I'm sorry I'm thought I missed well now let me just keep going on we've got a big strategic partnership with Intel we're ploughing a jointly affair to bit a fair bit of engineering payroll in the innovation on the platform and again working with the global community to drive its evolution we're concentrating on areas where we think we've got a real differentiated ability to make the product better so enterprise-ready because that's who we sell to write large banks hospitals insurance companies where data privacy data reliability regulatory restraints matter we're working very hard with the community to port those old processing engine scoop and and hive and pig and so on to the new system performance even though it is vastly faster than MapReduce it is not fast enough so we're continuing to drive that and we're driving sequel innovation on the platform in the context of our investments in sequel generally finally we're doing a lot of work on data science including we just announced recently a new open source project ibis aimed at basically making python for data scientists much more robust and able to communicate with the different processing engines under the covers more than 200 of our enterprise customers are running the platform now 800 nodes far cluster is a pretty monstrous computational capacity if you think about it and we see it being used in a huge range of workloads many of these continual data analysis on the fly model retraining real-time data service stuff that Google didn't conceive of when it thought about MapReduce back in the day we absolutely offer some ways for those of you who want to learn more about spark in particular if you hit the clutter a booth out here I think we've got copies of the spark book available Cloudera live is an online resource you can go to just fire up spins up an instance on the Amazon Cloud and or I'm sorry on a number of public cloud providers I want to be careful which ones i'm allowed to announce on and you can try out spark there without having to install it locally and we've got a bunch of training classes I've raced a little bit because I want to be respectful of the conference time we are even more bullish today on spark in the Big Data ecosystem than before is the next natural step for the Hadoop ecosystem I'm super excited at the work that the global community is doing and for those of you here in the room who have been involved in driving the spark platform forward in Scala thank you it's made a big difference to us thank you all for listening you