Large scale, real-time stream processing using Spark Streaming

SBTB 2014, Tathagata Das: Large scale, real-time stream processing using Spark Streaming

Recording: SBTB 2014, Tathagata Das: Large scale, real-time stream processing using Spark Streaming

thanks Jason for the flattering introduction um so a little bit more detail of who I am I'm basically a core committer on Apaches spark for those who do not know what apach spark is don't worry I'm going to explain all of that and also the lead developer of spark streaming I was in a PhD program in Berkley last semester but I'm on leave right now and I'm here right now uh to talk about spark streaming now uh even before spark streaming let's just start from the basics that big data I'm sure you have all of you have heard this buzzword it's being thrown around all the time uh specific in Big Data today I'm going to talk about big streaming data and basically you know that there are lots of high volumes of data streams that are uh that are floating around right now there's Twitter stream there's streaming data from sensors there's streaming data uh from uh uh uh uh distribution networks there's lots of streaming high volume streaming data that needs to be processed in real time for example uh you may uh have want to do fraud detection on continuous Bank transaction and find out fraudulent transactions as soon as they happen and you really need to find those as fast as they happen otherwise the harm is already done similarly there may be anomalies in sensor data that you might want to find again fast so that you can react to it and fix the sensors or whatever or you might do something much more trivial and find CAD videos on tweets that also works but that's also big data so how to do process this large volumes of streaming data and what we essentially need is a distributed processing system which is basically a cluster a framework uh that is running on a cluster which takes this large volume of tweets or any data stream that you like and gives out the processed results now what are the characteristics that we need from this distribut system it needs to obviously be scalable scales to hundreds of nodes achieve low end to end latency so that you get results fast you also need things like uh efficiently recovering from failures as well as something we we valued a lot from exper from our from our experience in production environments is something that integrates well with batch and interactive processing systems now let me elaborate on that why so what have people been doing in build uh till now in building these kind of systems and what people have mostly done is they build uh these sort of two independent parallel Big Data Stacks one for batch pressing like using Hadoop map reduce Etc uh another they build their their streaming stack either using their own custom home B streaming system or things like storm Etc that has been gaining popularity lately um thing U systems like this they build two different parallel Stacks uh which often process exactly the same amount same data one processes in in realtime fashion the other processes larger volumes in more bassle fashion let's say to generate weekly reports and stuff now why two different Stacks because none of the existing Frameworks well until now uh with spark streaming none of the existing Frameworks till uh till maybe last year could do actually could handle both kind of uh processing requirements so some Frameworks like storm could either do stream processing ING with hundreds of mb per second throughput with low latency and stuff or Frameworks could do high volume batch processing on terabytes petabytes of data like hardup map produce but not a single system that can do can be tuned to do both and the what is the side effect of having these two dual stack system each of these different Frameworks have different processing models they have uh that you need to learn you need to understand you need to express your business logic in different processing models that doubles effort of implementing doubles the effort of solving bugs double the effort of maintaining consistency across these different models if you're doing the same competition on both these stacks and obviously on the from operational point of view you have to deal with the managing two different Frameworks which Each of which usually has its own different idiosyncrasies and stuff so in short it's not exactly uh a simple thing to do enter spark streaming and what essentially it is is uh it what it does is this is a system that receives data stream from many different in standard input sources like Kafka Flume we recently added Amazon kesis support uh Twitter of course and playing old hdfs or hdfs compatible systems like S3 takes data from them processes them mes all the data that you want and pushes the data out to storage systems uh hdfs database no SQL systems like Cassandra H base and maybe update dashboards directly and things like that and this whole and this whole system is scalable fall tolerant and achieves end to endend latencies of the order of seconds let's dig deeper how does it work so essentially if you open up spark streaming inside you will find that there are essentially two things one is a set of receivers that receive the data as they as they're receiving streams of data from outside uh it chops it up into small batches of data and hands it off to the second component which is essentially the spark processing engine actually let's have a show off hands how many of you are not familiar with Apache spark well actually quite a few okay that's good I have then this is probably the appropriate time to do a little bit of introduction on spark uh so what is essentially spark uh as you can see I guessed that there might be people I have the next slide ready uh so uh so spark is essentially a very fast general purpose cluster Computing framework that is interoperable with Apache Hadoop map reduce and stuff and uh and it's a very general purpose uh allowing all different kinds of computations and it it improves the efficiency of the computation over Hardo map reduce like systems using two features one is inmemory Computing Primitives it allows you to keep the data around keep large volumes of data around in memory uh that can be processed over and over again and it exposes a very general computer engine not just map reduce uh by the is it how many of you are not familiar with map reduce okay that's good okay I didn't have any Slice on map redu I I would have been in trouble if if anybody said okay no map reduce okay good uh uh so uh beyond the map reduce very restrictive model of map reduce it spark allows General computation graphs and in terms of usability improves the usability of map ruce simply because you can write programs in in spark in Scala that itself is a big deal which I don't need to convince you guys uh and what it also provides is it extends the uh we provide something called a spark shell which extends the scalar shell that allows you to interactively query big data over a cluster in the background interactively at the scale of seconds just like you use a scalar shell now I will not demonstrate this fact uh uh spark yet because in the interest of time I want to focus on spark streaming but let's uh see what are the benefits of this so for example this inmemory Computing and this General Compu that improves performance uh in terms of your uh job processing times up to 100x compared to map reduce if your data is in memory or 2 to 10x compared to map reduce even if your data is not in memory but on disk and in terms of fusibility of course it it allows you to to do really fast interactive big data processing on the Fly U using your scalar familiar scalar shell environment now what is a programming model behind spark it is this thing called resilient distributed data sets or rdd and the an rdd is nothing but a distributed collection of objects now this essentially imagine it's as a huge array which is partitioned across mult uh multiple worker nodes in a cluster and now you can uh these are essentially like sequences in a very uh if you compare it with SC these essentially like sequences but what you and you can do essentially do these sort of map flat map kind of parallel manipulations except when you once once you apply these sort of functional operations that gets executed on a cluster so that is the big difference and these data sets can be cached in memory um uh on the worker nodes I'll walk through an example to to better explain it and another fancy thing about this rdds is that you're keeping the data in memory in a cluster environment you your machines can die or in a in a cloud environment that is often the case the rdds have enough information that it can recreate itself and restore itself in memory if any part of the rdd is lost because of machine failures so these are the very important characteristics that allows rdd has a very strong program abstraction for processing Big Data let's walk through a very example of let's say you want to have a large log file and you want to process that lock file interactively to find out error messages in the lock file and find out patterns and stuff so what you will do is you can do it either in a stand alone program or in the spark shell uh here is the let's say this is the cluster envirment that you have one driver three workers on the driver node you'll basically be running the spark shell and and you'll do something like this say spark text file you open up a text file which is sitting in hdfs and what it what this line does essentially creates this Base rdd called lines and and this lines represent the data that is sitting in the hdfs file next step we apply this let's say filter operation and we want to find only those lines which start with error and which creates another transformed rdd errors which uh essentially contains only the lines that have that start with error and uh next maybe we want to do further stuff like split each of the lines by Tab and things like that and and excite out the second columns etc etc and uh then we also let's say uh we want to process this set of uh this final rdd messages over and over again as in run many queries on top of it so uh why read it from the data over and over again we'll just basically ask the system to Cache it in memory so that for their queries on that is going to be much faster now while the system is actually executing this lines on the cluster nothing is happening it's essentially all lazy uh so the driver is just drivers and the worker is just sitting idle as of now only when we actually ask for some result back let's say filter these lines and give me the count of how many lines are there that matches this uh in the file we want to result back and you and count is essentially what we call an rdd action once we ask the result back does the driver actually go ahead and do something it actually uh figures out where in the hdfs uh where the data is present in which workers it is present it sends out tasks to the workers similarly similar to the map reduce fashion but in a more efficient way uh workers all the workers execute read the blocks of data execute the tasks and finally give back the results which in this case is count and the driver gives back the result finally into your program um now also what is happening in parallel is that since we have asked the messages rdd to be cached in memory the system actually caches the uh the materialized messages partitions of the message rdd in the memory so that when we when we execute another line um let's say on the same rdd it will it doesn't need to read the data from the blocks again it can just read it from the inmemory cache and give back the results much faster and using this we can uh once cach in memory the the performance difference compared to Hado map reduce can be of the order of 100x and you can process we have done things like processing querying one terabyte of data on 100 machines within a matter of 5 Seconds or 7 seconds so we have done pretty good benchmarks on this and because Park is such a generalized Computing platform it is essentially the Computing engine behind a whole slew of other other uh components bu bu being built on top of spark spark streaming being just one of them there spark SQL which is essentially SQL processing SQL querying on Big Data there's machine learning libraries MLB there's graph processing algorithms graphx all being being all being built on top of spark anyways going back to spark streaming uh there's this essentially spark streaming essentially has this spark uh sitting inside it uh streaming hands out all these uh the data stream in batches to spark spark does all its usual processing that it does and it finally gives out the results in batches uh and for stream spark streaming there is a higher level uh programming model called dams which is essentially represents a stream of data and underneath in the system is essentially implemented as a sequence of this rdds as a continuous sequence of rdds I I'll explain the example in the next slide and the the niin part is that the dam API is very similar to the rdd API it's a same functional map reduced kind of UI you can create input you can create input D streams from different sources and then apply these map reduced parallel transformations to create more D streams for transforming the data let's walk through a very simple example let's say we wanted to um get all the live hashtags from Twitter right now and what we'll do is essentially first start something called a streaming Contex which is essentially the uh starting point of all the streaming functionality using the streaming context first we'll create something called input D input daming tweets uh using these Twitter util functionality this is all inside the spark streaming programming model you can just go to the website no need to uh learn all this code on the fly so it's all there uh so input Dam what essentially the system underneath will do is that it will it will go to the Twitter streaming API get the Twitter the stream of tweets from Twitter and chop it up and divide the data into batches and store it in the memory as rdds then uh plus it would also replicate the data in memory for fa tolerance and stuff I will not go into those details here uh uh let me focus on the high level Basics uh next when we apply these sort of functional Transformations like flat map and stuff uh it creates new D streams and the system underneath what it does is that uh corresponding to every rdd in the parent tweets Dre stream it applies the same functional Transformations on each of those rdds creates a new sequence of rdds the new sequence of rdds essentially becomes a new dam and then again you can keep doing these Transformations and finally let's say you want to do something with the data you want to push the data out to let's say some file system and some into a text file or something so there you can do these sort of output operations like save as text file where each batch of data will be saved as a text file onto any hdfs compatible uh file system and the system underneath will take every rdd and save it not just saving to file you can do arbitrary stuff with the data for and this there's this simple Prem for each rdd that gives you direct access to the rdd environment and once you have this rdd you you can write arbitrary spark code raw spark code to do whatever data do you want to do with the data you can directly push the DAT out to Cassandra let's say you can push it out to age base you can directly up update a streaming a dashboard UI in real time whatever you want to do with the data you can do with it also besides the basic map reduce Etc there's some additional functionality that comes with Dam and some of these are called window-based Transformations what it essentially is that you have a stream of data what what this allows is essentially a sliding window operation on the data and in this case uh in this example basically what you want to do is that count all the values uh in the last one minute of data every 5 seconds and so as you can see there like two parameter what is the length of the window how much data to aggregate together and how fast should the window move every 5 seconds in this case and so there's like variations of this window operation where is Window based aggregations like Reduce by window and count by window and things like that also there's uh you can do arbitrary stateful computations let's say for example uh you want to monitor Twitter uh the whole Twitter stream and let's say for every user you essentially want to maintain what mood the user is in essentially you're trying to maintain state for every user well it's actually sounds creepy but but uh let's go with the example so uh let's say you want to maintain this state on a per user or a per key basis what you can do is essentially give this simple function that given the previous state of the user uh previous mood of the user the new tweets that he has made what is the new mood of the user you give this function to the system and apply it on using this transformation called update state by key and the system will keep updating the state for all the users as data keeps coming in and it's very functional obviously and it just the system just takes care of it by itself of pro applying it on large volumes of large volume of data streams so this allows you to do very arbitrary stateful computations out of it without requiring any external system for fall tolerance and stuff and uh and even more stronger thing is that uh the and which shows what is advantage of having this very streaming API which is very similar to a batch API of spark is that you can very easily intermix all these rdd and D stream operations like for example let's say you have a file which has all the data let's say information about spam tweets and you want to take the uh Twitter stream the full high volume Twitter stream and uh you want to join it with the with this spam file to filter out these the spam tweets out of it basically you're taking you're trying to take a stream and use a file to filter it out that is not something that is very trivial to do uh in any other processing systems like uh streaming systems like storm or even home B solution it's not very trival to do this in a fa tolerant way but here you can do it very easily using this transform operation which gives you the rdd here is rdd do whatever you want with with that rdd like join it with another file and things like that uh and uh it will keep doing it on the fly so um uh this gives you a lot of power for example this and what you can do is essentially in the same system spark Plus spk in the same platform you can take uh let's say compute statistics on historical data over last one month or one year of data and take that statistics and apply it on the live data stream uh using build models on historical data and apply it on live data streams you can combine streaming with other components of the spark stack like machine learning algorithms and graph algorithms and do things like offline learning online prediction then online learning and prediction as well simultaneously learning the data that is coming in and predicting using the models learned continuously and you can also do things like combine interactive querying with data stream like for example interactively query the Stream streaming data using SQL and in fact let me actually show a very quick demo uh I think time permits me so a very quick demo of how this sort of actually looks like and to show the demo let me actually segue into something that my company uh as in not my Company the company I work for uh datab bases we are building is this uh um let's actually start from the scratch is this uh Cloud platform that we're building which basically takes the open source Park platform so by the spark all of it is open source if I if I forget to mention just be clear that all of spark and Sparks stre and everything is open source but what we built on top of spark is this uh integrated development environment which is a hosted managed hosted Sparks environment that allows you to create spark clusters on the Fly and then do Interactive uh data visualizations using that and for example here in this uh this is essentially the database Cloud platform here we can build workspaces like for example I'm going to uh do let's say this is essentially the spark shell running in the fancier UI and uh you can do simple scolar stuff as well of course uh there is uh sorry oh crap that's yeah I think that is better is that better okay yeah better than what was earlier of course uh so uh so for example I have a cluster running which is a kind of a small cluster with 16GB memory just for the demo but I can very easily create clusters I can say uh my new cluster added let's say 8 GB or 10gb memory and create a cluster like this it's and it'll it'll automatically create the cluster for you and prepare it uh that you can directly start running spark you don't need to do all the management of actually creating clusters and stuff anyways let's I'm don't my focus is on spark streaming on this in this talk and so so what I'm going to do using this is that I already have uh um uh a cluster setup as I showed and what I'm going to do is actually run a spark program inside it to show that how it actually look and feels uh in in that so let's say I'm going to uh let's say first I'm going to actually import a few functionality um H import let's see it okay uh so what I can do is using this is I can actually upload um libraries uh so for example if you have written your custom Library code I can use that I I can actually upload that uh as a jar and use that in my program that's what I to do I have some basic functionality out here like getting Twitter authentication and stuff that I have written in a jar I'm going to use that uh okay this is going to be slow this I did not okay okay let's let's actually try something else on the floor well I cannot run the program uh for obvious reasons uh but what I was going to do is essentially create the Twitter stream and uh here for each of the TW batch of data that is in uh that is generated by the Twitter stream it actually um I can take the rdd and register it as a table and Spark directly supports that using this spark SQL platform as a table and then I can apply SQL queries on top of it um let's head back to the slides in the interest of time um so what I was going to demo was recently on the on the sample Twitter which is very small amount of data but instead uh but you can actually we have actually tested spark streaming on very large volume data we have processed like 60 million records per second which is like 6 GB per second on scaled out to 100 nodes all with a subsequent in Twin latency uh so it is really scalable fall tolerant and stuff and also another advantage of a unified stack using spark and Spark swimming is that you can very easily transition between different modes like for example let's say you have to actually figure out uh analyze the data and figure out in patterns like this so you can use a spark shell to take small amount of data and figure out some pattern then take a uh apply that the algorithm that you figured out to figure out the pattern on much larger volumes of data by writing a standalone spark job and running it on on Thousand machines and then uh ex and once you have written that spark program you can take essentially the very similar spark program and convert it to a spark streaming program and run it on live data stream as well it's very easy to transition between different modes anyways to conclude uh so spark streaming comes packaged with spark so just go to the spark website and search AP spark and you can get the website it takes really 2 minutes to download and try it out locally and you can very easily create an E2 cluster out of it and so you can actually test it out on E2 cluster we have very active spark mailing list where people are very active about answering questions and helping each other out and look at the programming guide for more information thank you