data.bythebay.io: Sal Uryasev, Building a Realtime Receiver with Spark Streaming
Recording: data.bythebay.io: Sal Uryasev, Building a Realtime Receiver with Spark Streaming
so I said I'm Sal GoFundMe formerly of relay take you and LinkedIn on and let's let's learn some stuff about receivers so I know what you guys are thinking I each of you just came in to managing a small football team but you don't have a lot of money so you've come to me and you said Sally you're a data guy can you help me pick the best wide receiver for my team right no okay all right we're going to talk about spark streaming receivers so first a little tiny bit because the nomenclature can be a little weird receivers are actually the sources of data and spark so they receive external data but they're kind of akin to the source of the River Nile basically that's where the data starts flowing and then it flows down the rest of your stream where all your cool processing happens nicely though it's a little better than the River Nile it's not going to take you a hundred years to find it you don't need the resources of a small Kingdom I it's much much easier and much more exciting and you can do a lot of cool things with it so I want to actually mention a quick quote because it specifically applies to the type of receivers I'll show you how to build today and this one's from Peter Norvig we don't have better algorithms we just some more data it's theoretically attributed to him I think we might have denied this quote but what this quote appears and lots of general data science presentations where it says like more features it's better than more data and I'm sorry more features is better than better models and more data that are the more features and there's kind of two ways you can grow this one is if you kind of go from one to two million data points you can build a much better model but another and in a lot of cases more powerful way as if you increase the breadth of your data and that could actually give you some new new features that you wouldn't have had before so I'm going to go through three relatively simple samples of receivers the first one is a very simple financial web api it gives you foreign exchange data basically euros to dollars pounds to dollars then I'll do a quick demo Twitter of how to build your own stream from Twitter and just an honorable mention for a queue system which is kind of a classic place we build reliable receivers and i'll i'll talk about what different types of receivers are as well so i'm going to do a small little local demo at the end which will be based on the twitter receiver it's the one we're actually going to build and i'm going to run it and will be a nice little word cloud with some live aspect so maybe closer to when i do demo if any you guys tweak to the data by the bay make sure someone tweets they look a little cooler but okay so I'm gonna go on I talk about slightly about the internals of spark streaming because it really helps understand how you build these data sources so spark kinda has two paradigms under its roof the first as you guys probably familiar with is batch spark and back spark is very akin to MapReduce Hadoop pretty much everything that kind of does a big batch of reason data does processing reads out so the kind of looks like this you start with some data usually in a distributed format and then you have two types of transforms some of them are just 121 transforms and some are kind of reduces repetitions these will be your group buys a lot of the joins and then you kind of write the data out so spark streaming is actually very excitingly built on the same exact framework and under the hood spark streaming is actually batch spark and some streaming engines they kind of you kind of get a piece of data in you do some processing and you send a piece of data on spark streaming on the other hand is what's known as micro batching so every time you work with data you actually work with a small batch of data and they've actually done some nice code reviews where the base unit of processing and spark streaming is a data stream which corresponds to kind of one thing that happens and addie stream is under the hood actually it looks a lot like a generator of our d DS and our d DS are the base units of operation and sparkles one of the possibility units of a separation and spark so one of the really cool things about spark streaming is under the hood you can actually just kind of run a loop and do things that you would normally do in spark and you can actually when we use the same exact code just kind of fun so we're going to talk about the very first piece here which is an input D stream which is kind of the source of the data and I'm not going to really talk much about MPG streams because one of the nicest interfaces of these input D streams is just to wrap a receiver so receiver is a simpler interface and for all practical purposes just remember that there's a receiver in podium that kind of wraps a receiver and then you don't really have to think about it too much okay so why did I choose this little simple foreign exchange API well it's free great for a demo it seems to have didn't have stupid rate limits the Twitter gimme it was kind of definitely a pain point on that for some types of Twitter data you can only pull 15 times for 15 minutes which was very brutal the other one is this is historical data and that is actually really cool i'll talk about that little later and very simple HTTP calls so very nice for presentation okay so i'm going to throw a little bit of code on those cases feel free to ignore it there's actually a github repo over a tattle the code if you care to look at it later so this is a really simple example that this is it's not the way you usually want to call a URL because you kind of want to check for the server being down all the kind of stuff but anyhow very simple connector just does a simple get call and returns a case class and that's all you have to do so this is just your connector nothing to do with spark streaming okay but this is how we actually build a receiver so this is the simplest possible receiver you can build on nothing too fancy if you notice we have our connector we're doing a simple get and what I can I want to talk about is the what you have to do to make this your see where happen so so spark provides you this receiver interface a sexual abstract class which kind of provides you some functionality for start starting stopping but most importantly for storing data so I'm not going to go through the line but line the demo is actually pretty good online but I do want to talk specifically about the storing function so this receiver the receiver interface provides you or char class provides you several types of storing function some of them are blocking and some of them are non blocking so the this is particularly the non-blocking function where literally all you do is every time you get a piece of data you just hit store and it kind of saves it into the spark back end and what happens is it kind of builds up builds up builds up data and then whenever you have your little micro batch pulse it will take all the zeta and then we'll save it to spark so kind of the disadvantages here as you might see is if the receiver goes down during the operation you will lose anything between what is stored in between one spark somewhere under the hood figures out that it needs to save this data so this might not be a problem this might be a problem it depends entirely on what you're building but this is by far the simplest possible thing the other thing I wanted to call out is just the boring thread.sleep Web API is have rate limits so you're going to have to do something along these lines so this leads to one of the major disadvantages of kind of these little simple web api is you really often limited to a single kind of node because rate limits and because you can't really just sharding very well with these simple api's so the question is why would you even bother building the simple web api when it's just a demo but there's actually some legitimate use cases so the first one is when you want to actually join with another large data source so i think one of another the takeaways i'm hoping to bring is that it's really easy to ingest additional data sources so you should so let's say you have another large data source even if you just have a very simple small stream it's very easy to kind of connect it to another alligator source and I have like a dummy example of that in my git repo so for this foreign exchange API in particular this one is actually a pretty compelling so it go fund me we process a lot of don't patient data and this donation data comes across in US dollars but also in pounds euros all sorts of exciting stuff like that and just by joining on the simple simple steering we can get kind of real-time analytics understanding what's going on another one would be heavy processing down the road maybe if you cross your data like a crazy person there's some neat spark streaming specific functionality windowing won't talk about much example of it will down the road backfill data is kind of cool so one nice thing about this true for X API is they also provide you with all the old foreign exchange rates in batch data that's warm so this is specifically interesting kind of for the financial application where you kind of want to do back testing so you build your crazy money-making algorithm you kind of want to look at everything that's happened in the past and what happens is if you actually build your stream to process this data live sparks tree me actually provides you this nice little interface of a file input stream where you can actually plug in and read stream directly from a file and the format of the back testing data here is actually the same as a live API so you can actually very easily just plug in a different input D stream and suddenly you can do back testing with the same exact code and the last one okay we use go for better passing the last one is kind of a prototype thing where like I said earlier you can really only have one receiver with a very simple web api but a lot of times the web api is kind of just see something you see if it works and then if you are a company with money you go back to these guys and say build me a real api and there you can ask for sharding you can be like grab the shard shard the shard and then you can actually have kind of some element of distribution and you can also get rid of those pesky rate limits okay so here's again it'll just treat it as code vomit you want you can find the code later but I kind of want to point out the one line this is all you have to do to actually create your receiver stream once you built your your receiver and yeah then use the window just some demo processing and I know I told you guys that the github repo has all of the cool code if you want to look at it I emitted this one function on because I wanted to keep it from myself but if you really want to know talk later um okay I mentioned this already the limitations with a simple API so mostly no sharding rate limits and the other one is kind of no acknowledgement so if you build a very simplest kind of shooting processor your kind of starting over if you if you take your shooting processor down your kind of starting or just where you left off work sorry you're not starting over where you left off you'll start off somewhere randomly depending on what other API looks so right now we're gonna go kind of to Twitter and we're going to kind of fix this last little piece all right the swine take my quick drink I know football references are not for this crowd right wrong yeah but it works really well though it's like okay so my second example is with a Twitter and here we're going to actually build what's known as a reliable receiver so before we do that I think we need to talk about whatever liable receiver is so the previous example we just did that is an unreliable receiver which means what I said earlier if the receiver goes down you lose data very exciting so oftentimes you don't care but every so often you do so usually it's with something like a queue where if you do you actually have to acknowledge that you got the data but there's a lot of other examples or it might be also interesting so the way that an unreliable restore works every time you get a piece of data you just call this non blocking store function and let's park carrying handle everything else underneath for you a reliable receiver on the other hand you actually drop down and take manual control so what you do is you grab a window of data and you kind of manually kind of build up this window of data and then you call a difference blocking function and the reason you partially want to build up this window data is as the blocking function can be kind of slow so you only call it less frequently and once you kind of do the blocking then you do some kind of acknowledgment or under standing that you've properly process your data so usually for cues where it's the knowledge a mechanism is correct or kind of standard so you empty p SQ s all these kinds of cues but we're going to do one without a queue so this is what is a client's ain't reliable receiver and I made that term up so don't don't google it too much and kind of the idea is and partially if we don't purposes but how do we actually build a reliable receiver without a queue and we're going to keep the state client-side so in my case I spun up a Redis database so Twitter API I'm sure everyone's here used it at some time or another very nice libraries in every one language everyone's used that everywhere in academia and Industry lots of work around it in fact it's so popular that spark streaming already has a built-in to spark streaming so you shouldn't need to build your own stream Twitter API it's not reliable but you really don't care a lot of the time most of the time but the rate limits really suck and I was actually going to do so much more of a Twitter for this demo but um it's just very very tough this is stuff was very hard yeah you talked I think if you talk to Twitter you can Fame lots of money and you'll get a better better stream all right so I'm gonna do a really quick ok so the code i'm gonna show right now I'm we're actually running right now Maya I'm a laptop and at the end I'll kind of present it so if you do a tweet that that mentions the data by the day hashtag it'll show up hopefully and yeah okay so really quick Twitter connector again code vomit ignore most of it very similar to the previous one except it uses the library underneath and it does a HTTP calls correctly all that kind of stuff I want to point out the one thing that allows you to kind of store the state from this API and this is a sense ID so you can actually keep track of the twitter ID as you're running this connector so every time you do a call it gives you back a list of tweets you grab the last idea and you kind of save it and the next time you you make a call with that sense ID and here's the reliable receiver so I can ignore most things I actually the one line above this you can see how it and it does the caching but notice how we're actually storing our tweets ourselves restoring them in an array buffer and there's kind of two ways to get an array of this one is the API gives you back an array the other one is you kind of can build it up yourself so here the API gives me back an array and i'm just literally storing this all right and then notably after I store the right this is where I update my state so the next time I want to run my receiver I will know where I live and what I'm doing and what's going on and notes honorable mention again the really long sleep for 20 seconds which is a kind of brutal like at some point if you have this you're like why do I even need this massive infrastructure why didn't need a computer I can like manually do this and create an annual stream but again you don't you will not usually have this longitude production this is a demo purposes okay let's do a quick demo boom boom boom if anyone's tweeted not many guys have tweeted recently told you see it oh just now right well so twitter has a little like a delay in the woods alive but if we give it 20 seconds that should uh the newest tweets will go green but come on one of you guys refresh nope alright broke it okay Oh ah Syrian data grid oh well my bed those might be why there's fewer tweets than I'm expecting yeah um well you know what if you come find me after I will make the swap and we can look at it all right I have a few more slides oh ok so I'm going to throw in just an honorable mention of kind of what a true reliable receiver looks like and I mostly gotta read the code but this is basically just an acknowledgement so you do the same thing as you do with Twitter you basically acknowledge I to your queue system and this is a my previous job me work a lot with rabbitmq so end up building a little library for spark streaming for rabbit and cute my github if you any of you guys are suffering from the same malady but okay I want to briefly mention a couple more topics so a spark reliable receiver is meant to prevent you from losing your data during kind of the receiving process but that's really not sufficient because as exciting it is to not lose data and just that very first step you could lose data elsewhere so there's kind of two concepts surrounding this and spark streaming right ahead logging and spark streaming checkpointing and so we're ahead logging it's kind of disabled by default because it's kind of slow it needs to kind of write everything after every rgd but if you care about like if you care about it you definitely want to enable it and you basically one of the nice things about he's going to distribute frameworks is the moment you're kind of being slow and if you have lots of money just literally spool up more more nodes and this is before good and a bad thing about our world because I think we've write some very inefficient code because of this it just doesn't matter um okay and I hope you guys can get my joke now because I think this is the only time I'll ever be able to make it no I picked a guy who technically fumbled mm-hmm all right you guys have any questions yes so with a spark streaming basically you can the data between the processing it is basically saved and it can be saved in memory where it can be also persisted to disk and I believe there is a second I'd actually don't know much about but there's I think a couple different ways that it can be encoded which is some of which are more concise in terms of compression but are slower so it's tends to be kind of a sit down and thinking like well do I have a lot of data well if not then I can put it in memory if I have a lot of data and I'm worried about spilling over then we can put it to disk and if you're worried more about even more better than you can kind of compress it so that there's no really strong guideline to it and a lot of its kind of as you build it you're like well this let's optimize it a little further if you do the check pointing and the right ahead logs then you can bring up the data in the case of a failure you can actually bring out that data but it does not arbitrarily replay a really long stream that's not specific good park of spark streaming but if you're using certain types of streams like kafka kafka allows it and you need to just give it a pointer and then that the whole piece will replay but it's not part of spark shaming except for kind of failure situations so if you don't um well this is basically if you want reliability so a lot of times and I was actually say for most streams you don't necessarily care about reliability so let's say you're building a very simple Twitter listener it's a visualization and if you lose a few points of data you really don't care so the moment you start adding all these little reliability components it adds a cost it boasts that it adds a costume development time you don't have to think about it you have to do more pieces but it also adds a resource cost so specifically for the right ahead logging it now persists everything to disk so one of the nice one of nice things about spark versus hi tube is that spark ryan's in memory it's like mostly in memory so what this means is that on hadoop in order to run kind of the MapReduce iteration that usually will take on the order of minutes and spark could be an order of seconds which is why spark can use this little bit of micro batching so the moment you start kind of writing to this you're now you might be looking at a multiple of resources that you need to kind of maintain the same performance what was the question again yeah so when you someone forgot to post me repeating questions so when do we need this enabled um yeah so specifically when you really care about reliability where you need at least once delivery or or even exactly what's delivery that's where you're going to not worry about your resource constraints in your time constraints back sorry which I yet ah i have not too much I've looked at some of the specific examples of them but I would the API is more complex and I would just avoid it in the simplest cases sorry I can't hear you all you're so taking a risk so you just build a receiver and uh not save it oh you're talking about the new API for it um sure I haven't looked with the new API too much yet but thank you it's basically saying that the new API allows a simpler approach that does not process the data is necessary the other I okay thank you appreciate it you system hacking questions is that is I'm not everybody's job okay thank you guys you've been a great audience and I'll stick in the correct hashtag you