data.bythebay.io: Eric Anderson, Predicting Hacker News with Beam and TensorFlow
Recording: data.bythebay.io: Eric Anderson, Predicting Hacker News with Beam and TensorFlow
uh first is this maybe I'll just Advance some apologies um there was a bit of a scheduling snafu um we I do have a little side project predicting Hacker News uh with tensorflow and beam but I don't have that presented today um so apologies for that uh but but we will talk about beam and data flow which sound similar maybe uh and I'll start with um kind of just the evolution of of distributed data processing and we'll kind of walk through um how we got to where we are today that'll give some context for beam and data flow and then it's a discussion about what's still hard about distributed data processing um which maybe get some motivation for both the flow uh included in that are stragglers scaling um I think we're also going to talk about streaming operations um like how do you manage a streaming Pipeline and then finally um if if uh if a demo cooperates we may um poke around it beam just a little bit so uh if you're looking for more details on beam I just want to give a plug for 4M session by Eugene uh uh kpov I think I got that right um he's also uh a collaborator on these slides so how do we get to where we are today 2004 there's the kind of seminal maap ruce paper and that spawns a whole bunch of work inside Google uh Colossus um mil willll Flume um mil Flume are very related to our conversations today um meanwhile the same sort of thing happens in open source actually um maybe partially influenced by Google but also on its own accord so we have Hadoop of course and then all the all the subsequent and related Big Data ecosystem Technologies uh and and this is I guess the the position that Google found ourselves maybe in three or four years ago as we try to figure out what's what's next do we keep publishing papers or maybe it's time that we participate uh in the open source Community um maybe it's time we release products and code instead of just papers U and that was part of the impetus between behind data flow um and and the subsequent um Apache contribution Apache beam and then I'm going to jump um from here into a bit of the motivation for these two so you know three or four years ago what problems were there still left to solve okay so actually there's lots of problems Sol to solve but maybe maybe here's a few um that we were eager to tackle within Google so event time processing uh you maybe you've heard that a lot this week um I think it's a Hot Topic uh in in data pipeline communities uh Eugene will be tackling this later I imagine this is kind of core to to beam um we're not going to talk about it today but these other three we will so say I can stand up at a streaming pipeline how do I make changes to it um this thing is live it's it's receiving data it's um has correctness promises um the other concerns are more about how I resource pipelines streaming or batch how do I handle straggling workers and um can I can I deploy these in a in a kind of flexible autoscaled way so starting with streaming operations um if you if you you stood up a streaming pipeline you you faced this dilemma where you're sudden like okay now I want to improve it or I want to make a change or I want to debug but um but you can't mess with that thing that's in production because you there's a there's a risk of losing data that's already in the pipeline um if you take it down for some time you'll be missing data uh during that downtime there's also these systems provide um like DD exactly once Pro exactly once processing uh oh sorry exactly once processing promises and if you you know take that pipeline down um you may compromise those promises and then finally uh this could impact your right pattern so perhaps you're writing a file every hour out of your streaming pipeline if you take that down at the half hour and put up a new one you're going to get half hour files you then recombine those files so uh how do people tackle this today so one approach is I I do a parallel swap so I deploy a second up updated pipeline the pipeline I want to move towards swap traffic over take down the first one otherwise I can do um what you might call a stop and restart so I stop pulling from my streaming sources uh you know Purge or drain my pipeline um stop the current one start a new one restart the sources so this actually doesn't fulfill all our requirements we listed it's um we we do preserve all the data in the pipeline but um we may be missing new data in the case of of a stop start if we can't buffer The Source um our DD guarantees are are a little unclear um it's possible for recers to be pulled twice into the different pipelines um and and it's unclear how we would preserve our right pattern which may not be a big deal so what do we want the ideal is maybe an update in place so what if I could just submit a new job to the cluster um with the same name and an update flag and and it it kind of updates in place so so this is what we've been working on in Google and and it it works for minor pipeline changes um and it's it's included in in data flow today uh but we still encourage some of our users to to look at parallel swap or stop and restart in the cases of these major pipeline changes okay so that was our our first want um straggling workers so what if uh well anybody who's run you know map reduce or or big data jobs knows that a system is only as fast as its slowest worker and often those workers can be really slow uh so so what what causes this worker lag well um it's either that the machine is unusually slow or that you've given an an uneven amount of work uh but the uneven amount of work can happen for a variety of reasons maybe it's difficult to predict or or difficult to distribute the elements on the other hand what's maybe more common elements may have an uneven uh expense like they may be uh variably expensive so uh I could take in a stream of um URLs to videos and then process those videos and you know someone's going to get left behind with a long video to process so the usual approaches again um we could we could kind of filter for bad machines so we we run a probe and figure out who's slow and get rid of them um a common one is speculative execution so worker a finishes worker B is still going so worker a just starts doing worker B's job in parallel um and whoever finishes first gets credit for doing worker B work uh well so this doesn't actually fix the uh variable work per element so A and B are still going to both get long videos and they're still going to run for a long time um the approach that we're looking at is called Dynamic we we call it Dynamic work work rebalancing we also like to call it liquid sharding which we think sounds a little cooler uh here we have to the high level ideas make sense so we just identify where we can split work who's behind split it pass it on the next worker um and then and do so you know make this handoff in such a way that we get still our exactly once processing standard uh so this sounds great but it's actually a little tricky um in uh so sorry excuse me the hard part may be how to identify how long a certain item is going to take to work how do I identify where I can split it these are challenges uh we've been working through that we can talk more about but the status is this is actually working so uh you can see before and after here on the left we've got the common case straggling workers all over the place and on the right we're redistributing those stragglers excess work to to more efficient workers um earlier I shared that we we're baking these things in data flow but we've also released an open source SDK beam um this is where beam kind of becomes important for us to do this work we need a signal from the source as to where we can split um and standard you know Hadoop sources don't necessarily provide that signal all the time so within the beam uh open source SD we provided semantics for sources to report where they're splitable um okay so now on to um maybe a third concern of ours which was how do I how do I scale um my workers maybe some framing on why this is a concern so in the traditional approach you have a bunch of machines at your company and you stand up a data cluster on those machines of some kind and then you submit jobs to that cluster this works fine um until you want to try uh doing this in the cloud where things are infinite and ephemeral at that point having a standing cluster gets expensive and it might be cheaper for you to to separate a compute and storage but then you have to ask yourself how do I then dynamically manipulate my cluster to save cost how do I dynamically move my job um and even even if I can do that add workers as I need them how do I assign work to those workers uh okay so the common approach is let's um most most of our data processing Frameworks come with some kind of built-in scaling Auto scaling functionality um in the next stage I can add workers and when we rep partition data for that stage they'll get new work uh but I also still need to resize my cluster um so now maybe I'm relying on my cloud provider to resize my cluster and it's unclear that these things are necessarily working in concert um and they're probably relying on pretty rudimentary data like CPU utilization rather than data of available specific to data processing so what might the ideal approach look like perhaps I don't need this job cluster Duality I can just assign a job to an to an ephemeral cluster um and they're just kind of one thing to do that I probably need fast spin up spin down times because I'm going to spend a lot of time booting up a cluster tearing it down booting up cluster tearing it down um and then I'm going to also have to again get these data processing specific signals that might not be available to me um and then I have to be able to split work on the Fly because I don't want to wait for the next stage so so how how are we doing on this well um going back to the BAM SDK so we're baking into the sources and the io of the bsdk um how paralyzable is a source how much work there is left to do from that Source how much backlog th those give us some of the data processing signals we need um this is where great in batch so you you can see here a demonstration using both um Dynamic work balancing no stragglers as well as autoscaling so I'm adding workers um and I I haven't really explained these charts so hopefully they make sense but you know I've got maybe 10 green lines going at the top 10 workers and then I'm adding you know maybe another 100 over the course of the job because the job determines that uh it can parallelize further uh on streaming jobs um still early days so so we have working prototypes working users using this in production but um still not fully baked into data flow what we're really excited about is now we can only think we can think in terms of just jobs so uh you can submit a job to the data flow Pi it spins up a cluster relatively fast accomplishes your work and automatically tears down the cluster so I've kind of given you um some motivation at the beginning for for why some of the things we wanted to tackle with um with data flow and with beam may maybe to kind of summarize and put these into greater context so we have uh data flow represents that that product is where we're putting some of these our approaches on Dynamic work rebalancing autoscaling um and uh and streaming Ops the ability to update a job the beam is where we're putting things that we think as a community we all need to be collaborating on which is if we can do great things with having additional uh input on on sources then then we should all work from the same library of sources we should all benefit from having um increased visibility into our our sources that allow us to split data and auto scale um so so the beam is um formerly the data flow API uh but now it's open source submitted to Apache currently incubating uh anybody can go and develop a new beam compatible Source uh report the backlog report where it's splitable um anybody can then write Runners to beam So currently beam works on Flink on spark uh also on Google cloud data flow as well as um a local Runner so I think I've got just a few minutes here um we I don't have a full uh demo for you but let me show you a little bit what what using the bsdk might look like so I've got here this is fairly small so let me uh see if I can go full screen um setting up an open like this is pretty straightforward um all you have to do is go grab this the um python SDK from GitHub uh as I mentioned on the last slide beam is available currently in Java and python um here this is a very rudimentary example where I'm just uh I create a data source which is just a a table of time stamps um and some skew information um in in beam just like any other SDK you you create a directed AC Cal graph you know a dag of computation steps so I I've created my pipeline object here um and and then I apply a list of transforms to that object so in this case my transforms are just read and then write I'm not actually transforming anything um and then I I tell it to execute and here I get my output which is a set of time stamps um a more comp icated uh you can see I'm having a demo error here but a more complicated pipeline is visible here so I'm going to read from a source of input files I apply a flat map um kind of like a group by key I do a combined by key here I can do some um mapping of values and then once I've I got the data in the place I want I can write it to some in this case I'm wrri into an analytical database big query uh and applying schema now you could run this locally as we've done here in this notebook or you could uh uh run this in Cloud using data flow service there's a fancy monitoring UI where you can see the graph of a pipeline um and see some monitoring steps so jumping back to um the deck here we've addressed some of those issues we're pretty excited about them in data flow and we're exposing the parts that are relevant to the community in the BM um maybe I'll take I think we have like 3 minutes for questions rebalancing reing you can detect the splitable points from the point of view why don't you just schedule with a smaller CH of of work yeah yeah so so why don't we just um scale up more machines spread out like fully paralyze the work and proba you don't have to schedule at if you have a degree that you have a anticipate what kind of compation you need you can even prech or a pre initiate comp same I think was they released a toolkit called air yeah yeah you want to contrast want today yeah so um and so as I understand airflow is is kind of an orchestration and um and scheduling framework so you um it typically relies on other execution um environments to actually to do do most of the work um I one way to think about it and and forgive me I don't think I fully addressed your Dynamic question question but I'll come back to it um I got carried away so uh a prob a data processing job might have its own graph of processing steps but that job itself might live in a larger graph of processing jobs yes um my understanding of airflow is that it orchestrates across data processing jobs um that may within themselves have a graph so so data flow could be um one processing job uh within an airflow graph as well as spark job or or a database query I come is kind of Machel kind of actually need and imagine many many more into this glow Global schedule yeah so and then your Dynamic worker balancing question we could scale out quickly first um our concern is that the the processing might be inexpensive we might overscale um and spend a lot of time booting up and spitting down schedule yeah so so we schedule uh so we do a few workers um see how fast we're progressing through those records and then we we do a um which breakable schedle a big Chun of data and then we scale from there yeah great I'm out of time thanks [Applause] everybody