data.bythebay.io: Eugene Kirpichov, Apache Beam (incubating)
Recording: data.bythebay.io: Eugene Kirpichov, Apache Beam (incubating)
okay hello everybody so I've been introduced already so let me just get to the meat of the talk so I'm going to spend the next 20 minutes talking to you about the Apache beam project which was born out of cloud dataflow and I'm really excited to be talking about that because I think it's a fascinating programming model for distributed data processing so this is the rough outline of the talk I'm going to approach beam's programming model by showing you how Google arrived to this model from the from MapReduce until modern days then I'm going to talk a bit more in detail about the programming model and then I will talk about the current status of the Apache beam project itself and how you can participate in it so Google's data processing story in the beginning there was my produce my produce paper was published in 2004 and this was a system that basically does one huge select query followed by one huge group by query it was revolutionary in several ways first its main contribution is realizing that many data processing tasks can be formulated as a combination of these two kinds of queries select ungroup' by the second contribution was building a really efficient and fault-tolerant system that takes care of everything except your except your functions that you apply in selecting true by takes care of the fault tolerance and distribution and so on so providing an abstraction on top of that yep this were the two main contributions of my produce and another one was actually that it provided this abstraction as a library rather than as a custom language such as sickle and the reason for that was that there is already plenty of processing code written in different programming languages that would be harder to include as part of these queries and another goal was to make it easy to him run it programmatically from programs accomplished and higher level tasks so these were the two the three main contributions of MapReduce however as I said it was pretty low level and people kept reinventing systems for writing for sequencing a chain of I produce this together so approximately in 2010 a higher level tool was born called flume Java this was a tool for expressing higher-level data processing pipelines basically directed acyclic graphs of data processing transformations this is a graph with two types of nodes collections and transforms collections logically contain data not physically and transforms take one or more collections and produce one or more collections as a result so as a programmer you construct the pipeline graph programmatically and then you say okay please run this for me and the system takes care of the rest such as deciding how to best distribute the execution of this graph how to optimize it for example avoid materializing some intermediate results or do various database like optimizations like pushing down predicates or avoiding grouping somewhere or avoiding sorting and so on but the main contribution was really providing a high level API so this was a pretty successful project and this is an example pipeline in the flume Java API it is remarkably similar to the modern dataflow and B major and it probably looks familiar to anybody who ever used SPARC so you manipulate these logical collections and you apply transformations to them this hypothetical by plan reads some collection of sensor events from one from some kind of source and parses them and then computes for example the mean temperature by sensor location and writes that to BigTable relatively straightforward so people used this programming model to process lots of datasets sometimes the datasets were pretty big and sometimes the datasets were really really big and a recurring pattern in these kinds of big data sets is that keep growing over time and they're usually organized by time so these large amounts of data usually don't appear at once they appear organically over time so a very common pattern to process these kinds of datasets is well there's two common patterns one is to have a running bad job processes the whole thing which obviously at some point this stops scaling because at some point the dataset becomes too big to be processed quickly enough and the other pattern is processing new chunks of these data as they arrive this is probably more common so this way of processing these kinds of large data sets has a couple of failure modes so there's a couple of ways in which Bechet batch processing is not quite good the first failure mode is latency you cannot get low latency out of this kind of processing so for example if you have an hourly bad job you're going to have at least one hour latency so that's something you cannot afford in many cases such as fraud detection or anomaly detection or any kind of monitoring or rapidly learning machine learning models so latency is a pretty big problem and the second problem is sessions by sessions I mean when the data set can in some way be partitioned by user which probably sounds familiar to many people in the room and the actions of a particular user can be partitioned into sessions which are like contiguous blocks of activity by this user with some gap between them when you're processing datasets in these kinds of chunks sessions get torn up chunk boundaries sometimes this is acceptable and sometimes it isn't so the need arises to treat the data set as a continuous entity so really datasets like this are continuous and unbounded and new data keeps arriving into them and you have to embrace this fact so upon embracing this fact a bunch of people in the outside world came up with with a number of stream processing systems such as well you know storm and all kinds of others and until recently the state of the art was the state of the art was the stream processing systems did processing approximately for example they provided guarantees like events will be delivered at least once or at most once or they might be duplicate sometimes are lost so people believe that this was just something you have to deal with when processing streams and in cases where exact results were removes I needed people use something called the lambda architecture where you use a periodical batch job that computes exact results and you have a stream system running in parallel that computes approximate refinements to these results and at any point you have you're kind of combining the results of these two systems however in the end you always get an approximate results and that's something that you cannot always afford to do so at Google we had systems where you cannot afford to have approximate results you would think that it's not a big deal to lose or duplicate a single event because there's billions of these events right however it becomes a big deal when you're talking about duplicating or losing rare events such as detecting a fraud instance or detecting an outage from a stream of monitoring data or something like that so if you have a system that inherently provides some leagues and at least once or at most once guarantees you were going to have to deal with these problems for rare events too and that sometimes not acceptable so in 2013 Google created a stream processing system called mill-wheel and this is a system that was designed to cope with these issues its main contributions were the following first it was deterministic so it made no compromises on what kinds of events will be delivered and how many times it works as if everything had been delivered exactly once and it takes care of providing that abstraction it turned out that this is possible another contribution mill-wheel did was dealing with events in event time which means that we're looking at the timestamps at which events happen rather than when they were ingested into the system which can be sometimes while the different for example if you're talking about some kind of in-game events in a mobile game that you're developing if a user played the game in an airplane and then got back online and the game push these events to the server than they are clearly delivered way behind when they happened so mill will provided this kind of programming abstraction of directed acyclic graph of computations between which elements are flowing and provided an illusion of exactly ones processing however it was relatively low-level like my produced so it did not really borrow the flume Java API for high level processing so we ended up with two classes of systems streaming systems though at least we figured out the way to to get correct results and batch systems which were easy to program at a high level and in the outside world there was also a trade-off between correctness and latency so about two years ago we realized that these don't have to be different and we need a programming model that allows us to program batch and streaming data processing not only similarly but identically in fact in fact many cases of batch processing are just poor imitations of stream processing because of some kind of need for a compromise is being felled by the people developing this but the compromise does not have to be there it is possible to have a system that is programmed identically in these modes and this led to the development of a new programming model and this was a huge chance to to do things right in this space because things have not been done before in this space and we decided to do them right and we summarized the principles of that programming model as what where when and how let me talk about that in more detail so the what part is the computation itself so what kind of processing you're doing for example element-wise processing like filtering or transforming cross element processing like aggregations or groupings and composite transforms that are basically just sub graphs of transforms that have a name so that you can reuse them multiple times or all kinds of other things on them this is the easiest part now the more interesting part is the where like where in event time are you applying these transformations when you're dealing with an unbounded dataset you cannot say that I want to compute the answer because the dataset is constantly changing so there is no answer for the whole dataset so usually what you do is you slice the data set into some kind of chunks by time and you omit answers for these chunks so you say like the result for for that particular hour is this and there is different ways in which you can partition the data set into these chunks called windows you can partition the wholes data set into fixed or sliding windows and many stream processing systems today allow this but the programming model I'm talking about allows more flexible windowing for example it allows you to partition the data set by key and apply different when doing in different keys this is necessary for session-based been doing then so this is an example of how windowing looks in a pipeline it's just a transform so yeah it just fits into the programming model as a transform for example here we're redoing the data set in two minute windows and then within every window we're applying some the next more interesting part when in processing time in particular when do we declare the results for a window complete or when do we emit the results for a window because because of these data delays new elements can arrive into the window so it is possible that the result will change at any time and you have to decide how are you going to emit the updated result and there's another case for the for asking this one question for example if you're doing windows by day and you want to emit early intermediate results every hour so that's also kind of a triggering condition for when to emit intermediate results so the programming model of a patchy beam includes this concept called triggers and they control when to emit intermediate results for Windows and there is an important related concept called the watermark it is basically a guarantee provided by an input source on how wait can the future data be unfortunately I don't have enough time to talk about this in more detail but I refer you to the similar excellent presentations by Francis Perry and Tyler Aikido and other members of my team so I'm going to continue talking about the fourth component which is called how how do the refinements relate so how do what happens to their hymens do the results as late data arrives or as we omit intermediate results for a window and there's different kind of strategies you can apply here some more simple some more complex for example you can just discard the new results or you can replace old results with new results or you can add them up together and subtract the whole result to get an affaire running sum that should represent the total so together these four components allow us to deal with a wide variety of use cases starting from classic batch processing where you process the whole thing all the way through windowing the data set and dealing with speculative results or later writing data and finally to expressing how late results affect all the results and how they are communicated so these are the pillars of this programming model and they were implemented first in Google's project called dataflow dataflow my colleague Erik Anderson already talked about the dataflow service in detail today so I'm going to skip talking about it I'll just mention that it's basically the programming model I'm talking about plus 0 administration service so you don't have to maintain any clusters or do any tuning and the key topic of my talk is Apache beam so this January the dataflow team decided that it is that it is prudent to make the project open-source so we decided to open source the programming model behind dataflow and try to make it a unified programming model with different ways of running these pipelines and the project was donated to Apache and I'm excited to say that this Apache proposal what was met with huge enthusiasm it got I think 44 unanimous positive votes for being donated Apache and this is as far as I know the highest number of positive votes any Apache project has ever received so this project is very much supported by the open-source community by many industry players the committers include people from Google of course but also PayPal talent the task data or designs and others so what is bheem first it is an SDK it is the API that allows you to construct and run pipelines currently we have Java and Python API is for constructing pipelines and pipelines can be run in different ways this is one of the key components of beam this portability across different runners you can run it on your local machine you can run it on SPARC or run flink or you can run it on our service which is of course hopefully more efficient but if you cannot afford to run something on the cloud you can run a spark of link or luckily this is just an example pipeline written with the vm api but it looks pretty familiar to you by now probably I believe Spotify has an open-source project on github providing a scholar educator yeah so as I'm saying it's an open source project and people are already actively participating this is just a single slide view of the ecosystem and users write pipelines by plans use libraries of various transforms or data sources or sinks this build upon language specific SDK is available in different languages these all implement the same beam programming model which I described before and that programming model runs using a particular runner on a particular execution environment so it looks like I'm mostly out of time but basically you can interact with the project at all of these levels you can contribute new runners or writing transforms or write pipelines so I invite you to do just that and yeah I think I'm going to finish here and ask I'm all ready to take questions so the B model is fully comfortable with running machine learning computations and that and I am pretty sure that you would be able to do that using this part Runner which is optimized for low latency processing of batch data set so beam is the programming model it's how you express transformations and machine learning algorithms are just examples of these transformations so it's totally possible to write them yourself yes he was intimating the first light mentioned so the way this works is you write the pipeline and you say you run this pipeline using this runner so you can for example use the spark Runner to run this pipeline own spark or use the dataflow Service Runner to run it on the data flow service sure it will be two separately running programs but yes relationship between a beam and flink windowing API is beam API higher-level does it replace when we doing I believe the latest flink windowing API was inspired by this programming model in particular they specifically added some windowing features to be able to implement this model and to implement the flink runner so I'm not familiar with more details of the flink API but I'm pretty sure that the window me pay is very similar because it was intended to be very similar I'm not that familiar with link itself I'm sorry I'm sorry I'm not familiar with this I'd love to learn more