Structure and Interpretation of Stream P...

Scale By The Bay 2018: Adil Akhter, Structure and Interpretation of Stream Processing

Recording: Scale By The Bay 2018: Adil Akhter, Structure and Interpretation of Stream Processing

you hello good morning thanks joining this talk the title of the talk is the structure and interpretation of stream processing so as you know that it has become a de facto standard in dealing with large-scale data large scale and growing data the stream processing this mic is working right ok thank you so as I was just saying that same processing has become a de facto standard in handling large volume and growing volume of data in a recent year it has also gained a significant interest from the research community because it poses several challenges such as programming abstraction resiliency fault tolerance and so on so in this talk what we're planning on doing we're going to look into the fundamental of swim processing and we are going to analyze the state of art stream processing systems from two distinct perspective the first one is the structure so what are the programming model offered by different stream processing systems and how they can allow us to write reliable stream processing programs and then we'll see that how those program can be executed or evaluated by the stream processors and also we're going to take a look at the architectural consideration and semantics of stream processing while discussing about this execution models and last but not least we are going to take a look at the recent trends in the stream processing landscape and also look into the research direction that why we are heading at in this domain so without further ado let's get started a bit about me my name is a deal and I'm coming from Amsterdam this is the second time I'm here in Scala by the bay it's a great conference and it's the second time I've been speaking we do it with the jet lag so please bear with me for the rest of the presentation so I work in a global organization called is a financial organization based in Amsterdam and we are operating in 40 different countries serving around 36 million customers over the role and we are collecting huge amount of data from different data sources and analyzing it to detect some actionable insight for our customers mitigate security risks and also looking into fraudulent transactions and detect that one so if you take a look at it we we have growing a large volume of data from different data sources one such data source is real-time transaction stream so that brings us to the definition of what is stream a stream is unbounded sequence of events which has several attributes for instance in this case we're referring to transaction so it has transaction ID customer ID transaction time and so on also it has some meta information such as timestamp which refers to the time when that event actually happening so if you take a look at the use cases of stream processing frameworks there are a certain common part there but there are diverse data sources that we are reading from and also there are different communication challenge channels out there but one common part is the pattern and prediction part that we are reading large volume of data try to try to detect some kind of pattern and then try to determine the relevancy of those there so we're reading huge amount of raw data from different data sources trying to filter those data try to detect some pattern in it and then the relevancy and sending notification to the to the end-users so if we think about this kind of architecture it poses several requirement one size requirement that the system has to be perform in a low latency and high throughput it must support fault tolerance and a high availability and also it sub we should support visible processing and that's why the stream processing landscape has been evolved from a single node processor to multi node distributed stream processing systems in last 10 years so in the next section we are going to take a look at how the stream processing system has been evolved in the last 10 years it has a significant interest from the research community so the initial project were basically the research prototype which basically then become an industry scale project as well so if we have a close look of how this has been evolved the first generation stream processors were basically an extension on top database engine and second generation stream processors where they provided a bit of advanced capabilities such as operator expressiveness and advanced query processing but important to note here is that the first generation is stream processor were quite domain-specific because they were serving only certain purposes but as we are going towards the second generation and third generation it becomes general-purpose stream processing frameworks in the third generation it focus on focus on scaleable and highly performant robust towards fault one most popular stream processing framework for instance tome is from that generation and in the first generation the focus was towards deriving the exact results and also the for generation stream processor provide expressive abstractions richer windows specification and its efforts operation in a highly available way Muro Muro pearl and their four generations in processors they also introduced some additional features like transactional processing and in a real time situation it wants to support some asset transaction for instance fling in the reason here provided this streaming ledger support which basically provide transactionality on top of a stream processor so let's take a look at the most data part stream processing system out there and the stone that was English initially the first popular stream processing platform that made stream processing mainstream although it has several limitations this limitation were in which by extension such as fried and then one improvement on top of Strom was Heron that was built by Twitter spark streaming originated from a research project at Berkeley and then it has become a mainstream stream processing framework it is built on top of spark as its core then there is other open source framework like summer for instance was build by LinkedIn that uses Kafka as its streaming there and in order to support for tolerance and yeah snapshot it used after log then we have another research prototype like flink that was initially built as a research project from tu Berlin now it's supporting both streaming and batch processing then we do have cloud dataflow that assimilated several other stream processing framework from Google like mill will and flume Java but this is a proprietary stream processing framework from Google so we're not going to focus on that one we'll be focusing on the open source stream processing framework for the rest of the presentation so in this section we are going to take a look at how the programming abstraction has evolved this is fin processing framework they provide different level of abstractions but there are some fundamental commonalities among them so we're going to highlight them and try to distinguish different kind of stream processing programming abstraction that's out there so in this section we're going to take a look at programming abstraction and the primitive that they offer then we briefly look into notion of time and then discuss about the streaming window a programming abstraction that we can find in most stream processing API and regardless of how declarative they are so the programming abstraction there are different levels and they're like the low-level programming abstraction is basically data flow graph and then on top of it we provide some functional API and then on top of it there is this declarative languages like c ql link signal and sparta crawl so we will first discuss about the low-level dataflow programming the modern stream processors such as Strom and Sam sure they provide this level of programming where the stream processor the streaming program is encoded as a dag a directed acyclic graph and where each node represent a competition or some kind of operator that working on streaming input and user actually an extend some kind of interface to build those operators and they're interconnected by channels thus those operators and contain some arbitrary complex business logic as well so just to give you an example this is a directed acyclic graph where each nodes are computation and then they are connected with the stream as a channel to give you an example let's take a look at the word count which is basically the popular is about popularized by the paper Matthews we are reading a set of sentences stream of sentences that's in food and then there are two operator like speak and count the speed of our splitting those input as work and then accounting other occurrences of word and outputting it an output channel so if we take a look at this pretty operator the pseudocode how we can implement it is like okay we have a channel this channel is basically the output Channel so we are receiving input as sentences and we are splitting those sentences to work and emitting those who are in output Channel on the other hand if we take a look at it has to maintain some kind of state for instance we are reading we are receiving particular word and we are keeping the count that how many times we have received those words and we are emitting the word and updated count and our food channel so important to note here is the managed state the map state there this is managed by the stream processor which has several advantage like the the memory management part is handled by this thing processor so it can be checked pointed or safe pointed so it just in case this offer at a time then it can restore that operator from its previous state that's why the water and stream processors has a dream they provide this explicit state management construct but let's take a look at how this logical data flow graph that we define hourly here it gets executed so this data flow graph graph when it's the same processor schedule this data flow graph it actually creates multiple instances of those operator so these data flow copies into multiple machines so there are multiple instance instances of this tag getting executed and the same processor maintain a couple of things it maintains the network communication between them it partitioned the data so the different source can receive the partition data and then it also will handle the program recovery in case there is any failure and execution of stream processor is modeled as a tag of task interconnected by stream in this situation we can see that the stream actually connects two different kind of operator like split operator and the count of murder so this is an example with with strong and strong the topology is represented by two different and operator one operators proud that that at the source and the second fold those are basically doing transformation on stream so how can we implement the ball operator it's pretty simple it's like the pseudocode that we have shown earlier in the book end up in the word count both it's actually receiving some topo and then it's computing the account from it and you can see two things from here is that it's not using any managed state it's using some JVM map there meaning that the failure recovery is a responsibility of this this bolt so it's not main managed by the stream processor inherently and the second thing that ok this execute method is like a normal Java method that can contain any arbitrary piece of code there so as an end user it's your responsibility what will be encoded in into your bolt but it has several disadvantages for instance this state management now become responsibility of the end user so you need to write your own code to maintain the state reliability and also the in case of out the memory exception you need to handle that kind of situation as well so explicit mutable state is a requirement for more on a stream processing systems and using this data abstraction or API this kind of station managed state is provided by modernizing processes such as fleeing and Google Cloud Data Who and so on so just to summarize there are couple of pros and cons of a low-level dataflow programming the one important aspect there that ok you can write any kind of custom code there so you could write some optimization on dataflow education you can free you can have the freedom to implement your own business project but the cons is that ok it warrants a good knowledge of execution internal and abstraction is kind of leaky internal of the streaming system is exposed via those instruction and it lacks expressive and rich support of operation so next level in the abstraction is functional API the function ID I'd give you more declarative abstraction where I deal with stream passes as a first-class citizen it offers a higher order function as EDF so you can define with respect to math flat map and and filter that kind of Combinator's and these are quite known and due to the popularity of functional programming and user only need to deal with the what for the how part is handled by the stream processors so typical example of stateless operator like map into a flat map split marg our union and there it can also support a stateful operators such as join and window operator we're going to take a look at the window and in the rest of the in the in the later in this presentation so getting back to the previous work on example it can be easily written as with few lines where it's getting some input from data source and then doing a flat mammal and that's splitting the world into multiple splitting a sentence into multiple world and it's counting it and one important aspect there is the key degradation so the state management is handled by the abstraction itself just to get a bit into detail like we we did some partitioning there as well with key by so it has to operate operator like split and count and you can see that that up to the split operator the the word is getting into particular count operator which is basically due to the fact that we keep by the first fill which is the word in this context so it allow a particular operator to receive certain keys and but we need to be careful about this kind of partitioning because it might ended up winter if the data sq so next level of abstraction is basically high-level declarative languages such as c ql fling sequels particle these are the sequel abstraction on top of the functional api but till now it has been basically some research prototype that that was providing it if you take a look at the physical documentation it says that it's not ready yet for the production use cases so we're not going to dive into this part that much for this presentation the next thing that i would like to discuss before I explain the sliding windows is the notion of time in a stream processor there is two kind of events sorry there are two kind of time one is the event time another is a processing time and even time refers to the fact that when the even is actually happening whereas the processing time refers to the local clock on the machine when the operator is being executed one of the popular example that distinguish between these two kind of timing to kind of time is the Star Wars analogy where the the Phantom Menace was the first on the series but it was delivered in 1999 whereas the New Hope was the fourth inter-cities but it was delivered or process in 1977 so that means that the streaming system you need to deal with this out of order NIST to some extent but this notion of time is used in in defining the stream windows and stream window collect messages based on based on finite size of time frame any new messages based on a specified trigger and after that it turns the given competition on the collected messages so just to give you an example here we have said we have a stream of integers and this batch window is running on certain interval and completing the average it's taking like two elements every time and competing there ever a moment so this kind of window is also called tunneling window this window to not overlap in time does a message is processing exactly one time though then the second kind of windows sliding window whereas what it actually slides and overlaps with in time and this message has collected several messages collected which is a 40-minute talk I see that there was some confusion about the time but yeah I have some time left to discuss the rest of the presentation I guess so okay so I was I was discussing about the signing within a sliding window ambassadors can be and the window can contain the same messages like it slides over time so slanting window can overlap in time the same message can be collected in multiple windows they'll determine the windows like session window in session window it grooves the element based on the activity so it has this notion of gap in activity whenever some gap happens it start a new window there so it's very basically a bit of deterministic to some extent but it it groups element by the session activity and this is one kind of window that's provided by fling at this one so just to recap spark streaming and fleeing the board supports sliding and time windows also support batch windows there but some or other other framework like strong doesn't even support any kind of windowing mechanism by by default so you need to write your own window mechanism there some in case of sunlight only support the time with this so let's take a look at some more code in this situation we are connected with a Twitter source we are going to extract the hashtag format and then count the topic and omit the most training topic for certain time frame or certain windows so again due to the declarative Ness of the flink API you can write this code in just few lines of code in the first part we are reading something from the data source and then then what we are doing we are collecting all the hashtags and then we defining a sliding window which basically described as follows that every 10 seconds most intuitive last 60 second would be emitted so this testers sliding-window specification and then we are looking into the twitter has to feed we are partitioning it based on the tanks and counting those tags and apply this window on it so at the end it's just omitting the top to top the twits from from last 60 seconds so if we take a look at the top it's basically a very nice implementation of of this thing algorithm but it based on the based on the window it gets a collection of elements as its input and then it can you can sort that input and find out the talk talk elements from it so the sliding window or the windowing mechanism gives you this ability to discretize your stream into a certain chunk and then apply some transformation on that chunk and emit the output format but if you have to implement the same thing with stone it gets really big because it doesn't natively support you that kind of operators so you need to write everything by your own hand and then you need to maintain that and provide resilience and fault tolerance as well so there's a lot of boilerplate that you need to handle if you apply the low-level dataflow programming but as you move a bit upward and apply style using the declarative API then it become much easier to handle all kind of concern for you so but it one drawback that it has this control the programmer that is to write the low-level code now writing an abstract way which has a benefit of automatic optimization so it allow although it allowing experienced users to dive into a stream processing it takes some kind of freedom from them but which has definitely some positive side effect like automatic optimization and so on so in this section we look into different programming abstractions like the low-level programming abstraction function library and declarative language and all you also looked into like how you can in as such that it can be used it can be it can be applied in in different contexts so there are three different abstraction level we discussed and three different in three different abstraction you saw how the operator expressiveness and state management can help us in dealing with different concern concerns of stream processing system in the next part of this presentation we're going to take look into the runtime of of this this abstraction so we define a programming streaming program with some kind of instructions but then we need to be executed some in at one time so in this section we are going to take a look at the execution model and some interesting property on the segregation model in detail so there are several paper on this topic then that outline the requirement of this DVD stream processor and its execution model but in this stop we are going to focus on certain aspects of it for instance the processing guarantees fault tolerance flow control and so on so if we take a look at the execution model that is basically the core of a stream processing system it encapsulate the coordination of any streaming computation in fact different stream processor exhibit alternative design consideration but if we categorize it into two distinctive category there are two main thoughts there one is the stream data flow and second positive acrobatic so I'm going to start with the streaming data flow the streaming data for its own every event of a stream so as the event comes in it it executes some code one of the benefit of is that it is it's its syntax or its primitive tract white it is basically close to the streaming architecture assuming semantics it's provided direct mapping of streaming semantics and execution entities so the even live and clog granularity has some platform as well like you it provides the low latency but one of the problem of the streaming data flow approach is that the fault tolerance and the control flow become much more complicated the second kind of approach is a micro batch approach which basically discretize the stream into multiple batches and then that will get executed in on top of a batch execution engine for instance part streaming follow this approach one of the problem with this kind of execution model is the semantics is no longer restreaming semantics basically you start writing code on set of events or set of badge but one of the one of the positive side of this kind of abstraction is that the fault tolerance you get these fault tolerance by default from from eventually entered system like Hadoop or spark has a native support of fault tolerance and control so in this kind of model you get that automatically and also it enables high throughput there is an additional benefit that it actually combines both streaming and the batch processing together so you could use some intermediate result from the batch processing while executing this stream streaming part but one of the drawbacks is that it introduced the discretization of latency so each time there are some even coming you need to wait till the micro batches company so therefore as its provide and provide it doesn't provide low latency in this context so we do have these two different approaches and there are different streaming system that's actually used these two different approaches fleeing storm here on some degree follow the stream data flow approach whereas proc streaming and Stone Trident extension offer micro matching one of the benefit of micro batching that we have seen in recent years that they offer better processing guarantees since five swimming has been supporting exactly once for a long time on the other hand due to the recent advancement in the stream processing Flinx now also support exam exactly one semantics so next we are going to take a look at the processing guarantees so what does that mean a processing currently referred to the state of the application upon failure the execution in stream processor is a model as a dag of task when this task are interconnected the stream system deals with task failure to satisfy one of the following from its processing guarantees that is at most once at least one exactly once I'm going to define what does that even mean in this next slide so if we take a look at the task from a data flow task what does it do it basically a process some input its update the state and emits amount so we can derive to into property from it one is that guaranteed processing meaning that all the input in the task input dependency would be delivered to the task and fully processed at least once and then the second property is the consistent update each input record should lead to exactly one update in the state so if we have this to properly define then we can actually based on that we can define all kind of processing guarantees for instance in at most one situation in ada properly one or property to satisfied in case of at least once the only property one is guaranteed when in the locate might result in the replaying of some events but the system has to deal with the V idempotence idempotency in case of exactly once both property 1 and property 2 must be must be connected so based on this processing guarantee the stream processing execution engine support fault tolerance and all other kind of interesting property to enable reliable stream processing in the next section we are going to take a look at how different ystem processing system offer this kind of care for instance parks for instance don't offer only at least once but with tried and it and offer exactly one semantics sparked on the other hand it support exactly one semantics fling sample also support exactly one semantics and recently support additional support for the transaction via streaming leisure sums on the other hand has a FIFO Channel and it's opposed exactly and sorry at least one semantics so next we look into the fault tolerance part that how the system handled fault tolerance and ensured this kind of processing warranties the fault in the distributed system may appear in several different places but we generally classified two different kind of Falls for instance consider this data flow graph there that could be to kind of fault one that okay a message is failed to get in the river and the second one is like okay a node can be inactive or it can it can die so in this two situation can be handled by two fundamental approaches like even tracking and state management so what is even fracking and even tracking situation deals with the fact that is is a message might not be delivered there are two approaches for event tracking one is the repeatable stream logs and the second one is a recurrent acknowledgement and in the repeatable stream dodge logs it persists this stream into a log for instance it uses Calcutta versus devolve something use this approach so one of the benefit of this approach is that you can replay those events and ensure the guaranteed processing part of the property one that we discussed earlier in this in the second kind of situation it's record acknowledgement meaning that okay you send an even in the downstream operatory wait for the acknowledgement from this downstream operator if you do not get any acknowledgment then you stand that even again for instance strong use this kind of this kind of mechanism to ensure some kind of some form of fault tolerance it used this notion of hackers which actually keeps a tree of event that's happening a new system so at any point in time if it detects that one of the one of the even is not being delivered then it replayed and even and that's how it ensures some some notion of at least one steady very one thing so other systems like fleeing some job that has this check warning capability so this even tracking is quite related to that one as well like okay you restore a checkpoint then you start replaying some even from it in order to get to the stage where it was before the failure [Music] the second kind of mechanism that is used for the fault or an estate management in the stress management situation deals with the fact that a task can die and in that situation it wants to rebuild the state where it was before the task yeah was done so two approaches for enter one is a deep replication meaning that okay there are multiple redundant operator processing at the same time in this situation if one node died immediately the other node can start processing the events one of the problem with this kind of approach approach is the computational capability it requires it has to deal with replicating the messages to all operators and deal with it so there are a lot of redundancy in this situation but one of the plus side is that the fault owners become much easier the statement has been easy easy in this situation and recovery is also trivial you just need to swap from one filter in the passive backup situation there there is basically one node that then is keeping the state and updating the check points but as a node ty then the other node can restore the Check Point from from the stove so there is some persistent store in this situation different system like storm that doesn't provide any kind of any kind of reliable storage so it uses stone try them for this state management but it provides an API then you need to build everything by yourself on in case of stone sorry in case of spa streaming you use our DD 2 2 2 to do this kind of check pointing an RDD stratospherically check pointed to the meet the number of operation required for restoring the state sometime use Kafka simple as that it's just write everything to Kafka it used a lot of compaction of Kafka so he's heavily using Kafka for state management link all the other and similar to come I use fairly cheap ones and support persistence back-end such as rocks TP and it improved upon how the check pointing is done it uses some kind of a synchronous a lightweight snapshotting to keep the stuff and to keep the stuff in the reverse way so there are some additional nodes like for instance Meanwell and Google Cloud dataflow that use the external data sources for this persistence of the consistency has to be guaranteed by this end every morning the storage flink and sparkly streaming use the partition local storage stone harem they do not use any kind of realized storage so you need to deal with they need to deal with yourself the user needs to deal with the reliability part then there is a another important aspect of a offer of a system it is that is the flow control mechanism the flow control mechanism deals with the fact that let's say that you have a upstream node at a downstream node and the downstream node is having some performance problem so the optional should not send all the messages to the downstream in that case it might result in the drop of bad packets or the same signal can be could be storing that in its own storage and it exhausts the memory so that's why the flow control mechanism is very important and there are two distinctive way this works one is the source back pressure in this situation the overload operator sends overload signal to the source at the back pressure signal a spark streaming here on stone that we used this kind of back pressure in case of spots framing it actually since it's using the micro patching it can find out the injection rate and then from that it can derive that how how much micro I was about how much magic needs to send to the downstream node in case of Heron it used a sprout back pressure so there is any kind of problem with the underlying node then it sends initiate back pressure signal to the twelve and then it just worked from there and it also broadcasts that initiating back pressure to the other node so that others prompt can slow down when the situation is over it can send all the air signal to do that knowledge strong uses the same approach that hero news but one problem is the red envelope that one Alberto can receive a lot of input and receive input and create a lot of output from it in case of joint operation and in that situation if you send a back pressure signal to the source it doesn't make much sense because yeah it doesn't solve the actual problem and you might result in a resource accession the second kind of approach is the H by H back pressure is similar to disappear flow so a overflow operators and signals to its upstream node which in turns and signal to its other upstream node it has this notion of credit best buffer where you can you can a be a border has a limited number of credit to send information to the downstream M so if it exhausts that credit then it can't really send anymore information flowing for instance employ this kind of back pressure one of the problem with this kind of back pressure with this with H base back pressure that in the liveliness is not guaranteed in cyclic topologies so if you have problem in a node then you send a back pressure signal all the way to your own stream node which might end up in a black pressure signal coming into European that exactly the same known result in a deadlock situation so there has been lot of research that is going on in this in this area how to handle this kind of cyclic EE and pressure at this point so just to recap the flow control strategies strong doesn't use any kind of easy user similar flow control strategy from here on sometimes the interesting one that does not have its own flow control strategy it use Kafka for its flow control mechanism so it's right maybe everything it gets it right into Kafka immediately so it doesn't need any flow control by itself it with our own Kafka for that so we looked into different properties of a dataflow system but if I take a look at if we closely take a look at the trend we see that there has been a unification between the pattern streaming model when initial stream processors were domain-specific the one are very general in purpose there's a stronger guarantee that we get from steam process system like the acid transaction capabilities and the operators will be getting more expressive we have the higher-order functions and we can apply regional window specification so the primitives are getting more expressive the research direction there are a couple of interesting things that's happening one is a society centralized deployment in case of IOT applications cost efficiency so we want to run this framing system on cloud so we want to minimize our cost and therefore there has been lot of research instead of using the simple load balancing we could use slf base load balancing to mitigate that kind of issue then we need to integrate different kind of engine so there has been active research going on so given your use case different stream processor can be selected to do part of it then there has been lot of research going on with respect to for instance our unified obstructions such as beam offers so you can specify in one programming model and then it can be transport it can be wrong in different kind of runner like flink being runner or spa streaming being Roman as long as they support all the semantics that beam requires and infrastructure harnesses and other research directions like okay you just don't deploy the stuff in without--without we don't take into consideration the structure but look into infrastructure as a as one of the key component while deploying and the topology or dealing into those infrastructure so the key takeaway from this off that okay we looked into different key programming abstraction and stream execution model and a fundamental attribute of stream processing system like processing guarantees for torrent state management and flow control mechanism and we also discussed the convention and future directions so although there are different streaming present system fundamentally they offer similar capabilities with with some architectural constraint they're like processing guarantees fault tolerance and so on but the programming abstraction what is a streaming based abstraction other is a microbus base abstraction and based on that primitives are bit different but fundamentally they are offering the same thing so we need to take those into account while selecting a distributed stream processors without considering this kind of property if we just select one then might not be the right one for your situation so we only scratch the surface of this landscape I wish you all the best in your future journey in a stream processing yes if you have any questions yeah [Music] this is one of the interesting problem in that domain as well so in our situation we have a cough cough cough cluster that's running on a different data centers and I often see them from problem with the granularity of the time and then the even time we need to deal with the out of ordinary so there has been active research going on that how to I mean how to make it consistent with respect to the even time so therefore we move from the processing time to even time notion meaning that ok when the even is happening we we track that power and we use it inside the stream processing system but it's not a solved problem yet we still deal with this granularity issues yep ok thank you so much for joining this song