Robust Stateful Stream Processing with A...

scala.bythebay.io: Jamie Grier, Robust Stateful Stream Processing with Apache Flink

Recording: scala.bythebay.io: Jamie Grier, Robust Stateful Stream Processing with Apache Flink

thanks for coming everyone I'm going to talk about robust stream processing with Apache flink today before we get started quick background I'm Jamie Greer I'm the director of applications engineering and data artisans data artisans is the company everyone hears me okay yeah data artisans is the company that formed around Apache flank it's the co-founded by the some of the original creators of flank little background on me I've been working on stream processing of various forms for most of my career I did work here at Twitter for a little while on the streaming computing and on some other things I also worked at a company called which is like a basically a big stream processing engine is the the core of the company and before that I was working on like ultra-high speed video processing which is a bit different but still a lot of the same kind of algorithms and stuff so that's kind of what I've been doing most of my career what we're going to talk about today is first of all just a super quick introduction to Apache flink we're going to move real quickly through this stuff we talked about what is stateful stream processing and how is that different from kind of the previous generation of stream processor we're going to talk about windowed computation over streams the reason that's interesting is because if you have an infinite data stream and you're doing something like just some simple statistics like group by key and count obviously you can't compute a sum over an infinite data stream unless you win to it so you have to define things like windows and stream processing it's something that comes up all the time so you have to be able to handle that stuff properly then we're going to kind of get into a little bit of the meat of the talk so we're going to talk about robust time handling we'll get into a little bit the idea of event time versus processing time and what that means we're gonna talk about failure handling so in the context of stateful computation not just simple like stateless services but in the context of stateful computation over streams how do you handle failures a much more challenging problem when you have state we're gonna talk about then how do you handle like production oriented issues like planned downtime upgrading your job upgrading the cluster upgrading flink reap and then finally also we're going to talk about like reprocessing data so this is something that if you're really running things in production comes up all the time where you need to return like say you discover you're interested in some new aspect of the data stream or you find a bug in your code and you want to back up to some previous time and reprocess all the data from there that should be a you should be able to do it correctly and B you should be able to do it fairly easily so I want to demonstrate how that works in flink and think about as we're going through that we'll talk about you know things get a lot more challenging when you're doing stateful computation so if you're gonna back up to some previous state in the system there was no point in time when like sort of you're at the zero state except for maybe you know a year ago when you first started this job everything else is stateful computation after that so if you're gonna go back and reprocess data you need to go back also to a consistent previous state and reprocess data from there so pretty challenging to do if you don't have the right tools we're gonna talk about that okay just for reference are there many people in the audience that have heard of Apache flink so far looks that's pretty good and is there some general familiarity with stream processing with other frameworks and stuff okay cool that makes it a little bit easier in general flink is an open source platform for a distributed stream and batch data processing most people probably don't know this about flanks everyone's really interested in the streaming space these days but plink was actually created as kind of the the the idea of the project this was five six years ago at the University Technical University of Berlin the idea was what if we apply a streaming data flow engine to big batch processing right and the reason to do that is because rather than like if we simplify the problem and just think of map and reduce rather than like have all your mappers run and then they generate a pile of data and then have all your reducers run it's a lot more efficient in terms of use of hardware and like that in the end time to compute some job if you actually pipeline the shuffle between the mappers and the reducers you don't map and then reduce the reducers start to run as soon as the mappers start right as soon as the mappers are producing data so that's applying a dataflow engine to this batch processing idea so that's actually the genesis of flink from from years ago but it turns out if you've already built a streaming dataflow engine guess what it's super good at streaming data it's a great choice for an architecture so these days most of the focus in flank is around stream processing that's kind of at a high-level view not a super interesting diagram but a super high-level kind of what it looks like the main point is that the core is the same it's a run it's a streaming dataflow runtime there's the data stream API on top of it which is kind of like the stream processing API or to get more clear about this concept it's an API that allows you to deal with unbounded data sets right data is continuously being produced and you have a continuous computational model over the data that's the data stream API the data set API is when you have a bounded data set and you want to do things the the api's are very similar but in the data set world because you know something you know something about bounds on the data you can use some different algorithms if you're doing big joins and things like that like a typical batch processing workload but api's are slightly different very very similar if you can use one you use the other and the runtime engine is streaming data flow engine no matter what just with some different algorithms plugged in around joins and things like that to make it a good batch processor what we're gonna talk today about or about today is really the data stream API because that's really what motor get gets most of the attention and focus these days layered on top there's also a complex event processing library you can think of that as like basically a state machine over streams so you have a stateful stream processor it's really nice and easy to build correct state machines that are that are still correct in the face of faults and everything so it's a great it's a great CEP engine because of that and then there's a relational API there's actually missing from this diagram there's a sequel API now so you can express your streaming programs as sequel machine learning graph processing and then on the back side there's also a table and relational API the core of the technology what gets the most focus and development in the project is really the these api's in runtime and then there's these other projects also on top but the real technology is in yellow is in my opinion oh and then of course on the bottom there's just a lot of different ways you can deploy flink and this is a little dated you can deploy out with docker you can to play with kubernetes meso s-- yarn you can have a standalone fling cluster as well if that's what you want to do and there's improve the yarn support is really good already like it's a it's just hey I've just got this job run on Yard and everything just works running on meso soar with docker takes a little bit more effort from you but people do it all the time and then you've got coming really soon in flink 1.2 which is a several weeks out from now they'll also be first-class meso support so you can just like hey this is my library run this on meso somehow and it just works uses Makos to acquire resources and everything else ok this next little section it's good that there's a lot of experience in the room this is gonna be the world's quickest introduction to stream processing and then we're actually gonna for the rest of the time we're gonna write code and like demonstrate things and it's gonna be not too many of these slides but what the heck is stream processing this is the simple version you've got a data stream which is an infinite an infinite data set data is just continuously being produced and arriving all the time it's unbounded you've got some code that you want to run some kind of computation you want to run over that data stream and then you want to produce some sort of output as another data stream often as another data stream not strictly necessary but yes so that's kind of the model of early stream processing and you hook a lot of these things together and you get these topologies of these streaming operators and you can build kind of cool stuff this way however what's missing from the picture I mean what's missing from this picture is state if you don't have any state you can you can't do stateful computation correctly you can do you know really simple things like ETL and filtering and maps and those kinds of things but you can't really you don't really have a full computational model so stream processing becomes a lot more interesting when you have state right same basic model consume infinite data streams you've got some code here but that code can treat can can keep track of State so now your output can be affected not only by the current input message but all messages you've seen before this if you think about it this is equivalent to like imagine if we had to write code with no memory with machines with no memory right if you wanted to be if you want the thing to sort of function correctly you can't really have state because sometimes you just lose it it's a pretty uninteresting programming model this is a full programming model you can build anything you want with this and we'll talk about and I'll show you as long as that state is fault tolerant even when you distribute across lots of machines you can build anything you want so that's what stateful stream processing is by the way if we're talking about state just to get just to put something concrete here if you're doing something simple as consuming a stream group by key and count that's stateful computation right the current count for every key is the state if you don't manage that state fault-tolerant you can't even do sort of distributed counting not correctly right so you need you really need state for interests for a stream processing to be truly interesting okay slightly more complex example like I said you've got sources of data streams on the on the Left sinks four data streams on the right you know RabbitMQ maybe HDFS files maybe Kafka doing some things in the middle here's a stateful aggregation then you're writing to a database may begin that's like a really typical stream processing topology this computation in the middle can be anything and then we don't just want one flow one thing like this but we want to use loads of machines distributed this stuff across loads of machines and process like this at scale so that's what flink is for that's stateful computation model over streams and running a very high scale you can do all of those things with Lincoln that's kind of the point benchmarks are what they are take it for what it's worth but I did some benchmarking which is basically consume but the typical Analytics workload consume from consume a stream group by key and then do a windowed count for every key and on ten machines with 40 giggy interconnect because often flink will just saturate the network that becomes the bottleneck right away so when he got to the point where I had 40 gig e interconnects between all these machines I was able to run 72 million messages per second on these 10 machines so it's super efficient and I think about this if you think about this group by key that means full shuffle of every message across the network right it's not that's exactly it's just a simple computation just the whole idea is to sort of measure what the stream processor can do right so what's important in stream processing is essentially parallelism group by key is super important because that means you can repartition the data as needed so that's like the big shuffle over the network and then in this case a windowed count so for every key for for this hour what's the count right that's the computation pretty simple it's a real standard sort of analytics type application but yep um group ID I'm I wouldn't say that actually so like if you just wanted to know you know for each of my ads how many clicks per hour that's group by key so you do need to you do yeah whenever you do stuff like that partition by key you do after you worry about data skew but yeah that's none of that changes in stream processing it's the same it's the same yeah okay and now we get to the meat of the talk and we're gonna actually just write some code this is where it's more fun okay so this just sort of like a total stub of a flink job it's just a main method you get an execution environment I set the parallelism to 1 because it's easier to look reason about this during demo so when we you'll see why and then I'm gonna execute the program alright this is this this is the stub of a flank job you can't get started with flink just like this on your laptop you don't have to install anything you don't have to download flank you can actually just take a maven dependency and start writing code like this and when you run this thing in your IDE it'll spin up a cluster and embedded cluster and actually execute your program on the cluster so you get all probably if you wanted to you could all be like writing flinging programs 5 minutes from now super easy okay so we're gonna define a stream I'm gonna cheat a little bit I have a data generator to make things simple and I'm going to simply so you can see what the data is like I'm just gonna print whoops and I'm just gonna show you what the data looks like and then explain a couple things and then we'll go from there ok can you guys actually see this in the back ok the data is super simple it's a time stamp some key in this case temp pressure or door and some value the idea of this exercise is that these are sensors maybe IOT sensors whatever you've got a door sensor which is going to be like an open closed thing a temperature sensor and a pressure sensor which are going to be just a sort of analog waves right so that's kind of what the data looks like it's probably a lot more interesting to see it visually so that's what we're gonna do with it next take that same stream and we're gonna add a sink to this job and we're gonna write it to in flux we're gonna take a look at that if you're not familiar with in flux it's time series database it's got a really nice flexible data model which I like a lot I don't have a ton of experience trying to use in flux at any reasonable scale but for these demos it works great I really actually like the data model a lot and the cool thing about in flux is we can sort of just grow fana on top of it and quickly be looking at the data so it's kind of nice ok so you've got a temperature sensor pressure sensor and a door sensor that's what the data looks like it's all coming in as like one stream of data and then we're partitioning it by key and doing an graphing it here ok that's so that's what we're looking at and then we're gonna do something a little bit more interesting with this what we're going to do is build this so a really common pattern in stream processing and it's also a good thing for demo because it's very obviously stateful it's easy to see what the state is is this pattern we have like a domain data stream and a control screen you're gonna join these two streams together and there's gonna be state I'm gonna explain to you what that is and there's an output stream so I'll give you an example say you had a topology that was filtering data right but you want to sort of dynamically control what all the filter rule so imagine this is this is ad clicks and this other stream says hey some user writes into this stream I want to see I want to see information out the other end of this filter about this set of ads the set of ad IDs so this is the main data stream this is a stream of essentially rules that we're going to use to update this this computation here so that we're sort of dynamically controlling it using the state and then we gonna have output in the case of our example that's like more real-world in the case of this example these are those waves this control stream is gonna be it's going to allow us to control the amplification factor for each of those keys individually and then we're gonna have output I'll show you it's going to make more sense in a second but this is a really common pattern and building sort of interesting dynamic stream processing applications the way you do this in flink is going to be first of all I'm going to add the second stream oops for this one I'm just gonna read from a socket which is that you can flank and I'm gonna actually you can see whoops let's see we build blow this up so that I can sort of control I'm gonna type messages here so I can control for each of these sensors what the amplification factor should be and we're gonna literally be able to dynamically change our waveforms so I've got this control stream now I'm just gonna read in from that socket next thing I'm gonna do is sort of essentially parse the message now I've got now I've got a mess sort of a stream of control messages by the way everything is strongly typed in Flinx so what's the type of this thing data stream of control message right if I did another map step here you know something like the type here is gonna be string so make sense to everyone so if link is strongly typed all the way through you use your types you don't have to use like weird you can use tuples and stuff but you don't have to do that like some sort of big data frameworks you actually use just like strongly typed all the way through and also because we're using Scala here we don't need all this cruft which I love and we can go back to this cleaner looking thing okay so we have this control stream and we have the original stream what we want to do is I'll come back to this keying we want to connect the two and we're going to connect and then from there we're gonna execute some function and then we're gonna add this we're going to add a sync again we're going to write to influx I'm gonna give it another name and that's pretty much it so what we've done is we're reading the made the main generated data in the real world maybe that's coming from HDFS or Kafka to pick off really often Kafka we're gonna write that raw data out just as it is we're also gonna read a socket as a stream I'm going to read from localhost and parse the message so now I have these two type streams I'm going to partition by key and I'm going to partition the control stream also by key so that's scale if you think about this you know you're consuming all these files of data your group by key that's really sort of in a sense a routing function so all the keys with the same value end up going to the same host I key the other stream exactly the same way so all the interesting data you know if I want to do I want to keep state per key if I keep both streams the same way they can share state does that make sense to everyone more or less so in really what you're doing when you're doing this step this step in the updating state you're creating you kind of in the stream processor you can think of it like a distributed key-value store that shard it on the key you shard the stream on the key and you shard the state or the other stream on the key and all this stuff that kind of naturally goes together ends up on the same machine so you can do things like stagger against stateful computation all that stuff but you've got a sharded state store throughout the stream processing topology okay so you can build a lot of cool stuff with that but so we've connected the two streams this stream key by this stream key by connect the two and apply this function so we when we run this guy you see down here same waves as before but we also added this other this is the output of the amplifier function that graphically looks like that right that's what we just built exactly what we just built and now I can do things like for the set the amplification factor per key okay so I just dynamically changed the output I sort of programmed my amplifier function dynamically and there's state held per key which is that this amplification factor for every key there for every sensor is everyone with me the state information is stored locally throughout the stream processor right exactly that kind of key by function also partitions the state right so you can scale horizontally so it's stored locally there's a number of different ways it can be just stored in RAM it can be stoic and if you want to get bigger than that you can spilt a disk it happens to use and as an implementation detail rocks DB it's not important at all it's just a way to spill from Ram to disk but for it to be fault tolerant you have to make snapshots that's we're not going to have a ton of time but maybe we have time at the end to talk about how the fault tolerance mechanism really works I'm gonna show you how it works and I'm available after - but it does this thing called lightweight asynchronous distributed snapshots and flink makes essentially snapshot copies of all of the state of the whole computation periodically and it does that in such a way that it doesn't interrupt the data flow and it doesn't introduce any latency and what it also what you can get from that is actually around all this state that's being computed in the stream processor you can get exactly one semantics it doesn't mean exactly once message delivery what it means is that the state you're computing right so if you're counting the counts for every key you can compute that state correctly even in the face of failures it's as if the the result will be exactly as if each message was processed exactly once so that's what's called exactly once guarantees in the stream processor what's that the this key we'll just have a new state and that let me show you how the function works I think maybe it'll be more clear so that's the door if I say door 10 so you just changed so I guess I literally four keyed four key door I'm now storing a value that's 10 and that's the yes yeah in this case because that's the topology I built right that's how I designed it no yeah I think we should totally talk about at the end it's probably the most interesting thing about flink but it doesn't really fit in the context of the rest of this but at the end I'd be happy to talk about exactly how it works okay what's that the graphing the the nice dashboard is graph on ax and is backed by in flux DB yeah no it's completely continuous processing yep there are no there are no batches of any size yep event my event yes let's do that thank you okay it's super simple okay this is the amplifier function connect and flatmap right we connected them and then we have a flat map function this is sort of like the left hand side and the right hand side this is the control stream with how we saw how it was all typed right the control stream and the main the main data stream which is just of this type data point which is not a flink type that's just my type for the demo okay so all we're doing in the control stream is we're getting state called the top line here that's a state descriptor it's called amplitude that's just the name I made up it's got type double and flink will manage that state for you this is an example of using you that state can be anything you want as the user this says get this get that state and update it with the amplitude from the message right so we just updated our state super simple the when we process the data stream side because we've connected the two process the data stream side when you get a message on the data stream you get the current state which is the amplification factor for the key and you say you know multiply right that's the whole amplifier function but the point is it's completely stateful and it's distributed and partitioned at scale so this will work if there's you know 100 million keys yep you so you understand flatmap the fact that it's a connected stream it's a way to process continuously on both both of the over both of those streams the whole point of Connect is really that you can share States between the two so that's kind of that that you know input the control message side and the other side and no there aren't three and four and five versions I think in a future version of the flink api there will be something like that but it won't be static like this you'll use annotations to yeah so be a lot more flexible I can do whatever you want it's just it's a full stateful computation level so you want to do you want to keep two values for every key okay so you want to you've just queried this line get the current state right line 21 on the control value side okay so you have you want to do that and then you want to do yeah I did jump the Java yeah that's true that's the old value then what you want me to mix the two somehow that yeah yeah yeah this is a query that's exactly the same as this that's a query right it's flink state so there's a such thing in flink as a state back-end right and their pluggable so it just depends on the state back-end exactly where that comes from but the point is flink actually controls it it can guarantee the semantics around it that's why it has to be a first-class citizen in the API flink has to understand state it's the only way I can make it fault tolerant for you that's what's that no exactly not so you want to partition it right so you key by partition everything right and so you can scale horizontally and so you partition the stream and you partition the state as well you've deployed these operators across like 100 machines there's local state on each one right charted by key I think it we have to move on we can save time at the end though there's a lot to cover that gets more interesting than this ok so that's this program running right the other thing I said it was really common to do in stream processing is windowing this has got to be done correctly because you have to do it all the time most interesting applications have Windows somewhere so I want to do a window to so you can see how it works so let's add one more thing again we'll take that stream key by and then let's compute a time window and then compute and then have a reduce function and once again we're gonna add we're gonna write it to influx so we can take a look at it we're gonna call this the summed sensors okay makes sense that's a super common type of analytics things to do it's that many lines of code because it's kind of flink gives you those primitives so consume group by key so for every key compute a time window and then reduce over that window and compute something whatever you want here I happen to just be summing things up and then I'm gonna write it to in flux that one is a tumbling window okay there's also sliding windows tumbling means you know it's every second sliding means they overlap so you have a if I add a second parameter here I can also add a second parameter here with a slide time so compute over the last hour sliding every minute whatever you want you can also do session windows which are totally data-driven which is like imagine like typical like website user session tracking user comes to the website a session starts they click around for a while some period of time goes by that's completely indeterminate but they're not active and then you sort of say okay that's a window and then you compute analytics over it by the something that's almost almost completely impossible to do correctly with a batch processor or any kind of mini batch system because the the window is completely indeterminate with regard to any kind of boundaries so easy super easy to solve in real stream processing really hard to solve any other way so you have session windows you also have custom windows so you can you can actually decide how the should get windowed when to trigger when dis trigger output all that stuff so this is kind of like the highest level of if the API but it gets more advanced okay so that let's run this and see what happens five minutes see too many questions we have a lot to cover so I'm gonna go really fast now okay so let's see what's going on this is interesting I'm doing a sum over the middle one's a pressure the pressure sensor is a sine wave I think it's a perfect sine wave it's a line on second boundaries its period is one second I'm computing a sum right here what should the sum over a sine wave be over its period anyone zero should like be approximately zero it should be exactly zero exactly zero does anyone have any idea why that's not exactly zero any guesses what's that exactly exactly right this is exactly the difference between processing time and event time one of the things and why it matters so much what we're doing here is really naive we're actually just drawing Windows based on the wall clock time of some machine and we're throwing data in that window and then like when the window is done we compute some result and that's the answer but sometimes data is delay that comes late it comes in bursts it's backed up in Kafka it's backed up in flume by the time it gets here god knows what it is and we're just kind of using machine time to like compute stuff not very smart and second thing is like say what if we want to rip through like a whole bunch of historical data the value here is approximately 4.5 right if we rip through a bunch of historical data at 500 times the speed we don't want this to be you know 4.5 times 500 and that's the sum we compute that's processing time and it's a very sort of inaccurate naive approach to stream processing this is this is what we want to fix I don't think there's anyone in this room right after us if we go a little bit long we have a little extra time this is what's happening in Green's the timestamps what we want to do is draw Windows like this by looking at the data and we want to do this exactly correctly and what we're actually doing is sort of drawing the wrong windows because we're you we're not looking at the timestamps in the data we're just kind of like doing it on the clock on the machine so that's what we're doing that's what we want to do and by the way we want this to happen no matter when the data comes in we want to happen no matter if the data is totally miss ordered right because it always is in the real world the data is totally miss ordered we still want to window it correctly regardless of the fact that it's coming out of order that's what event time is all about it gives you this repeatability indeterminacy and correctness and that's why the event time is so important if you're doing anything like analytics or you want your computation to just be correct and repeatable you definitely need it so to fix this in flink we're gonna do the following stop the previous job run this job see what happens and let's see if we get exactly the right answer okay there's some interesting whoops yeah I sense what I've done the right I think the best way to think about this problem is you've got this unread message data but you've got time stamps in it it's completely out of order but what you want to do is take the data and from just the data extract a clock a disciplined well-mannered clock that events is monotonically from this or unordered mess right so the data itself drives the clock and then everything you do with time you use that clock that's kind of like I don't know an engineer speak that's I think the best model for thinking about it so now if we just run through data at a hundred times the speed doesn't make any difference it's just a clock tick tick tick and if the data is unordered assuming we can build that nice clock from unordered data which you can in flink you get the same property so then anything else we do in the computation that has to do with time we use that clock instead so that's how it works at the end that's another really cool thing I think the two things in plink that are super cool as the base technology or how do you do that how do you generate it's called watermark generation and how do you like create global time from all this mess and to how do you do these asynchronous distributed snapshots so all the status vault talk those two things are what was kind of the base of like wife link is so interesting I think yeah and it's more than that to because it drives Flinx operations by looking at the time in the data but yeah the data Tom it does look at a timestamp in the data to figure out essentially what time it is all right so errors versus accuracy is the punchline Phil you're handling let's do this quickly I think this is super important to obviously so to do this the only thing I'm going to do different is rather than run it in my IDE I'm going to start up a Flint cluster I'm a laptop and then I'm going to build the same code we've been playing with and I'm gonna run that code on the cluster okay this is what the flank UI looks like by the way it's running one job it tells you what tells you about resources it has in the cluster it actually we're not gonna get into this right now but flink read your program which has got that nice sort of declarative API it turns it into a job graph a job graph is what flink understands and those how to execute that's that scale so that's a job graph that's our job graph and so what I'm gonna do now is cause a failure and the way I'm gonna do it is pretty brutal I'm just gonna like go and kill a task manager which is where you're kind of like the worker code runs this is exactly the same as like unplugging some machine that's what I'm gonna do I'm gonna kill - okay so Flinx is gonna tell you in a second it's noticed that that job is down it's gonna try to restart it that's tunable by the way I have it set sort of high so you can make this I shouldn't fix that but they're so Flinx notice that that job is down it's gonna try to restart it right now if you're running on yarn or maysa or something it would already been back up and running right cuz it's a really relatively small job it would have been really quick right now though it's waiting for me I'm like May so so I'm like yarn so it's waiting for me and the other thing is happening is we're not getting any data that's that's not good if we back up and look at more you know we've been generating data all I just zoomed out in time so you know we had we had some interesting stuff going on but then we we kind of went down and we lost all this state or we're gonna lose all this state here I can fire up a job now alright sorry I can add resources back to the cluster because I'm I'm ASOS and then flink is gonna figure out that it's got resources and that job's gonna start running again and there it goes it's a it's producing data obviously but we've lost all this we've lost our state which is bad and honestly there's also like look there's that big gap in data we don't want that either we want to be able to like we want to not be able to tell that there was a failure like we want to be able to compute exactly the same thing as before as if nothing happened so all I'm going to do is turn on this cool this asynchronous distributed snapshots algorithm that I talked about there's never a read like just to be clear this is only a demo you never run without this turned off there's absolutely no reason to write and we can talk about the cost of it later but basically you can do this without introducing really any latency and it doesn't really affect other than this if the state grows really large it takes some throughput on the network from you but other than that like it doesn't really introduce any latency or anything so that's kind of the cool thing about flank so if I run this again I know we're out of time let me build the code with that one change flink list tells me another running jobs if you're not into the UI kill the currently running job submit the one just without one line enabling checkpointing okay now we're at least back up and running right could be submitted this job let me give you some interesting state so the door one pressure to temp five let's be crazy okay so we're kind of running again maybe we look at the last two minutes okay so we're in the state now let's go and kill a test manager same things before holy crap we're not getting any data what's wrong big failure and then again flink is just waiting for resources it's trying to restart the job well we expect to see this time is that it's completely impossible to tell that there was ever a failure right all the in flight state will be will come back all the in flight state will be picked back up think about this like when I killed this thing there's a bunch of Windows just kind of in flight wherever they were they should be picked up right in the middle and just everything should just continue and all of that so now when I run what's that yeah I will I swear okay so there you go that's complete stateful computation even though we lost a machine for a while and everything picked up just like it was before not even a blip like imagine these are all those windowed sums right not even a blip anywhere and those were all in flight when we feel what happened in a nutshell is like there you're making these snapshots all the time right it's on some frequency when some machine fails you you're you you go back to basically the last good Check Point the whole thing just kind of ticks back to this consistent global state and just continues computing right so in a nutshell that's what happened and so when you tick back the the source operator is also stateful so it just ticks back and it continues in a deterministic way to generate the same sequence of messages at would have the first time so when you roll back sort of in the input stream you also roll back the state to a consistent point that's what gives you exactly once that's what gives you this behavior that's what exactly once means I'll you know basically in-stream processing it's not message delivery its computing the same result as if each message was processed exactly once it's a bit confusing but those are the terms that have been become popular in stream processing unfortunately it's as if he's meshes was processed exactly once each message is guaranteed to get applied to the state your computing once yeah exactly I would say the state needs the source needs to be stateful it needs to also be stateful it needs to be able to return to a previous state and then continue in some sense yeah yeah is that if you've end up processing processing two messages one message twice you've rolled back the state when you willed back in the input so they go together it's consistent right that's why yeah yeah and so here's the thing the source they'd like this durability idea is not really flink doesn't really care about that it's just that in this case say you're reading from Kafka the state of the source operation the stop stores operator it's just a stateful operator like everything else the state happens to be the offset in Kafka that's all it is so when it rolls back to a previous state it effectively backs up in Kafka and then everything rolls forward so at the end there's one more cool thing to like finish this sick demo alright there's one more cool thing I swear it's worth it so now we're doing this we're in this state full we're marching along I want to do one more thing fling save point and oops okay okay this always happens to be okay think safe point what that is is you just took a global snapshot yourself it's built on the same mechanism but you control this thing we now we have this global snapshot we can create as many of these as one and we just keep them around so what I'm going to do with this is I'm going to destroy the whole cluster right at this point everything's down of course because we don't even have a fling cluster and like we're not getting any data so think about this just for a second there's not a fling cluster anywhere we don't have any machines we don't have anything but we have us this snapshot let's say that we happen to have this data replicated into another DC and say that yeah we have access to both to the input source like Kafka and we have access to HDFS where we stored this stuff and it's replicated so we moved to a whole different DC say spin up a new Flint cluster maybe a new version of flank maybe a new version of your code whatever just start up a new cluster and we say flink run as before but this time we're going to say from this safe point right here and then the same as before run my code I check this out there you go so that there's a period of time we didn't have a blink cluster and we just continued this whole stateful computation without a hitch in the real world that's important because what you're gonna do if you're gonna upgrade your code say and fix a bug it's flink make safe point stop every stop your job upgrade your code blink run from safe point and you just continue without losing anything that's a production feature that is really hard to come without this you can't ever claim to do like correct stateful computation over streams you got to have it because in the real world you got to do this reprocessing I said I was gonna talk about is the same thing just make save points every so often and said like maybe I made a save point three months ago and I say flink run my job from save point and go and just use a bunch of machines and just rip through all the data again caches back up the real time and then you just continue processing so that's it for questions we can do whatever we can move to another room but I think that's the end of the the demo that's [Applause] you