scale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
you all right thank you very much Alexi for the for that introduction yeah my name is my name is Stephan I'm one of the folks who started flink a few a few years ago and together with other other people that you know started started Apache flink we actually founded data artisans together what we now do most of our time is actually continue to develop the open-source project and what we do have is actually a commercial platform built around Apache flink but yeah today I'm going to talk about the open-source part actually so I think Alexa kinda asked me for something in in that direction like what is if I can share some thoughts on what I think is the next wave of stream processing applications yeah so I want to structure this talk actually in in in two ways in two parts first of all kind of review what stream processing and flingers is especially because I think stream processing has become more than a lot of folks that I often talk to actually think that stream processing is it's actually a much broader thing it's a it's not just some real-time analytics and talked a bit about our Flinx approach to stream processing because that's also a bit different than for example you know structured streaming or yeah heron or so and then actually from there see like okay if that if that is what stream processing is today what can we expect stream processing to be to be next all right so if if everything works well I'll actually do a demo in the end that depends a little bit if the Amazon set up actually stays up and running and so on for for the time in order to start this in order to start this um this these are these thoughts off like what is what is streaming now and what is streaming what are the next streaming applications what can we expect there what I found very helpful is again to to ask actually this this question what what just faster your data or your query so if your use cases actually something where you know you have a reasonably slowly changing sort of yeah a set of data files or so many very fast changing set of queries something like you know you have you have a you have a snapshot of your data and you're you're digging through it let's say with the dashboarding tool that in the background runs a lot of sequel queries or you're training in a machine machine and machine learning model and you know in the course of training you're doing you know cross validation or not that which translates down to are running the same thing very often on different you know different sub samples of different parameters and so on that is one one class of problems the other class of problems is where you really have continuous continuous applications there are pipelines which tend not to shift every few seconds so so I'm standing queries um things like machine learning evaluating the machine learning model over the data streams anomaly detection all of that so that the the left hand side is really classical batch processing and the right hand side which I would which I would argue is actually a huge a huge portion of applications and I want to especially highlight the first part their continuous application because a lot of things that we we built and keep running are actually in that category in some sense all that is actually stream processing to to the approach that we're taking to it so we're we're you know a batch processing architecture just classically it looks like a data at rest architecture all data is usually born as a stream right so terabytes of data don't instantly appear they usually come over time as sequences of large events answer events and all of that and the the core idea of of you know off stream processing from from 10,000 feet is really just keep the format of streams all the time put some computation onto those streams and and go on from there so let me quickly ask you who here has actually heard of link or knows roughly what what fling is okay so as I spend at least a few minutes on giving a rough introduction into what it is without without being able to go into too much detail but just give you a give you a feeling of what this what this thing is and what it does ok so the way we like to describe link these days because it's actually evolved quite a bit since it was first started is a system for stateful computations over data streams so into the system go data streams as you know as as what you may classically think as a stream something coming from you know Kafka or RabbitMQ or I guess in Twitter's case distributed log but as well you know flink interprets basically historic data also too as streams that have been in all act back up backed up streams that have been put to rest but it retains the the idea of of viewing these as streams actually link itself runs runs computation you can actually you can actually do some extent even query the internal state of computation so you can actually process something and can ask it at any point in time you know what is the what is your internal state what is that variable and so on and well it yeah it outputs streams it can actually output you know like data data to rest and file systems into databases or more and more these days what it does it actually really just calls into applications with the results of what has been computed over the data streams and so it it actually covers that is really important to to point out here it covers real time and historic data streams and it's built it's actually built with I think a pretty a pretty good architecture that that actually covers many of the many other things you want from a modern stream processor in pretty good ways it's fast at scales it scales well it's fault tolerant with exactly ones guarantees it computes in memory as much as possible but you know it can also go you can also plug in storage for intermediate state that actually goes to disk so you can actually also exceed your memory if you want it actually it does scale to a lot to a very large in process in process state that is one of its really strong points and it's built with I would say like next to Apache being the most sophisticated event time support that you can get these days so the if you look at the like house house flink built it's built on it's built on four key ingredients I would say so the first thing is the first one is streams then state event time and snapshots and I mean let me just quickly say to say two sentences to each of them so the event stream sport means it's is actually it's modeling internally everything as streams and it tries to keep data moving all the time as much as possible so it doesn't in it doesn't introduce artificial boundaries and it actually treats batch processing also as a stream processing case so a batch is basically a bounded bounded window of a stream the the important part here is is again artificial boundary so it gets actually very low latency it you can you can think of it really as a as a continuous application to assess a piece of you know application logic business logic that runs continuously in a container as I said before it has it has a very sophisticated support for state state meaning really that at any point in time when you receive an event what you can do is you can not just look at that individual event and and do something with it but you can actually look at you know at your context let's say you're doing something as simple as you know counting counting events by key then that state is your current count for a certain key and the important thing is it manages this all internally so it doesn't go to an external data store or so in order to do that but does it internally and it does it in a it does it actually in a way that you know you can you can actually rescale it you can go in memory can go out of core it and it gives you it gives you a very high flexibility and how you can keep that state so it's actually way more than just you know just something like key key value pairs or so you can you can almost keep let's say arbitrary and process data structures fault tolerant the third ingredient is actually time and event time so for those of you who have actually followed the stream processing space a little bit and I would say especially that you are last year have have probably come across one or the other talk from from the Apache beam folks from Google that have actually described this all idea of how to model streams as you know as as streams of events that kind of have to time characteristics when are we processing it now processing time when it ended it originally occurred event time how do we use this to you know build sophisticated semantics so if link implements a a very sophisticated model there as well and actually yeah actually allows you to to Bill T's restreaming pipelines in such a way that you know they become actually consistent in the presence of out of order events and they they work really well on replaced and so on and the last one is what I'm going to talk about a little more now because that's really I think something pretty unique to flink is the way it it does it has poor tolerance by distributed snapshots so we're going to come back to that in a bit so how do you actually how do you use flink when you when your programs link these days there's multiple layers of abstraction that it gives you to actually develop streaming programs the the core API is the data stream API these days which is the API for a stream and that batch data processing and that one looks I'm not sure if you can actually see that here it's a rather small print it it looks roughly you know as as data processing every eyes look these days it's functional programming you can actually from that from that level and it has all these high-level constructs and are you working on streams with windows with with are you know with these time characteristics and everything and state what what you can actually do is you can at any point in time say I'm actually want to go a little a little more low-level in in the abstraction that I get and just want really really raw the concepts of events time and state that is that's what we call the process function at any point in time you can really just look at an event you can look at arbitrary data structures manipulate them you can schedule basically you can schedule callbacks for when the system has reached a certain amount of progress with respect to you know data data event time data completeness or just processing time if you want to keep in to within certain SLS and that is really that is a very um that's a very very flexible level but a bit more a bit more verbose on the other hand on top of it is also another abstraction streaming sequel and dynamic tables which is really the use case for yeah for for real-time analytics where let's say the questions are are more of yeah of a classical analytical statistical nature and that it also supports a lot of different types of UDF so you can actually it's actually very expressive level but if I have to divide kind of the the user base of link into who uses who uses what level then I would say the more you are really on but the more you're really on let's say streaming as as you know you come from batch processing and maybe SQL and you just want a bit more real-time there then then users cannot gravitate I would say to the higher levels there's a lot of users that come actually from the context of building you know stateful applications micro services and so on we actually don't you know you don't you often don't easily put this into a sequel statement but what you actually need to do is you need to run like a business logic complex per event per element most of those users actually gravitate more to the to the to the lower levels to the more flexible yeah there's a there's a good bunch of users that have actually implemented very very impressive things on on flink and and the last and the last year I would say he's just she's just a selection of I would say a four for ones that are I would say running very impressive projects the biggest contributor to flink and the biggest user is probably Alibaba these days also although it's always a bit hard to get exact numbers if you've actually like roughly a week ago there was this crazy shopping holiday single stay in China and I'm not sure if you've actually just just followed that a bit they have these you know they have these stages with their trial where they're tracking in real time what they're selling what the what the campaigns are are outputting that thing is actually the other thing was powered by by their application on top of link and at the same time though the whole real-time search during that during that single state it tries to pick up you know like shopping trends adjust the search index accordingly you know basically recompute the search rankings all the time that thing is also based on you know on their streaming platform on top of link a similar way actually uber least a few weeks ago a the system that they're internally using as the streaming analytics platform it's basically a streaming sequin abstraction that they give to their to their users to their you know both both engineers and data scientists - right - right basically real-time SQL statements that then power applications like estimating the ETA for reads or something like that they actually built this also on top of on top of a passive link and similarly actually Netflix current effort to build its distributed streaming platform that they basically offer as a service internally - to users as best as based on by some flink and ing in a very similar effort for fraud detection and for basically just you know data-driven financial services I would like to - basically okay so what what is what is actually the what is the what is the compelling thing about about about flink and what does this tell us about where we're are where are the next you know waves of streaming stream processing applications coming from and so on I meant and to do that VIN - that I would I would like to you know take a quick step back and and just look a little bit at at the way that no distributed applications are built in general because all of these things that we're that we're seeing there is basically distributed data processing infrastructure and if we if we look at it maybe it's probably not even that long ago maybe one and a half decades or two decades ago a lot of it really looked like this wasn't and it wasn't a very very distributed architecture you know it was a central database and all the applications were worked in the central database and you always had to get passed to people for every change that you made right Larry was always smelling dollars and the DBA was always showing you the door when you wanted to change you know add a column to the table or so because really that guy had to you know manage this one thing and if it went down then the company went down and you know there have been of course the micro service trends and so on there was one of the first steps or you say you would you tear the big central database apart into a lot of smaller databases but the the next step that is actually happening that I think is very exciting very promising and the future is actually taking the database apart as a whole and and and you've actually you've probably actually seen seen this let me let me take my every giving my view on this really quick this picture is taken from the docks of Apache Cassandra but it's it's in some sense kind of think representative for how a lot of database is log internal so what you're doing is you're receiving data that is written it goes to a commit lock and then it is and then this data is actually used to compute some tables in or some parts in memory and some parts on disk in order to actually show you you know what is the state of the computation so let's let's say you want to update a certain a certain piece of data the first thing you do is this update command goes to the commit log and then a mem table is updated and at some point in time that mem table is actually captured in an immutable state and flashed to disk that is all happening in one system now if we look at the architectures as a whole today you can almost see that they work the same way only they're pulling this in two different components right so pull it like the first I think the first big thing was really pulling this transaction lock out and this comes under this comes under many names right so you can view stream processing as being one case of that you have the transaction lock which is basically the stream or the durability of the stream and then you have the tables that are computed over that which is the you know the stateful processing if you're actually coming from from you know from a microservices world and you're really fond of the event sourcing paradigm that is actually almost the same thing so that's also pulling the the streams out the the log out and then building you know building the the application as you know a view over that stream and you know secure s command query responsibility segregation is a is also a way of taking this taking this further so all of these things kind of start to follow this at the same pattern an architecture where let's say this is the this is the this is the log the durability of individual events and then and then you start to compute over these events and build up actually your views and I would actually say that this part has been pretty successful and you know their projects like like Kafka like like bookkeeper distributed lark pro Vega pulsar and so on all of these have actually made an effort to make this here happen very well the stream processors are actually looking at at that point in time and I would actually argue that while in this and at this point here we've actually converted a pretty good to a pretty good state what we can do these days actually with stream processing is more constrained actually on the sophistication with which we can compute over these data streams so the limit what you can do in this architecture is currently defined how sophisticated can your computation get over the data stream and and as for most things I would say oh yeah when when we say like how sophisticated can can you get here you know computation by itself it's in some sense I would say not the hard part that that might you know some people might shout at me for that yes of course it's super hard you know it's getting more complicated as well with all the machine learning and so on we have to find better ways to to make the compute itself scaleable that is right I think from from the perspective of you know building building a live application the things that that kind of capture the complexity of making this application actually scalable for tolerant and you know very well defined can actually be summed up in the in the dimensions of of time and state very often so how do you keep your distributed state in an available in a persistent in a consistent fashion at the same time how do you actually handle time you know time reflects not just time as in the world clock but in many ways it affects like order of elements how do you process them do they come in order out of order how does it affect completeness of the of the data do we you know I have an event here am i actually really able to interpret this properly or is some other information missing still that you know might come might come a little bit later but that I would really need in order to properly interpret this event like solving those two things is something that actually takes this to the to the next and a fling flavoured view of how to how to deal with this would be the following the the core the kind of core way that fling actually deals with state can be can be viewed like this we're taking we're taking this approach events are persisted before before they go in in the computation in an event log you know be that long term or short term doesn't really matter it really has only to be first short term it's persisted for longer great based on these events you do some computation and you can just take an event and let's say you do in this computation or whatever you want to think of it it's just like a java scanner whatever application that takes an event and does something it's not actually it's not necessarily forced to work within a certain framework within a certain API or so and if you want to make this for tolerant to very like a conceptual model to do this it's just say you know you run this process your uh your update your arbitrary maps and lists and whatever you want to do or your you know some opak internal state and some library that you embed and once in a while you take a take a snapshot of that thing and persist it and you remember for that snapshot way we're in the event stream this is actually a like more of a theoretical model I think that has been suggested a few years ago I heard of it first from from like Martin Fowler and the thought works folks but things are very it's very very interesting one to think about about persistence of you know of state of stateful computation because I think the advantage that this has over virtually everything else is it does not constrain you in anything that you do in here right you don't have to use a library for persistence or anything and even if you you know even a good library that says you know I can give you a lot of very flexible maps and lips lists and so on every time you do manipulation it kind of goes back to a key value store then you know and the API it doesn't look like you're dealing with that but really you have to take it very much into account when you scale out the application because it very much effects what you can actually do because you know it every every update also introduces a certain amount of latency so just really working in your local process memory not worrying about anything else and I think this model take care the persistence is kind of the most flexible thing you can do because then on a failure all that that has to be done is the previous snapshot is restored a bunch of events are replayed you're up to speed and you're again working just on your low process local variables nothing else no problem so that is of course you know the case if you have only one process that you want to have for government in practice you know an application is often the sequence of such processes and there you know you can but you can think of it as as having these these many processes connected through some queues everyone does this kind of snapshot for tolerance model and then the the interesting thing is how do you actually keep those dependent processes and let's say this one here goes down but it's already has done some messages to that other process and that other process has done something with it you know so we're rewinding this we're replaying some messages can we actually get rid of the duplicates or the duplicates even deterministic should we actually hold them back until we know that this one's never going to go down again or never going to replay these methods again and so on like this becomes actually kind of a tricky question if you want to make this a distributive thing and that is that is what actually fling basically implements this distributed like snapshot snapshot for tolerance and and process models so if you have these you know if you have this difference to be different distributed processes events coming in and out you know modifying state not all of the processes are a stateful but you know many of them are what the system does once in a while is basically just think of it as just doing a sweep over this whole thing and storing a consistent distributed coordinated snapshot of this whole thing somewhere and that is that is basically but that is basically the the core like stateful for tolerance model behind flank and it's actually very a very very powerful model because not only can you use that to restore processes after a failure as I said before but we've actually that was the first reason why we built it but we've actually opened it up to be also used as a management tool and it's actually pretty fascinating what users built on top of that so you can actually use these snapshots to update and rollback applications you know if you want to go to a next version and you just want to carry your internal state forward again into an updated code version you take a snapshot replace some jar files and you know Lotus shut again and you actually carried you know carried the all the internal state forward I think ubers platform is actually replicating these snapshots in the background for disaster recovery so if you know if a datacenter goes down you you actually recover and the other data center and you you have the you know you only do need to do this on a snapshot level you don't need to replicate every intermediate intermediate change one important thing to point out is that these snapshots happen asynchronously in the background so they they don't actually impact our computation which actually makes this very feasible because you're not affecting the you know the latency and the pre message processing with you know the cross data center operations you can use this to basically archive the state of applications migrate applications between let's say an a testing and a production cluster one of my favorite use cases is actually some some user implemented if you wish an arbitrage model on Amazon spot instances or so right so they're running they're running their code on spot instances and then they're monitoring other availability zones and if the spot instances become cheaper and the other availability zones that just take a snapshot and restore this and a different availability zone and it looks like yeah it's it's it looks like it has like a like a small a few seconds of interruption or so but it it really you know from from the consistency level you don't see a difference it really resumes exactly where it left off in another region so looking looking a bit at how how does this affect our our way of you know building building applications be that stream processing and on stream processing I quickly check how much time I have left that's good the the core kind of shift in in architecture is that you're going you're going from a from a stateless computational layer and and having the state and some databases to to a model where you keep some amount of state directly in the computation and the storage layer basically persists the the events it persists snapshots and you can't you can also think of it it persists state as soon as it has become immutable so what what would that mean for example let's let's assume you have this this microblogging a persona I don't know why this is the example that you know comes to my mind here but let's let's assume you have that and you want to view something like you know how many how many folks are actually interacting with that thing in a certain hour right as long as these things keep changing with India it's actually part of the application logic state you know every impression comes should update something there as once the hour is up it becomes kind of an immutable thing and you throw it out of the application stage just into a like into an archival type of database so the the key point is as long as as as you're actually working with the state as you're reading it modifying it and all of that I think the state really belongs into the application and it isn't it doesn't belong into some other system remote where you only push it because you wanted to be fault-tolerant for ike i will that's a completely different question right once the hours up and it becomes immutable put it into a system that scales differently because because you do have different requirements in scaling the storage capacity and so on compared to the you know application compute but as long as you're really working with state isn't the work set it should really be part of the of the arm of the process because you get so many nice benefits out of it right rather than for every for every little bit of computation going across to your boundaries you're only updating some internal state and you're asynchronously dividing large blocks across these two boundaries you get very naturally exactly once guarantees you know as every event is reflected once and only once in the internal state and yeah the only the only thing that that you get a crew so the only way to actually realize this in the end the classical database architecture is too well to either use distributed transactions which a lot of people don't like to do because it's very expensive or to kind of rework your application that it actually works with the with a model that is not you know not really fully consistent that's only eventually consistent or maybe not quite consistent at all so you have to be you have to worry about item proteins at at very many steps um similarly if you actually want to scan out a service and again I'm only talking about you know the data that you're that you're really working on at that very moment if you scale it out in this in the streaming way you basically add all the working working set persistence capacity and the same in the same way I say I think capacity versus you know if you separate those things into databases and compute those two things have to be scared independently which can be an advantage but I would argue for the working set you typically want to scale them together and as a special case deploying a new services is also you know it's just bringing up another container which implicitly also handles the state versus you know preparing the database to handle in another application all right that is that is a bit of you know how how does you know the the whole streaming and and flings approach and handling handling State for computing over data streams how to think about it and you know what are some nice properties about it the second ingredient that I talked about a few minutes before was was time and when you built when you build actually a distributed application that process is you know processes events coming from coming from user sensors machines wherever there's there's a few a few a few notions to think through when it comes to time so first of all there's very many different definitions of you know of time in in that system let's assume we're we're treating a certain event that is in a impression of a tweet right there's the time when when this impression was actually made on the mobile device there's the time when you know what you first received it on your gateway service or stored in a data center there's the time when the stream processor first picked it up and now you're you know you're in the middle of your window and what what time is it about the time you actually processing this this impression all of them have a very good reason to be taking into account most important I think are the two ends of the spectrum the event time when was this really created and the processing time and and doing this in a doing this in a consistent fashion I think is something that is still it's still not quite trivial and I think again like the streaming architecture and flink have a few very nice answers to that so if you're if you're within you know a piece of piece of application logic and you're getting an event what you and actually doing fling casue you can always ask the system for these two different clocks you can always ask it what time is it now which is basically you know just the world clock time and you can ask it what time is it in event I'm like what time is it in my data and what does it mean what time is it in my in my data it means it means actually the following thing about it like like that you have to you have a different ifs distributed flow of events and you know these events come in here with different time stamps and at the when it enters the streaming data flow what what flink will do and it needs a bit of you know like configuration that you describe a few characteristics of your data here but then it will actually it will generate these so-called watermarks for you which basically give you a a notion of progress indication in event time so they a watermark basically says let's let's look at one here a watermark off of thirty-three flowing through the system actually says we've seen all events with the timestamp up to 33 so you can actually now advance your your data a clock your event time clock to 33 and assume that you know nothing before 33 is coming anymore now that is always only a heuristic because you can never avoid somebody you know completely violating this in an abnormal situation so there is actually hooks to handle okay what actually happens if some some data now comes in really late it actually came after that watermark how do i how do i handle that but it gives you it gives you a very very I would say a very good way to handle the you know to implement the the default handling you know you can you can you can relearn this moramax to say okay online on my main piece of logic I'm assuming completeness and progress based on this watermark progress indicators and you know if something comes later I have to handle it many many setups actually can model this in a way that no that data you know doesn't even doesn't even come late depending on how well behaved it is it's harder with mobile apps admittedly and this you know this watermarks also they can be extremely fine-grained so we do have users running running a a fraud detection system with that that has a 100 millisecond and still wants to work in event time because it receives events from different systems and can make a final decision only once it knows that it's actually in all the different inputs so it actually schedules these decisions on the watermarks only been time but with SLI of under 100 milliseconds so this can actually come extremely extremely fast extremely frequent and compensate for you know for all this this out of ordinance in in that in that area and and that actually makes the application logic really nice because you're not actually saying okay I have an event here I have another event let me wait for a grace period I might see something else but then if it doesn't come I have to make the decision and so on it's actually very easy the only thing that you say is I have I have this event here indicating a transaction and I'm handling it I'm basically scheduling the event time clock to call me back when that timestamp of that event has been reached because then I know I have complete information for that and then you then you handle it it's it's a very simple piece of of application logic basically all the other order handling comes almost for free the only magic is in how do you actually determine the event time progress in the individual sources but that is that is often actually not that hard so so now that we've talked a little bit about you know the the flavor of of cellphone stream processing here in flink what what is the next what is the next wave of streaming applications I would like to go back to the first slide that I showed here right and say you know in some sense data streaming is everything that is or everything where the data changes faster than the application logic and in some sense that's almost everything no I mean everything that is not data exploration and machine learning model training and so on for almost everything else were were you're building you know you're building a micro service or so you have a long-lived piece of application logic and then the data underneath changes it you're building you're building data pipelines they might be running you know they might be running once per day but really the data changes once per day in the application logic probably I don't know depending on how fast you iterate maybe once per week for a month or so so also in that category so the so and and that is a trend that we're actually very clearly seeing that all of these things are starting to to pick up actually stream processing as as a way to implement things so the first thing that that really happened or not this years I would say the following if you think of stream processing maybe a bit a bit further back to two times when you know it might not have been super accurate super performant and so on a lot of things were basically implemented that way sources coming event streams sources producing events event streams being captured the stream processor analyzing them putting the result in a database and then he had an application with application logic and whenever that one was invoked it would actually log into the database and and look at things that you know that you know what what was the current state of of some metrics or so that were computed in a streaming fashion we're seeing really a lot right now as more and more users putting the actual business logic in the stream processor meaning the streaming application computes it keeps its internal state and then depending on things that observes it really directly you know constant or FEC sends an actor message and actually triggers triggers the the action or even you know like response to you know pushes it to a service that then by a WebSocket communicates back to the you know back to the the app user also so so that is that is one part and just like one one piece of evidence for that is actually this this thing that over published a few weeks ago right the the streaming data platform all these these are these jobs coming in in you know applications being defined as you know with sequel and UDS of link job running then in the background and of course it always also you know publishes to elasticsearch my sequel Kafka and so on but a very like a very popular sync is actually just just RPC actually these days RPC has a streaming application as a streaming application sync and and the interesting thing is this thing is geared towards the use of sequel actually so the like a very common use case is to actually have streaming applications that you define as a majority in SQL adding you know adding a few UDF's and in the end - for example interpreter metric and say it's there's actually an outlier now is that maybe a fraudulent ride is that um you know instead of delayed delivery or something like that you know you can actually express kind of this analytical logic in SQL and then you basically sync it directly into an RPC services so that that basically triggers the triggers the direct reactions on that so I would say that is one thing that you know analytics and application logic are actually becoming one thing and that is that is tool to a good extent courtesy of you know being able to run both application logic efficiently and having at the same time the ability to run to run very very sophisticated event like event handling logic with with very flexible access to state and time now one could actually go a little further and say you know what what is the end game of that kind of of that kind of trend can we could we actually say we develop an entire web application let's let's pick something that's not trivia it's big it's bigger social network or so can we build this entire thing actually on a stream processor you know if stream processing really captures all the long-running pieces or so on is that actually possible an interesting thing is yes it is actually has been done so for those of you that actually know these guys that the ones that you know break all the cars all the time it was I think a BBC series now on Amazon there's actually a social network for you know like car enthusiasts that those folks have created um it's it's still fairly new I think it came out maybe earlier this year or so and this whole thing is actually built on on on stream processing like purely on on stream processing right so in the middle you do have you do have actually the persistence of the events everything that you know comes through through the Gateway put a load balancer through a risk it way is actually logged into locked into the into and to a Kafka a cluster then you have you have link as a stream processor which basically computes all the basically all the all the views that you want to have now which users in which group what should be the content of which feet and all of that right and then this is actually mirrored to em in this case it's it's a nastic search and Redis the important thing he is is actually to realize like this follows actually this pattern in an extreme fashion right logging the events all the all the ground truth of what should you know the views look like the groups look like the feeds look like is actually computed in stream processing and then you just have you know different different serving layers that that allow you to do different types of accesses efficiently you know like text search that access versus key value lookups and so on so it's kind of taking this idea of secure as to the I think to the extreme and basically saying yeah stream processing is a good match for that and and you can actually build this entire thing about stream processing so the entire business logic of this thing rests inside inside the stream processor and it's actually been one thing that you might see in this figure it looks a little weird you know it's kind of mirrored one thing that has been extremely extremely successful for them is this idea of because stream processing has this nice reprocessing abilities this nice you know if you build it on event time deterministic computation abilities you can actually just you know bring up new a new copies of the stream processor and mess with this you know in order to say okay actually want to tweak a little bit how my you know how my recommendation whatever works or so and whenever you're happy with this you flip the load balancer to the another stream processing application take the previous one down or you know start messing with the other ones you you cannot get all these benefits that otherwise you would have to you know really write your architecture for to be able to do all this it be testing back filling and so on you almost get this for free by building this on stream processing so that is um I think that is a some yeah some use case I think it's really exciting because it shows what you can what you can actually do you can do way more than you know speed up your dashboard to be more real-time and you don't really have to do much to get to get a lot of a lot of cool stuff out of it and let me actually hope that this let me actually try to do a demo it is running on on a on a small amazon on a set of small amazon machines because i couldn't make it run here on my on my windows laptop and i really hope that this stuff is actually up and running and behaving well man it's looking decent okay so let's see so one thing here I want to show you is how how easy it is to to approach certain certain issues with like with stream processing and with link so this this application here it's a very simple application right it's a it's something that you know handles handles payment requests across different countries here in Europe and then you know basically computes a computer visualization from that for where are these things happening and one thing that we're actually seeing if you look at the charts here there's there's something here that looks like something's going wrong right so for one country in this case particularly Germany everything dropped at some point to zero so we do have we do have an issue somewhere and where we're now looking into you know they're fighting this thing by itself is actually is it's a few lines of code because it really is mainly an you know an aggregation with a few rules over over over a stream and that's actually look at at how do we deal with how do we deal with a situation like this all right how do we actually handle something where we say like okay in the past something happened that we're not quite happy with and so the first thing that that we want to do is actually get this back to get this back to to normal operations operations so what we want to do is actually upgrade the upgrade the application logic now if you just replace the stateless container with you know with application logic that's actually simple right but remember that in this industry mning architecture what we said is we're actually keeping all the in flat working sense data so we're keeping this as part of the application so what we really want to do is when we when we replace the container with a new container with a new application code we actually want to make sure all the state gets carried forward because it's actually really in there it's not in a remote database so if you remember this snapshot model and that data that I showed you earlier this this gives us actually a very nice answer and let me let me quickly show you how to do this so if we're if we're actually I have to make sure I'm quick here because the dolphin works very fast and if I just want to upgrade that application here and say you know my developers were actually they were doing a good job they managed to fix the whole thing so I'm actually replacing this with a fixed and I'm upgrading this we can actually see that you know this should be okay this would actually just add another snapshot to the system and then it should be replacing the application logic should be adding another job just redeploying it based on the on the new snapshot give it a few seconds okay good if we then look into our dashboard that has actually fixed that has actually fixed our latest bits of events yeah so we see actually you know the the metrics for Germany are coming back up but so far maybe so good but the really interesting thing comes actually next I would say so let's say you know we're having we're having the new metrics being okay but we still have a problem in our past right so how do we how do we handle this you know we could actually say yeah run run some special application logic to backfill things one minute should be enough a really cool answer in three processes is again you know use use basically snapshots to do that so just you know look at when this whole thing happened we look the look at the next snapshot that happened before that and my case it's this one and basically say not sure if you can see this but you know fork an application from that from that particular snapshot so what this basically allows us to do in stream processing is just to say you know have a demo to have the same application that is using you know the latest code but it's actually you know using the state as of a previous version right and now we're actually starting this here and now it depends on really how fast these little Amazon notes there are what we should see once this is running is actually that we we just duplicated this application and we have you know we have our we have a regular application that we just upgraded it goes it goes keeps on processing here it actually you know that it did a consistent resume of the exact part where the application was before and we should have another copy and I think the snapshot was somewhere here in terms it's working its way through here and it took just like an a second keep start like repairing all of these things so the thing I really want to point out here is this this thing this application is written like as a 20 liner of coda so just you know event time stream processing you get all this you get all the notion of you know ik archiving intermediate states versioning it and so on for free because you can apply this nice snapshot monitoring processing right and and all these things that we did here like with we took a snapshot in order to replace the code into a consistent resume or we're duplicating the application in order to let it to let it reprocess I really hope that it should actually start doing this soon because otherwise this demo doesn't look all that impressive let's give it a give it a second it should actually be working its way through here because I think the snapshot was somewhere here here from the past oh yeah no it starts doing it yeah so nothing of this was actually written with anything like that particular in mind it's just a regular stream processing application and just by virtue of doing stream processing with a system that has this nice snapshot model you can actually do all these things like nothing no special code right it's just all a property or stream processing and snapshots and so on so with that I would actually say in some sense we can conclude this with saying the next wave of stream processing applications isn't actually all that yeah it's not mad that that complicated or it's not that it doesn't seem that crazy to think ahead and say you know all sorts of application that actually reactor data and time and this is in some sense all that applications react to which means it's it's really it's really the majority of all their long-running applications that we have is actually a good candidate for stream processing so my take is the next wave of stream processing is actually going to be it's actually going to be extremely broad it's going to be a just a way to completely change how we how we actually develop applications and not just analytical applications like of all forms of you know data-driven applications all right thank you very much [Applause]