Monitoring Reactive Streams

Scale By The Bay 2018: Stefano Bonetti, Monitoring Reactive Streams

Recording: Scale By The Bay 2018: Stefano Bonetti, Monitoring Reactive Streams

you all right yeah thank you very much for being here let's get started with a quick survey is anyone here work with reactive streams you go a few hands anyone used them in anger in production yeah few of them it's good my main point is regardless of tooling you use whatever implementation you're after streaming apps are becoming more and more part of our lives as back-end developers as basically the need grows to process more and more data right as soon as it's come now just like any other apps streaming apps need to be monitored right so this talk is going to try to analyze the challenges of monitoring reactive stream application and will try to answer some questions around these challenges and we'll see some practical examples around acha streams but most considerations are applicable to any sort of streaming app or framework now very quickly I'm assuming you're all for me or I'm actually hoping that you're all familiar with the advantages of monitoring your application in the first place so monitoring is about collecting and tracking what's going on in your system the reason being we want to be the first to know something something goes wrong right or if something was particularly well but being inherently paranoid we want to be the first to know something goes wrong right now for this reason is particularly important to instrument your app or your system with monitoring from the very start of the development process and not just treating monitoring is it like a nice cherry on top where everything is like in gold plating phase and of course monitoring is generally enabled by metrics these are raw measurements of behavior that you can collect and of course you can think about countless types of metrics you can have your own your very own business metrics that you can collect you can talk about metrics in the CI CD context you can have low level system metrics as in memory CPU whatever but when it comes to reactive streaming essentially there's two key metrics that are very important to understand how your euro lap is faring and we're going to talk about these two today one is basically want to know how many operations or requests or data items how you name it are going through your stream and this is what we call throughput usually and then we want to know crucially where it's bottlenecks are and this basically means in the reactive stream context if and where your stream is back pressuring so you can think of many more metrics again that you might collect and you should in your stream but these two particular alone will give you a pretty good understanding on how your system is performing and we're going to see why these two are important and what specific challenges they pose in the reactive streaming context so we'll start with throughput here why do we need to measure throughput like if you think of your traditional web service you usually think of throughput as in terms of requests per unit of time so we usually have some SLA right we want to avoid and we probably want to be able to sustain a certain throughput going through your app so this is pretty classic this is true also for streaming apps of course however when we're dealing with streaming apps some of the assumptions that we make for traditional web services are not there anymore we have to kind of nonlinearities of of sort that we need to deal with and these are two challenges for us first of all concerns topology this is your strict streaming sorry this is your traditional web service you have a linear topology you have requests an input and responses in output right however in a more kind of generic streaming app you have potentially a much more complex topology where you have a variety of publish a subscriber fun in fan outs any sort of graph right so this is the first thing to keep in mind secondly in a traditional web service you present one request an input and you expect one response in output right if the service is healthy now if you are dealing with a more generic streaming app you don't necessarily have this linear correlation between inputs and outputs right the generic streaming processor floo materialization if you live in the extremist world is not a function so in a function you have one input you expect one out here for one input you can have from 0 to infinite outputs on the other side or the processor might even start producing outputs if no inputs is given to them so you can imagine streaming applications this is basically a generic version of your traditional web service and this is actually how a web service is modeled in things like eh TTP if you're familiar with the tacky CTP sits on top of extremes and it's actually modeling service web services as a flow of incoming requests and outgoing responses now this set doesn't make sense to monitor like throughput for a whole streaming of not really right you throughput is something you want to monitor in specific points in your graph right in your topology so one way to solve this basically to instrument your graph with multiple check points in the form of transparent processors so these are basically sections of your graph in your pipeline sorry where the input is basically transparently presented to the output and metrics are collected for you now the main idea is to have basically reusable components that you can plug wherever you want in your graph to get a pulse on your streaming app how you're doing now you might not need to need to put these little check points in all of the junctions in your graph it's actually good to give the developer the liberty to kind of place these in like whatever sections are more relevant so let's see how this translates into some code so we're gonna use our streams so let's see we have a source that emits a random integer this is what we have on the top random integer every 10 milliseconds right we have a filter that throws away odd numbers and then we have a sink that prints right this is very simple extremes pipeline even if you're not fully familiar with our streams is it's it's pretty it's pretty readable what it does right so and it's it's a very simple example it's already an example where you can see you have different throughputs in different points of your pipeline right because you have a filter element now source flow and sync if you're not fully familiar with the costumes again they can concatenate it they can become cutting it with the very fluent DSL using via and run with to start the flow of data so to go to the throughput measurement part we can now create a simple checkpoint flow to measure the throughput this is pretty simple it's going to be a transparent flow that creates a metric now here we use common for example as a back-end but you don't have to stick to that and essentially you give it a name to mangle it in the in the in the common back-end and this basically increments the counter every time something passes through the stream so this is pretty trivial right we can now place these transparent flows wherever we want in our pipeline and give them meaningful names so that we can track them in whatever back-end monitoring system you have the key thought is that these are basically completely generic and will work by being plugged in in in whatever our streams graph you have right and this is of course courtesy also of extremes graph stage composability so this is what you would plot on a throughput dashboard for this very trivial example right so the unit here is elements a second so you can see we're producing around a hundred elements a second so this is one every ten minutes this is the rate we're generating stuff that and basically after we process that's the filter we are left with about 50 elements the second right because we dropped the odd one that is pretty simple sometimes it's it's even practical to aggregate these two metrics or two metrics in two different points of your graph to get some sort of relative throughput of kind now this would be basically a ratio between throughput collected at two different points right so this can give you an idea of how many like the ratio of how many events are being dropped or conflated maybe if you're have if you're having a ratio lower than 100% or maybe oh man you're generating or expanding if you have a ratio higher than 100 in this case you would have relative throughput of 50% of course now so throughput is what we've seen and it's kind of the simpler of the two to to monitor so let's move on to something slightly trickier let's move on to back pressure now why do we need to measure back pressure at all and this is the key question so we can explain this with with an example to make this a bit more concrete so we have a streaming up that you see here consuming from kafka you have some processing logic you mark some elements as invalid you want to lock them send them to a log file you mark some some of them as valid you want to put them through store them somewhere using some HTTP API and then at the end commit the cough cough so this is a pretty common pattern right when when consuming from Kafka in general and when writing data pipelines now let's assume that we are actually instrumenting the graph for throughput measurements using the thing that we just code it together right and let's see let's assume that we get back to following your results we are consuming from Kafka at 100 a second which is mark roughly 1/4 of them as invalid put forward 3/4 of them and then these are going through being stored and committed right so we're all happy that we have our measurement here but we have two possibilities now either we are happy with these numbers and we can all go home and or watch more talks or we're not right and this means that they not matching our SL A's and let's say for example that this is the case let's let's assume this calf get off topic produces thousands of messages and this guy alone cannot cook right lag would spike through the roof and we'll all be very very unhappy now you can solve this in a few ways and one might think ok we can deploy a bunch of these it might not be enough if you don't have enough Kafka partitions maybe your boss will not be very happy with your AWS bill at the end of the day so you know that that might not cover it right so we need to and where the bottleneck is in this guy in order to optimize it right if we can roll optimize but we need to understand where it is first and you can see that the throughput readings here are pretty useless to us we cannot really understand where it is is it one might assume it's the API maybe but it could be the comment stage if we're committing too frequently to Kafka we're not really sure and this is because all the readings they align to the slowest pipeline stage and this is because of that pressure back pressure is actually a very desirable behavior of reactive streams and it makes sure that we never overflow our systems with by consuming more messages and then one weaker actually process right and this bit is makes our systems resilient by design so it's very desirable when we're dealing with unbound stream of data when unpredictable volumes and this makes important to understand how reactive stream works so let's spend like 20 second just recapping this essentially in reactive streams is the subscriber itself that requests more element right request for element is basically propagated up stream up until someone who can produce elements in this case a publisher emits this amount of elements and these are in turn propagated conversely downstream to the subscriber itself so essentially our pipeline is back pressuring in a specific point so say between this processor on this subscriber when it's waiting for a request coming from downstream right this means that the subscriber or seen when the realization is busy and cannot process new element so stops requesting right conversely when a request has been made and we're waiting for a limit of something from upstream this is a state when our pipeline is not back pressure right so how can we measure this what if we could somehow measure the time spent between waiting for a request from downstream the time spent waiting for a limit from upstream a ratio of these two would give us basically the relative amount of back pressure that goes on in one specific point in your pipeline and basically in this sample timeline that you see here you might mark with green the portion where the pipeline is not quite pressuring and with red the portions where where the pipeline is back pressuring now this is easier said than done because in reactive streams in general asynchronous prescribed right this means that requests and emit might happen in very different contexts so it might not be an easy problem to solve right thankfully in inactive streams actually occur streams makes our lives easier here basically by sequential icing some of these events in the same context so next we'll see how we can get this done in extremes itself so enter the graph stage API this is basically our streams lowest level API it's an API that basically makes you define any streaming stage so bear in mind at any stage that you use in extremes uses this underneath and basically lets you define a bunch of ports for your stage and it lets you attach and lers for these ports it lets you define on pole handler for output ports and on push handler for input ports now just as an example this is out map is done in in extremes under the hood so basically you have two ports in and out they're typed right and then you have a logic with two handlers right so when upstream pushes basically grabs the element from the port applies the function that you create them up with and then pushes the transformed item downstream when downstream poles conversely it just propagates the pole upstream saying whatever is upstream please give me more because I'll be able to transform this is send it over right so this the API the data streams gives us we can leverage this to measure our back pressure so the most important thing to keep in mind that is the graph stage in extremes is net is never accessed concurrently basically it's just like the inside of an after so this means that the order in which these handlers are called is deterministic we can basically rely on these to be called in a strict on push on pole-dance right and this makes it a lot easier for us to to measure that pressure and we'll see how basically it makes it possible to discern the time between requests and Emmys or on pull on push calls select create a graph stage that does that so this is a very very simple implementation of this we can keep two timestamp variables one is last push and one is the last pull we can use VARs again inside stage this is basically like the inside of an act is one of the few places in which you can do this with impunity because the extremes infrastructure makes it possible for you to do it safely now we can create a histogram again we use come on here we store the back pressure reading the back pressure readings in it you can use whatever is your favorite back-end so when a poll comes in basically we first propagate the poll upstream and this is just what map does and we've seen this and then we can mark this last pull timestamp as of now right now when a push comes in we can actually do this calculation of the ratio between the time spent waiting for the pool and the time spent waiting for the push like this this is quite simple calculation in this case it will be a percentage and zero percent will mean pipeline is never back pressuring in this place and hundred percent means it will always be back pressuring I mean it's always back partially in this place but more likely will be something between the two right now this stage can be used in the same way that we use the throughput meter here we just place it wherever we want in the graph and we get a post right so let's consider the same example that we've seen before so we had a source that produce a number every 10 milli right and then we had a simple filter and a print right so say we instrument the stream to monitor back pressure as well what what you reckon we get is back pressure reading you just a second to think this was a tricky question this you you get zero back pressure here so essentially because the stream is very fast we filter and print it's it's everything we do in the stream is much faster than the ten milliseconds at which we emit the numbers right so this means that both checkpoints will never back pressure right it's tell us that the pipeline basically has no button I call actually better at the pipeline the pipeline's bottleneck is the source itself right now we can spice this up a little bit by replacing the filter logic with something much much slower we can use the facility from akka dot pattern which is called after to simulate like 250 milliseconds operation and we can use map pacing to run futures inside extremes as you probably know if you're familiar with the API this is basically something that runs futures inside your stream and the crucial bit here is that if the maximum parallelism that you give to map a sink is reached then the stage starts back pressuring right and in this case max parallelism is set to 1 so basically this stage will run futures of 250 milliseconds sequentially one after the other right note that we still keep the filter around so let's see how the dashboards change both of them so if we start with the throughput one for a second we see that the graph is pretty similar to the one that we were seeing before we just notice that basically the throughput is decreased considerably right of course because we're doing extra work so now we produce roughly four elements a second because we we basically have 250 milliseconds delay and we still have around 50% right because only the even one makes it through the processing stage so this doesn't really tell us where the bottleneck is again right so we could have put the slowdown in whatever part of the pipeline we still have got this this graph here right so let's check the back pressure graph for a more interesting insight this is what we would plot now we can clearly see that the produced checkpoint is now back pressuring right almost 100% why this because it's got a very slow down stream that starves it of requests right so at the same time the processed checkpoint is never back brushing that's the yellow line because it's got plenty of requests from downstream and this is because the print that the sink that prints is always hungry for more it's very quickly quick at doing its job right so this tells us that the bottleneck is actually the processing flow which we know already because we put this 150 million there but that's that's not really the point right so basically a more subtle but still interesting point is that the the Green Line is not 100% right if you check it is around 96 97 percent this is because the source is still emitting at a rate of 10 million right so for this 10 milli for the fraction of this time the checkpoint is waiting for the source to emit and this is a fraction of its time in which it's not pressuring right and this amount roughly to 4% of its time there is basically 10 milli over 250 million now this is nice because we know where the bottleneck is but it's also it's only half the job right we what can we do to ease things up when we find the bottleneck well this is this a whole different discussion right and very much depends on the kind of workload that you're performing and a few other things but basically extremes gives you a variety of tools and weapons that you can throw at your stream to shape basically stupid and you can basically increase parallelism she's one thing that comes to mind first you can the synchronous boundaries you can use map async and ordered if you don't care about the order you can add buffers you can used maybe conflation if you afford if you can afford to be lossy in your stream there's there's a whole thing you all things you can do I won't go in more details here because this could be an entire talk by by itself but basically for the top class blog post be sure to check out this from Colleen Bragg's blog this is really worth reading for anyone who uses extremes it's it's basically a very very concise sum of basically multiple tips to to make your your extremes very performant so summing up we've discussed an approach to monitor extremes applications but some considerations that we made can be applied to basically all reactive streaming implementation out there the key thought is that you need to make sure that you monitor your streaming apps in different points of your graph and crucially you need to get a good understanding of how both data and backpressure propagate so reactive streams are be directional streams so there's data go in one direction and the requests go in the other one you have to make sure that you understand both of them because otherwise if you only monitor the data or the throughput or if you don't monitor at all god forbid you will end up with a very op extreme that it's very very difficult to reason about and this basically means that as soon as you have a problem you don't necessarily know where to put your hands right now this stuff that I showed you that I showed you I put it on github as a micro library so again feel free to check it out if you use akka streams will be very interesting you hear some feedback essentially this gives you a single checkpoint stage that gathers the metrics that I've just told you about it does more it gathers push-pull latency it also gives you a metric that help you diagnose liveness issues this is something that I didn't go I didn't explain today because there was not enough time but what I showed you what I showed you today does not necessarily help you in case you've got liveness issues in your stream so there's something to help you in that direction as well and it counts failures and completion if this is something that you that happen often in your pipeline now there is a micro site as well for the commentation so feel free to check it out it's currently available for come on and drop wizard backends and it's it's available on maven central so really yeah if if you want to take a look and give some feedback I'll really appreciate it so this is all for me I really thank you very much for your time if you want to get in touch these are my handle feel free to grab me the conference I'll be around for these three days thank you very much for your time [Applause]