scala.bythebay.io: Petr Zapletal, Top Mistakes When Writing Reactive Applications
Recording: scala.bythebay.io: Petr Zapletal, Top Mistakes When Writing Reactive Applications
I you so good everybody my name is Peter and I work for cake solutions we are a consultancy company focusing on building reactive and video processing applications using technology such as color across park and many others I had a pleasure to work with a number of companies and fight often I've been asked to tackle problems they faced and some of these problems were quite repetitive so I take couple of them and made us talk about it so I'm going to talk about problems like actors verses future's cessation before shutdown distributed transactions and longtail latencies after that I've got a section with quick tips which are basically topics I like to mention about I don't have a vampirate I hope everyone will find something interesting so rotten one constraints liberate liberties constraint is the name of an excellent talk by Aruna be honest soon he gave like year ago at chabad conference apart from many things he talks about how constraints liberate us and how we should use the least powerful and the most precise abstraction which does the job for us and this makes perfect sense in the context of actors verses future's so this is how our toolbox looks like on one outside we have a future and the other side we have aqua actress future is a strongly-typed look abstraction used for concurrency a written just one value by definition on the contrary actors are very dynamic they are not useful type they receive method is a partial function from na 2 unit so basically give me whatever and I maybe do some side effect on the other hand they are location transparent and they have nice results mechanisms like watching recovery supervision and so on apart from that we have at a type which tries to take a short comings of the vanilla actors but right now it is still very experimental and definitely not an option for production usage and that by the way don't confuse this with type actors which replicated some time ago national grounds are extremes which are static way how to define processing then I'll go type and they also offer like pressure so as you can see kappa acha gives us a couple of options some of them are as powerful constraint and easy to use and some of them are very powerful where and with fever constraints but as I said in the beginning we should try to use the least powerful abstraction which did the job and now let's take a look at common actor use cases first one is the state management which is a very good reason to use actors other good reasons why you would like to use them are another akka core features like Lucas transparency or auras lens actors are also very nature grade if you want to take advantage of single rider principle and they can be used as an a memory work free cash as well another very common usage of actors is shouting and futures should be used in cases when we need just vocal concurrency surely this is achieved by actors easily but futures are much easier to use creatures require powers writing and we can take revenge of function approach so we can enjoy types of thing or future composition and all these things we love so do not overuse actors as you probably know Java cessation is very slow food been heavy as has scallop version binary compatibility issues and as you may know as well java civilization is the default and I cap and that's pretty bad for us drag too many reason why we want to slice data and using acha first one ascending data for the network so so we have an actor which sterilized a message and sends it to another actor the second actor this slice the message crazy response and sent it back this round trip is often called ping pong test and it's often used for a vote for evaluation of cessation libraries the second example is a distinct data into local storage there for sterilization is very important when working with a cap as it stands for example so let's have a quick chat a quick look at its graph it compares trips around around tricks or value of solution libraries I'm pretty sure it is very hard to read but so I added the arrow which allows that which shows us where the Java implementation stands and this graph shows us the footprint size as you can see the last port by the way if you like to see sources of this there are in the references slide in the end if you can see how civilized data looks I for a very simple class the footprint of generalization is far bigger than the footprint of not very efficient for Matt likes XML or JSON the reason for it is it doesn't sell eyes just just data they also services the entire class definition and all definition of all reference classes so it was the authorization but before we move on let's figure summarize what point we should have in mind when choosing between various libraries like aforementioned romance or footprints eyes and also many address like schema evolution support human readability language bindings and so on just a quick note as we are in sky by the bay or libraries I'm going to mention i'm going to talk about half sky api's this one is the most used a human readable format these days surely there are some others like XML or CSV or whatever we certainly have there's use cases but i will go mainstream surely it has disadvantages it's relatively slow and bolting but if you need a human readable format it's it's a good way to go and do a good thing is there is a so many great libraries available for binary formats we have two options firstly schema-less formats meaning data are sent together with corresponding metadata the main advantages we don't have to define schema so it is almost like using generalization but performance wise it is a very different story as an example we can have cryo and it's got a library named chill and by the way setting up this in acha is a matter of minutes and now let's take a look at binary formats with schema defined by some kind of DSL sulit offer great performance and minimal footprint and they usually provide building support for scheme evolution popular choice ace protobuf and couple of related projects like third buffers and Captain proteome great alternatives are briefed and a blow for summary I like to say you should always or if you plan to go to production at least replace your authorization with something else is very depends on your particular use okay so I'll leave it up to you but if you want to have some quick blind accommodation I would go for these Jason 4s cryo and perturb of all of them are great and proven libraries and they should work to your satisfaction another topic I like to discuss is a grateful shutdown so let's say we have thousands of shattuck actors on tens of nodes and we want to shut one of them down because of maintenance or hardware issues or whatever essentially we want to do something like this but we don't want to lose data which are already in the note this operation that they totally transparent to our users so let's talk about how the procedure should look like from a high-level point of view I really wanted to show you some code samples here but I also unable to squeeze it here but if anyone is interested they are in the makeup section in the end firstly JVM gets the shadowing signal which triggers this whole exercise when it happens we tell coordinator the procedure has started coordinator is a staff or entity which manages values states of reficul down as I like occur in my case coordinator would be implemented as a knife as an actor but of course you can do it differently now coordinator has to tell local regions they have to leave the cluster and wait until it's done so it has to send a cask graceful shadow message and watch their lifecycle updates when it is done coordinator asks with a note to leave the cluster and give Singleton's a grace period to migrate unfortunately there is no callback for this so it has to wait specified amount of time finally we can kill coordinator shut down reactor system and let JDM to be terminated at some point you'll have to add messages for interaction with if you're shy detectors just a quick note if you want to stop acta gracefully you specified if you need to stop it immediately use context dot stop and last thing i want to mention is the priority mail box it's good to have it because it alls your graceful shutdown / common messages to be handed in priority but basically it maintains GQ's 14 normal and one for purity messages and it also it may also respond with something like the newly shutting down so the message can be resent to another node so let's have a quick summary we usually don't want to lose data sometimes it's fine but most of the time we do care about it the implementation itself it's not that hard but some knowledge is necessary we need to implement no children coordinator you'll handle various states of a shutdown procedure and this guy needs to be instantiated and on every shattered node and finally we have to do some integration with our shadow factors so so that's it for graceful shutdown and now let's move on and take a look at the distributed transactions so disabilities actions or any situation where a single result so in single event results in the changes of two or more separate data sources which can be committed atomic rain of course this definition is very informal but I think it suits our purpose well the problem is especially for native implementations it works most of the time but there is something named seven forces of distributed computing in case if anyone has not heard voted yet please take a look at it it's very crucial you know if this would be the only thing you got from this talk it was totally worth it so just a quick insight ooh we are told we are talking about a set of false assumptions people tend to make about distributed systems like the network is reliable latency is zero battery life is infinite and so on and isn't this may distributed transactions fight changing a classic approach how to tackle distributed transactions is is the two-phase comment the point of two pc is a performance tentative operation first and if all succeeds the comments are confirmed so basically it prepares the operation which is but it for a certain amount of time or until acknowledged and this definitely possible to implement in in a reactive Bay using asynchronous messages but of course we need to have a diesel instead of Aragon tea and therefore we need to have some kind of persistent storage and also we have to take a replication and we have to all we have to use I'd important messages but the main problems are it does not scale well it's slow and has certain network issues as I said it was possible to imbalance in a car but there is a there is a better way and as the saga pattern saga is a reversal should how to tackle distributed transaction the point of saga is that each of the local commit also contains a country part the country park is compensating action of the corresponding transaction so the textbook example is that we have comments or sub transactions every well like Google hotel ran rent a car or buy a ticket with compensating actions like cancel the hotel or the car or whatever if something goes wrong we apply all composite intersections and reverse the transaction again we have to take care of requirements like at this one's delivery the replication and so on but this is the way you should go but there is an alternative option try to avoid distributed transactions where possible meaning that every business event needs to result in a single synchronous comment and the other data sources could be updated a single swing so basically we are going to give up all or nothing approach and we are going to introduce eventual consistency in our system and this is often the best thing you can do so the outcome of this section should be distributed transactions are hard expensive fragile and do not scale well and should be avoided when forcible if you cannot avoid them go for the saga pattern based implementation and how it's time to take a look at longtail agencies so computer system where each service typical response in 10 milliseconds but with the 99th percentile of one second the distribution group like this latencies we would expect our green room tells are arranged so the 99th percentile could be like 40 times taller and in case of nineteen ninety nine percent five it could be like 50 times slower and do this graph by the way also showed us how important is to study all latency distributions not just means using means only the problem could be overruled easily the 99th percentile of 40 minutes means that one rick fehst 900 experience for the most items instead of expected let's say one millisecond moreover most of our systems are distributed microservices so one request and creates a bunch of other requests in our system for example let's say one kind request generates tensa breakfast and assume there's a one percent probability to hit a sorceress so we got nine to five percent chance the request is affected by the slow response on details can be caused by various reasons like hardware or network issues with configurations architecture implementation problems and so on but it's important to realize it's not just a noise or some or some peaks or whatever it's a real problem usually when the first thing we do when we are facing problem is trying to drag out we try to narrow it isolate it and tackle eight and so on we all know this but this is usually not that easy actually tackling these kind of issues and the stupid environment is very hard but I don't want to talk about it today I want to talk about torrenting long tails and in a similar way how we delight failures there's a bunch of various approaches so let's have a very quick look hedging requests meaning sending the same request to multiple servers and center and use whatever and comes back first just a very important note to avoid doubling or tripling your old units at the hedging request immediately but try to wait until the original request is has been outstanding for more than like 95th percentile so additional should be around five percent but long tails should be short on at significantly then we have tied requests it means instead of Deline before sending out hedge request we ank you request scimitar Nestle on multiple servers and but and we tied them together of the information about the others when the first several process the request it can tell the address to cancel it from their cues in a real google system this reduced median latency by sixteen percent and the 99 99 percentile was reduced by forty percent selectively increase replication factors this one is critical ways try to have more copies of things which you find more important we can also try to temporary exclude the sole machine from our operations since the problems might be temporary we can continue sanshiro request to the machine and monitor even if the problem is appears we can put it back consider good and our responses meaning responding with incomplete results in exchange for better end to end like dancing I also do last one Hara Hara updates can be the fastest and reasonably cheap way to tackle the program so consider this as well so they would last to pick ever talked about but before they finish I get just a couple of points I'd like to mention at least a day having enough time to fix the first first one is monitoring Moira tunic is very important try to implement it as soon as possible but when performance has two weeks before production goes wrong secondly network partitions are real and surprisingly come on and also spread where is over is a must when implementing distributed system it is very funny when you don't have it a new face split-brain scenario with passes and actors or cluster Singleton's on board booking this one is quite tedious try to ever working where possible if you need to block us dedicate the dispatcher or thread pool for read most of the time a singer actor by JVM is all you need acha provides knives volcanic features so use them instead for error handling tried to take advantage of provision and now it's time for a question but I'm afraid I'm a little bit late so if you have na try to find free yeah she's 20 minute sessions are tough on presenters so a good job getting through all the content there yeah thank you right [Applause] you