SF Scala: Jonas Bonér, Building Reactive Applications with Akka

SF Scala: Jonas Bonér, Building Reactive Applications with Akka

Recording: SF Scala: Jonas Bonér, Building Reactive Applications with Akka

[Music] so I'm really glad to see so many people here they put some pressure on me I guess I'll try I'll try to deliver so my name is jonas paneer I was I was I was part of family this company typesafe but I also was the guy I wrote the first lines of the Oh Bob of the Aqua codebase a long time ago now almost five years ago we're actually celebrating the five-year anniversary seems like the first public complaint I'd say but Victor clang it was sometime in June and the product had been out like a couple of hours so I think I tweeted it and they say oh why don't you do it like this and I said okay fix it and like yet another hour later I had a pull request so so then we had a little small community that then just grew over over time so yeah let's get started I'm really not comfortable holding this mic but I'll try did not wave too much so we're in an era of profound change I think I mean for us as developers things have fundamentally changed the last 10 years just five years the user they demand way way more nowadays I mean the the application needs to be close to real-time at least have a real-time feel they need to be interactive they need to be fully collaborative the user need to be feel empowered that they own the data they own the experience and they have extremely low patience right for for for delays for downtime if the app is unresponsive they will just go somewhere else also for others developers like writing applications that we deploy things have fundamentally changed as well I mean in the 60s 70s 80s and 90s applications were usually written to run on a single machine running on a single core Ram was very very cheap no expensive I mean sorry very very expensive and and disk was expensive and the network was usually very very slow today is the universe I mean the applications need to be deployed on well like lots of machines like cloud computing and all that stuff all running multiple processors and it's really hard to find a machine now that doesn't that doesn't have at least two to two cores even my my Samsung Galaxy has is it is a quad-core device and the RAM is really really cheap nowadays we're not limited by that that means that we can keep data around way we're longer we don't need to service or go go and clean up that stuff from its disk is also cheap and and and then the networks is really really fast so I mean really need to react to users much much more now I mean big business requirements changed way more rapidly like it is a faster turnaround time for everything so I think new requirements that we're all thrown into need new new tools and we need a new set of sort of not just platforms but in like like physical tools that we actually were actually I mean a software but also meant other practices we need to think about software in a different way and a lot of these sort of new ways of thinking are actually old right we need to go back in history and see how have these problems been solved in the in the past and very often it's I mean then the answer to current problems are to be found somewhere else so in in general I think I think as when you when you look at prob problems were solved the last ten years and try to like solve them the same today they just don't cut it anymore I think miss Tanner je style of applications can't deliver on these new on these new requirements and longer so we really need a new way of writing applications and that is reactive applications react their applications if you look at how applications need to be need to be developed today I think they need to have sort of four fundamentals or core traits very distinct traits in which in which the O's or support each other and built on top of each other and form a cohesive story and without any of these if you just use three or two of them you don't have that story and and it's and it's really hard to deliver so starting with event-driven at the bottom here I can just point at one of these screens have event or even really mean the react to events right an event is something that that's happening in your in your in in your application can be can be like something triggered from within it can be an outside stimuli or something and so the event-driven applications really needs to be written in a way that they are asynchronous and that they are non blocking if you do that then then you then you get systems that are usually very very efficient in terms of you in terms of resource utilization they tend to be very very lazy I mean not do anything and actually until actually being told okay and and and and tend to rely more on push rather than pool and and and and so it's a side effect or something that sort of falls out of a fully event-driven system is that it's usually very loosely coupled okay and these are sort of forms to sort of Foundation or many of the other traits here I'm just gonna actually put this mic up here just I don't want to hold this any longer so the the second important trait we have here is scale is scalable system two they need to be scalable and that means to react to load okay it can be both like like predictable load so that we that we can that we can plan things my girl over Christmas or something like that or it can be unpredictable Oh are we gonna get like Slashdot it and then there are like unpredictable spikes and and the system needs to be able to deal with this in in in a nice way and and to really scale scale applications out either scaling up meaning utilizing the harlots or the multi-core architectures are very well or scale out on cloud computing many many details or applications both either in virtualized environment or dedicated hardware there really need to do two things and that is minimize contention contention is really the the absolute top scalability killer as long as there if there are many sort of threads or components or contending for a single resource then there is wait time right then then there are components they can't do useful work because they are sitting there waiting for for for that sort of for that resource to be released the second thing is maximized locale locality of reference that means that that the data is close to its processing context that is they don't need like shuffle data around and so on and if you look at how modern first of all like CPU architectures work now is that is that all this really comes in comes into play and and and and and and its really for me the way I look at it is that it's really no difference if you scale up and out it really does it's really all scaling out and either you scale out on multi-core these distinct processors that you can almost look like being like small tiny machines right or cieaura scale up in the large onto machines that holds all these store tiny little machines that's one way of looking at it and the important thing then is that there'll be one single unified abstraction for all of this you don't want two ways of solving that's basically the same problem right and I truly believe that the only way forward to solve this in a nice way is to adopt a fully share nothing architecture where we're I mean because in it in the true sharing nothing architecture then data is is it sir so like data that's used together or like put together so you maximize locality of reference and also if you have a single right to think and adhere to the single right to principle in which that often data can be written and then read like will completely without contention so that is I mean all this is something we thought hard about and sort of forms the basis of akka so that's what I'm gonna tie tie all this into the arc of story in in a second resilience is the third very very important trait and that's something that really makes me sad I'm when I'm out I mean meeting in Sora meeting clients the last like five ten years and was actually the primary reason for me to to create ARCA that I think the way resilience is looked upon is comenta ly broken right it's usually something that that we try to avoid one while we develop their up our app or application and once we go to production or and sometimes in staging rapid often in production we just feel like I'm gonna sing oh what about resilience there we are ask table skies just turn on clustering in our app server and hopefully things will work right that's not it's not a like sustainable strategy I think I really think that it can't be bolted on afterwards just need to be part of the of the design of the application from day one and if you look at resilience I mean things will fail right that's that's a fact of life especially if you start adopting like it may I mean think I mean scalable system really start running on multiple machines and so on network can go down and this can can crash and so on it's a fact of life that things fail so instead of seeing it as something exceptional let's sort of embrace it as the natural state in your applications life cycle you have the applications startup that's once this one face it might run one face you my top gray this one fades shut down as long as skins failure is just another one of these natural stages in your applications lifecycle and if you look at it that way then you then you can can sort of have in have a natural way of managing it when the failure happens no big deal I mean we just know go to this state that we already have prepared for and then and then we fix the problem and continue there's nothing like scary about it really okay so I really think that we need to have a fundamentally different way of approaching the problem share nothing is part of it because if you have a shared nothing architecture then then what you get is its isolation so you have all these components that are fully isolated and that means that they will not like lead to cascading failures if one component dies you can die in in in solitude right in a way I'm sure we both sort of all feel sad about it but the nice thing is software that is that we can we can like to revive it again right so you can write the system that you can restart the component and it's like you're fresh and eager guy comes up ready to do work so I really think that's the only way to bring back sanity back to the way we deal with with with resilience in software and the final crate responsive I'm it's an early if you write a system that is like man has this all this nice trade right sorry it's fully been proven it's now blocking it's like it's resilient like crazy like 99.99% uptime and it's like scaleable out to an infinity infinity or whatever it's all in vain if our system is not responsive right because our users will they will not want to use sources or our application anyway if it's too slow if it doesn't meet their sort of the expectations of the user doesn't meet the SLA is that that are set up so and this is true for him across the whole application everything needs to participate all layers if one layer like takes too much time it like slows down the execution pipeline or then then it's all in vain as well so I I really think that the applications need to be responsive at all times not just under the blue sky scenarios when things are going well but also when things fail then responsiveness need to be predictable you need to know what you what you what you're dealing with Andy in this situation because as we just talked about things will fail and it's natural they can't be that responsiveness goes across the roof then and the same thing with doing spikes or whatever things that are unplanned so and they're these are hard stuff right but there are good ways to dealing with it's like you not having a solid a platform that deals with back pressure in a in a in a good way that applying things like the circuit breaker it's a graceful degradation if things go wrong if you can like back off you can be either be I mean throwing messages on the floor or it can be it can be like presenting the user with their with a with a we sort of the page that says are responsive for now or whatever I mean fail wail I don't know but but it's better to do that then the slowing down the experience for all users right so there's a lot of tricks you can you can do there and that the important thing I think is to build really reactive applications you need all of these for and I'm proud to say that akka supports them all so so let's let's let's get down to it starting with about with with event-driven then we already saw recovers or the basics well a sinker is non-blocking loosely coupled architectures is something that neither the full force out of this as I said and that means that the systems are thinking might in my view at least easier to write they easier to understand because I mean you can actually reason about components in isolation is you don't have this like this coupled monolithic mess and and it's also easier to maintain I think and and evolve over time and some important things here is that if your system is event-driven then it is concurrent by design if you use to write concurrent applications in Java like the old-school way or using threads and locks I mean I mean I can rant on that for for an hour but I will save you that but I'll just say that I think that that is just a low the wrong level of it written the wrong level of abstraction you shouldn't have to like tinker around with with with with with critical sections I mean you can access these data an hour whatever leave semaphores to have uh some counters or whatever it's all too little too low-level but isn't if you instead use event-driven architecture then the runtime takes care of all that nasty details for you and you can you tend to think in workflow I mean and workflow is actually application semantics right it's about what is the application all about I mean a talks to be and he talks back to today and he sends a message to see or whatever so it's actually that means that if you just think about the system in this way and in designer applications around that then then then then you then you will get concurrency and paralyzation if that's if that possible for free basically just falls out which is really really nice so now I'm going to talk a little bit I'm gonna sum the story of this talk I'm going to go through all of the traits I'm gonna see how akka supports them and how and how you then can can can go about writing in a an application using ARCA there is a venturi of a scaleable resilient and responsive okay so the actor model arc is all about the actor model acting all this old stuff it's like from 73 I think Carl Hewitt threw out the first paper on actors and and it's a computational model that that embodies three things processing storage and communication they are similar to objects but they but they are they are m and true objects that say but they are not like the year your like your old Java and C++ object because because communication is asynchronous storage is sharing nothing and and processing is sort of single threaded so you don't get parallelization or even concurrency no concurrence you get but you don't get parallelization we're just running one actor you need you need you need two five ten and these in the nice thing is that actress at least implemented naka or extremely lightweight so you have a design to Lee which you can in which you can actually create millions of actors I mean concurrently living actors so we which is really nice so actor supports three different axioms and these are the things that I'm going to look base the actor how explore how talk about actor is about right so when it when an actor receives the message you can do three things you can create new actors you can send messages to accurate knows that it has references to and it can do once or a funky thing that it's extremely useful and the LeFleur brings all the dynamics city into the actor world look at its again you can you can change this behavior for the next message when it receives the message you can say for the next message I want to redefine the way my myself right I can change the way I should respond if the next message camp comes in so if you look the essence of an actor is that free if you first define it this stays zero then you then then you can create the actors you can send messages to them you can you can become that is redefine your own behavior and the first thing that is the fourth thing that is not part of of coral hue its original definition of an actor but I think it's probably the most interesting one the Soraa came out of the work done by Joe Armstrong at Erlang is supervision because you have these isolated components like falls are perfectly to to have the two built in a sane failure model into that and and that's something archives also adopted we borrowed a lot from earlier i/o Joe a lot so okay so if we start with defining an actor the first thing you need to do is define the actors protocol which messages should it be able to respond to so you define its messages in this case I do I'm creating a simple greeting actor you can respond to a single message called greeting that holds the string saying who it should it should greet okay yeah so you just create a regular scala class extends actor in this case almost extending after logging so I can long things to the eventbus by the way all logging in Anaka is done right it's it's it's it's done completely asynchronous and and it's not blocking on IO or anything like that and this so that the second thing you create you do then is you define the actors behavior how should the actor react to the message it receives okay so here here I here I'm using Regulus call a case who's Patrick matching some saying if I receive a greeting message here then I just log it basically I mean y'all know Scala I don't need to explain the details the skull are sort of at least it's almost self-explanatory because because Galway is so simple right yeah I think it is it sounded like a joke but yeah sorry anyway so so how do I create this actor now okay for the first thing you do is you create an actor system actors they do come in systems seeing like called Hewitt said that in this or in the presentation now so they do want actors no actor actor soul becomes in systems and in which they in which they communicate so you create an actress system and all that actor system you start creating your application you can if you use spring you can see like the springs being factory well I could container for your application but I don't like the even though if I don't likely to use to use the word container so you you just use the system and then you use this methods factory method will act all in which you first pass in a props that's the actor configuration you can configure the actor in many various ways I'll show you later some examples of that in this case I'm just I'm just saying that I want to create an actor type greeting actor that's all no the only thing I'm doing then I'm giving the actor name names are important in our in our car names are although I mean things to you names are for assembly ward what defines the hierarchy of actors names are what we use in the configuration when you want to configure the actor system from the outside the names are also what do you use even like the run time to look actors out and so on okay so out of this actor if no actor actor or method comes in actor ref so the greeting the greeter here field thus entire the type of greeting actor it has to type actor ref so you have a level of indirection here which is extremely important because it's this level of indirection that allows us to do some really cool stuff for example when I create this actor I don't know because I know now because I wrote it but I don't have to know if that actor actually running in process or if it's running on another machine it all depends how its configured from the outside and what's really nice is that I can actually that can actually change right there the actor can move depending on if a node went down they can be restarted or now the machine or or or it can actually be be moved around and so on by the runtime to optimize the behavior so then it is this decoupling that makes that possible so so the only thing you have is there are references to actors the completely decoupled from their location this is exactly how our line works as well if you know that they have the appeared process IDs they do send messages to okay but it's also this this loose coupling is also extremely important when it comes to the fail that the failure model assess we will see because this allows the actor to be restarted without the client knowing because he yes he just holds the reference to the actor not the actual instance okay so actor can form hierarchies the top-level actor in our system sorry it's always this is Guardian okay so when you create an actor using actor all from system like here then you get a child actor of this Guardian okay but each actor also has has a context it's implicit parameter inside the actor see if I if I call the the actor of method on the context then like what I'm doing anything creating a child oh my oh my oh my self so I become the parent and the disc becomes the the child so actors form hierarchies in this way okay file system so if you want to in the configure or look up the the foo actor then you just say /foo and and his his child is / true / a and in / su / a / b and so on and what's interesting is that you can actually use use this to to query out multiple actors as well you can say / food / a star for example that will give you an actress selection back with with both of the children or both foo in this case and so on you can fish if you send a message to that accurate selection you will reach both of them and and and so on so it's very it's very very intuitive way of navigating your system and get a good architecture going but the most important thing and the reason why we what we do it like this is for the failure model that we're talking about later okay so now we have defined an actor we created it let's let's send a message then we're using this funky operator called bang but if you don't like that you can just say to write te ll tell because that's essentially what you're doing you're telling them a this actor to to to to around this mouth this message this method returns immediately this is what a funky scholar syntax you know you can actually write greet or dot and then pepper a bang and then parentheses it's just a regular method call and this method call returns immediately after putting the message on the mailbox of this actor so that's all the job of this method then there's this scheduler we called article dispatchers that eventually when it finds suitable will will then take this message out of the mailbox and put it into the actors receive method to be run he puts him on a thread and have him run on that the interesting thing is that it's only when the action when the actor is running on their thread sorry when they actually is processing the message that is actually running on the thread else is idle and then the only resource it consumes its memory it doesn't sit on the thread it doesn't sit holding on any resources or anything like that and this is what allows you to create millions of actors easily it's just matter of RAM and of course how much they act each actor is doing because the waiting time can be can be severe if all of them need to like music or for ten minutes and of course but but you know what I'm saying so so this is sending we also have another source sending a way of sending messages could ask if you can use a s key a s K or the question mark operator and this is to model the request response this or type of conversation that you might have and then ishaq you'll return to scholar future that you can know that you can wait on not block on though please is that register callback or something or use for comprehensions to to to to work with that yep so so if you bring everything together this is just a small actor system I'm creating a message and I'm creating the actor and then the system and then I'm sitting then the actor out of the system and then sending something to it so I was planning if he thinks if the demo demons are with me just to to to run a short demo your so on it oops I need to switch screen so here we go like you see it over there but knocked over here okay let's see let's try to us to luck dude another way mm-hmm we'll see disco tune I can't see it that's because it's mirrored okay well there you go sorry about that okay so okay so now can't see it on this one now yeah it's crashed Google hangout crashed we're using Google hangout here to get the screen doesn't get to get on all the screens around here so okay here we go so none of the wrists solution might be leave a small bit okay but now let's create a class here then scholar class this is sort of a game of ping pong I was thinking showing so can you see that or should I increase the font is this enough it's good cool so so first I have some some snippets here to make it go faster so so the first thing I'm just doing I'm just I'm just creating an app and I'm creating a ball I mean eating inning in a game of ping pong you won't you won I have a bowl right they've been creating it the simple actor system I'm calling a ping pong and here I'm instantiating both of my actors I'm calling it ping and I'm calling it pong and it's an instance of paying an instance of pong okay so I'm just going to import these things there we go so let's now create the the ping actor for example on to do that by extending actor I'm creating a regular class this actor is state fully tests it's a very simple state called the counter that's or a counselor as many a sort of yeah times it hits the ball if you say so then I have this receive method here so what what do I want to do with that I want to be able to accept and do something when I receive when someone sends me a ball message when someone puts that on the mailbox and I'll get that okay so what I'm doing then is right in this case I'm can you hear me [Music] okay sure so so here when I when I receive the ball message oops there you go and when I receive the ball when I receive the mic I'm incrementing the counter and just printing out pain counter and then I'm sleeping you should you should never sleep in an actor but just for demonstration the purposes that's what I'm doing here to actually everything else things run so extremely fast right because akka is so wickedly fast so yeah now got that said as well okay so here we have our little actor let's now do placate thus this so oops and then call it the pong so and Sierra two actors now if we now send ping the bowl message okay and run this thing what's gonna happen then my first music compiled wickedly fast yeah so so it just print prints ping counter one right I don't know if you hardly can see that there but yeah where you believe me thanks so so the reason your piece of Israel is of course that we need we need both of these to communicate somehow and now the the ping actor we get the ball and it'll sleep and then would do nothing so so so akka has this funky thing it was really nice thing called sender so whenever send a message the sender of that message is passed along with the message implicitly and you can just use it it's quite it is contextual right so if you the sender can change between each reception of each of each message if there are it's come for multiple parties so if you use this then you should always store the way if you wanna use it for later and so on this is also why we already use this this these parentheses to show that it's potential dangerous like it's side affecting it's using a bar underneath so so so here I'm now sending the ball back and I want to do these of course do this for these other guys well probably want to do it actually before sleeping so now it's let's run this again you will see so I don't know if you can see that here but but things didn't go that well either because because here we here we here it says this message was not delivered he was still delivered to inactive or dead letters because the thing is that if you if you look at this little application here who is actually the sender of this message it's not an actor right is the object ping-pong so so how can we get yet to get out of this problem yeah akka has a way of sort of passing in OOP still tell passing in the sender reference this as a second argument to to tell so if you're now sent right pawn here then sort of well then we bootstrap the whole thing now we can if you don't like tell you can do you can do it like this as well so if we now the roundest thing should just fit ourselves to get a nice game of ping-pong yeah I'm really excited compiler weird did I do something wrong Thanks because all you akka experts what did I do wrong I can't now there we go okay so it was just hanging so here is the area ourselves a little game of ping pong then this North let's go back to the slides I probably here we go okay so the next thing I want to talk about this is the third sort of thing that the actor can do is I said read redefine itself for the next message so that he will receive that is using something called become okay so become is it is amiss is a method on context so in this simple application what we do here is if there we we define two different behaviors happy and angry and and we set the initial the initials or behavior to happy by by setting it like receive equal happy here so so when when when we receive a regular greeting and we just print things out but then if we for reason if you instead receive the angry object here see we will do context become angry this means that you actually swap the receive from this to this and here we have a completely different new protocol now clang gree that receives when it receives a greeting reprints and out go away this is it and and and also accepts the happy message to be able to go back to the already original state right so you can see here this is a very nice way to mold state machines like finite state machines know the senator knows nothing about it yeah so akka also has an FSM module if you have some more advanced and so type of behavior and and and and and sense and state transitions right but if for simple things are really I really like this way of modeling it and and you can use this for really advanced stuff if an actor for example do sort of finds out that it's overloaded what he can do it cancer because become a router in we should response up like ten different children start starts like refined like Fanning our work instead of just doing the work himself and these this can be on different machines right and then and then if things go down or if you like to send him a message that it should actually let me go back to a real estate you can it can swap to behavior that first shots down the messages and then go back to a real estate processing all the message all the messages himself and so this gives a lot of power a lot of dynamics it into your into your into your application yeah there is the question was if there are conflicts but there is a definite order right the ordering of the messages in the mailbox and that is not to be tampered with so so so there is the thing is that there is there's a complete non determinism outside the actor but as soon as the messages are put in the mailbox it's it's fully deterministic they are processed in in order and and by a single thread right it might actually be scheduled on different threads between each execution but but but but still is fully detector fully deterministic it's that's a good question thanks so go to resilience I really think that as I said resilience is something that should be designed around right it's too important to be bolted on afterwards it should be part of the application designer this is why we have it as a like first-class thing you know in in in in ARCA really a first class citizen in in how you design your applications from from from start and it gives you this complete serve isolation that I talked about avoiding cascading failures each actor runs in complete isolation the only way to find out about the state of another actor or anything but an actor is to ask him to send him in a message and get something returned asynchronously so so since the doctors already is completely isolated islands that means they can live and die in isolation as I said and that's an ethics tree extremely powerful so one way of thinking about them this the sort of way of designing replication is just think of how a vending machine works you might have this this this this programmer he's been converted he's like eager for coffee and and he wasn't he wasn't get his coffee he he comes up to the to the vending machine and he puts in a lock one coin let's save the machine except I mean requires you to put like two two quarters and he puts one quarter in what's gonna happen then he will get a notification from the machines that add more coins because because it's sort of validation error he puts in one coin and presses coffee and and and what in the machines there's no I mean you just gave me one I need to so so and then if you diffident puts in two then it gets his coffee right but let's think if it becomes to the mint to the machine the second day in the inserts is not two coins and they now gets like an out of coffee bean exception right there yes just throw it in his face what would he do I mean most programmers they don't know I mean where to get the coffee I mean we have Adrian he actually I actually asked him today when the coffee was out he just ran down and bought more coffee but that's that's not that usual actually actually that doesn't happen in Sweden that's usually so I think that's that's that's not what you would expect right you might get and sort of a notification said we're out of beans but you are you as a programmer or USDA user of this vending machine are not expected to go up and buy beans poured into the coffee machine and then continue instead I think that's completely wrong so instead this should be like sent out to a service engineer somehow like a service guy then through the beeper or something like that he might be here that he might carry and he goes in and fixes this add more beans and and and then the programmer is happy I think this is really the way we should design software okay meaning that if we're the client in the service the client asks the service for something like validation errors should go to the client I mean things that the client actually are in charge of if it puts in the bad email address or whatever it's natural that the little client should actually deal with that right but if there is an application error it doesn't make any sense to just throw that into the users or the you through the user of the of the service face here he might have no clue how that thing is implemented right and just reading the docs or whatever that's just wrong I think instead it should be dedicated channel to someone watching that component to someone that actually knows how its implemented what if it's possible to do something in the first place and if you look at how systems are usually written in Java it is like like that or that the application errors is just thrown in your face right and the the main problem is that there is not really much you can do because because in if you redirect a regular job application then you have that that single thread that you're running your code is the only thing you got exceptions do not propagate across threads so the only way to survive the context and not have like things like these cascading failures that will get you like the whole stack is like it was blown out is to add your try-catch statements and things since things can fail basically anywhere you need to do you need to add these try-catch statement everywhere so it means that the at first it's scatter all over your own codebase and secondly is tangled with your business logic in the business logic might be very clean but if it's like half of the room or sometimes like 70% just error handling usually not doing anything useful at least in most cases very very very rarely I've seen actually used to be able to fully recover and know what they know what to do with with the exception being thrown if we have a model where failure is isolated you have this release components that are illiberal and run in isolation and have a mechanism of refighting failure as a message and sending it somewhere else that means that then the failure can be dealt with somewhere else okay this is extremely important then you don't need to like for all four for all good reasons like is keep this thread alive because that's the only thing I got so so instead application errors should go upwards in the dedicated error channel up to what we call the supervisor okay and the supervisor should manage the failure so this is very much how how how ship construction works have you ever had worked for ages like this this this this page is images from 1940 when they were the server built this uses Dakota so I think the right ways as I said to reiterate hammering in a point isolate the failure compartmentalize even if you don't use actors this is a solid pattern that you should you should apply manage failure locally and this will get this will make you going cascading failures it's so this way of working has it has a name called bulk heading so if you want to read more about that one example actually that some people bring up as bulk heading is the Titanic you know but the Titanic actually sank but you know why because it didn't have proper bark adding these these these bulkheads the words in between didn't go all the way up so so so so well so if when when when when one of the compartments started filling up and and then then the ship started to look tilt a little bit it spilled over to the next one to the next one to the next one and they're in the whole ship sank so I think Titanic is really an excellent example of cascading failures how to not do things if you have fully isolated components this cannot happen right so we need supervisors that deals with a problem okay and in ARCA you know bringing beast now we're coming back to this to this hierarchies right if I could if I create a child if I create an actor he because my child but not only that Anaka then I saw supervise him that is not optional right that's mandatory so we have mandatory supervision Erlang that's one of the design flaws I think of Erlang he but if I should should complain a little bit about her Lange is that in Erlang is optional we have that Enoch earlier but but users actually did it wrong so much so we win and we enforced it you can't create an actor in our car is superb iced it will be supervised by someone if it's a system actor it will be supervised by the top-level Gorgon okay so how does it look if you if you don't do anything it's still supervised because each actor has the default supervisor that is doesn't okay job right basically what it does is that it's that if an actor fails if it throws it away restarts a new one and let it go but but but I think as I said it's resilience is too important to just rely on defaults it's actually the application semantics so I think you should override that and and and and define a supervisor strategy that actually has semantics that actually deals with your work with application is really doing so in this in this Apple in this example here you can see we're creating supervised a strategy that in case of an arithmetic exception then it's probably user input valid if some some something's wrong then I'll then I will just resume the actor and hope that doesn't happen again right in the next time the user puts in something if it's a null pointer I probably have corrupt state somehow then a restarting and if it's none of these then is something I didn't expect then I can't handle it so what I'm gonna do then is escalate up to my parent hopefully that guy knows right Lisa look it's above my pay grade you can say I don't have enough knowledge or enough power to deal with the problem one example is for example out of memory exception that's not possible to deal with so what's gonna happen if if a now the memory happens in be down there right it will escalate all the way up to the to the top level Guardian and the JVM will exit but the nice thing is that since supervision is based on message passing and message passing works across machines right as across course that means that this is the failure model the way you write your system actually works equally with the same semantics across machines so you can have this act decreased bad multiple machines that this means that if one we and blows up someone will notice it's not just just your regular like try-catch is not caught if it's not my court like it's like to silence the best the best you see is like it lucky like you like stack trace in your console all the way up to your executor service all right it doesn't happen we'll have to deal with the problem oh no I can't no he has to do like himself no he no but everything no no but but but but each one so no he can't do that I mean it's only one level one level deep right so it's you have to design replication in a way that will deal with that and I think it will be too messy to have to have a supervisor like John pirate key levels and them do do funky stuff so okay so in this in this simple case the actor I'm creating with context that's a child of mine will be the managed under this supervisor hierarchy now a supervisor strategy okay so that is from there from there from the parent perspective from the client from the from the child's perspective he can actually do some some stuff he can cook into the lifecycle callbacks to do something just before his killed for example on a restart then he can clean up things usually the pre restart callback you can get the reason for things so you know how did how to deal with that and I had citizens and the same thing with the post-race to arrive before he's new when a new instance is created the post restart will be called and before he just hands out to handle messages it to reinitialize ation as well and there are more callbacks than this so this is so now we talked about sort of the hierarchical way of managing failure sometimes you're perhaps who want to if on the same level you might want to monitor a child or someone else in the hierarchy did I see if that guy's goes down then I know I mean I should shut down for some reason is no purpose in life now anymore when my buddy's dead and and what what you do then is that you add you use this context of watch so you can watch another actor's life cycle by just passing in the nets or their actor reference to that into the watch method and then you will get a terminated message that's down here whenever the guy dies reactor exits somehow okay before your threats a lot for scaling up usually and then you have like you some fancy clustering tools messaging middleware or whatever to scale out right and this that way you are very constrained you can't move you can't move components around and you don't have the flexibility to do things sort of to change the topology at configuration time things are hard-coded I think that's completely wrong you shouldn't a car code in the in the topology when you're into replication when you're when you're right it when it wasn't where they were where to decide it it should be something that can be configured you might want to deploy it didn't insert in the different kind of upset things and with very different sort of ways of looking at the how the application is is deployed and even even more when you have externalizes like that you can actually change the topology at a runtime and this gives you the elasticity you know elasticity there's a little the promise of cloud computing you spin up instances like really really fast need to take them down yeah okay that's all nice but our applications need to be up for the job to to to to do that as well in our case there are the tools they're also that's really nice but our K is one of them that actually allows you from with them from within make that happen right you can just add add notes and have an mu have actors to be moved out and vice versa shrink the cluster and so on so I think you need to deal with scale up and scale out and they are as I said essentially the same thing to reiterate what I said in the intro we need to minimize contention and maximize locality of reference and sure nothing really really is the way to go here that gives you both of these things almost almost for free at least if you have a very efficient implementation backing these concepts if you don't do that armed all would come down and how come and hunt you down I mean he will hit you in the head hard like like like like Martin Thompson said that if I can quote him I mean scalability is the house were armed all wakes up early in the morning and the rules with an iron fist alright so and if you did that's a no armed alts law it's it's it's essentially saying that that you're limited by the speed up your application in so if if your application is 95% parallel if you look at it's hard to turn around like this if your application is 95% parallel that's pretty good right if 5% it's not parallel it might be oh you're I don't know I mean your database calls or something like that then in this case I mean the actually access is might vary depending on the application of course but in this in this example here you're limited to 20% speed-up in this case at 1024 course if you go beyond that it doesn't matter how many course you add can add a million course you will not get more than 20% speed-up and if your application is like 10 percent not parallel is you're limited to 10 percent speed-up and so on so it really really matters that your application is reactive and event-driven a non-blocking all the way down every single layer need to participate else yeah you know arm dolls coming so so how do we deal with scalability knockout we have various ways you can define an actor to be a router for example this means that he want to process any messages himself instead he will create children fan-out work to all of these children we have various strategies for routing sort of algorithms round robin is one I'm showing here you go random broadcast we have a scatter gather there are there are also some very interesting cluster routers that are fully adaptive looking at metrics how the application is being used like CPU mailbox sizes or RAM and stuffing and then and and you can configure its or the thresholds how it should act if you spin up more and so on a route it can also be pooled and you can set like Max and low threat low water marks and stuff like that all right but what's also very interesting is that you can do that from the outside and you go and you can do it from code it doesn't matter sometimes you wanted to build you want to get good mood defaults in the code we also want to override those from the configuration file archives using type safe config library there's a job a complete library that's brilliant written by one of our engineers have a Pennington and it's it's it's actually starting to be used way outside not just types if not a scholar but also in the I mean across the whole Java ecosystem really encouraged to take a look at it we use it play uses it so this is an example of how it can look there so here you see also here are the names come into play here I'm saying path to actor that's like my path like a slash B star C that's it that's the path of this actor that I want to now configure okay so if outing is not enough if I want to really scale out not just up then we just can just just turn on clustering there's also done in the configuration file you just say as my actor if like the provider should be cluster actor if provider and of course it has more configuration or piece if you want but it's really really simple everything is driven through convict through configuration and then you have a whole set of cluster routers and the only thing you do then he says I just cluster a section here that you can say for example when enable it I want to have them like max number of instances per node should be three that means that whenever you spin up new instances it will deploy three new workers of this router on that new instance but no more right and so on we also have other tools we have clustered pub/sub which is a very nice pattern that we that we have in India in the country Bligh berry in this case you tell you just create a like a distributed pub/sub extension you get the mediator and through that meteor which is also an actor you just send it a message say subscribe on a specific topic and then you register yourself self by the way is my self it's my own actor ref I should not use this in an actor because then it then then it and my my this is a fair one one one one of the corner cases one of the things you should watch your after because if you if you if you publish this somewhere then someone might actually be able to access your state from the outer should always use self so you are publishing subscribing to the topic content and from someone the one that want to rights to this contest he just get the same mediator and he says mediator publish and since the content and some message right and then everyone regardless where they are in the cluster we just received that message and if you spin up more they will label it if you receive the next message that's being sent and so on rice is fully elastic you don't need to configure anything things just work as many nodes in the cluster you have or part of me yeah the question was how what type of objects yeah you should not you should not use java stereo civilization for sure I mean that's that's a really bad ways it doesn't have any any versioning so you've got a versioning problems is really really slow for D marshalling and marshalling and so on innaka or protocol is based on protocol I mean that's a great one right but you're not you're not constrained to protobuf we have a whole set of serializers and all those are configurable from the config library as well and we can write your own right easy to write or the same thing with router so you can you can just write your own router if there are if there is an algorithm that you want that's not there so arc is very extensible definitely so other because the cluster feature that I don't have time to go through is that cluster membership is something you can hook into you can actually subscribe to messages when when knows commonly even when they've become available and not available and so on we have a concept of a leader we have we also have cluster single dump after and if people if you really want to have one guy in the whole cluster and make sure that it's only that I don't know I mean if you want to like expand massaman send send emails to your to your users or whatever you press don't want to do that on every single node right it will be duplicated by how many how many how many you know you're currently running right you want to do that from one single place and have a single writer there that can be done but Singleton's are all of course a bit tricky and then and and you shouldn't use them too much right because you get contention they we we don't want we have cost to rolls in cluster sharding if you will like split up your cluster in different regions and stuff like that all that works is really nice so yeah I'm running out of time so how much should I go on okay deke huh I go you can leave if you want I won't be offended I would probably left by now acha persistence that is that is the the last thing we added in will be working on the food that the whole fall and to just give a short overview it I'm really excited about it it's it's it really fits or adds a missing piece in the in the arcus or a landscape right I I think it doesn't do persistence on I mean the actual persistence on this I can say it has a whole set of journals that actually does that with various implementations so it so it it's really not a really full-blown there's not a database right it's the pattern that we said we've seen clients use I mean over the years it implements event sourcing CQRS style and and and it does that through two different models command sourcing which means that that you will get at least once guarantees and event sourcing which which is the opposite at most once guarantees sometimes you need either or right and sometimes you can use them together and and and and we used to have something called akka durable mail boxes which we now deprecated this is a lot it's a way better replacement for those if you use those earlier it's it you can use this for like replication you can use this for four four four so lot better I would say guarantee deliver because I don't believe in that but but the not reliable delivery really and and and the building box here is that it is an event sourced and they're called events fourth processor and we have the processor is just the command sourcing right some others some other interesting things we have web so for example view the component that you can just basically for free just hook in to listen on the same journalist as the processor and this is perfect for like 64 a CQRS like command the query cigarettes segregation if you know what that is or replication or something like that like the - to listen in on the honor on an event stream and do something without affecting it so so the benefits of a cure persistence I say it's like it's really is you have it the nice thing is that every single state change everything that happened in the active system is persisted that means a bulletproof historical record nothing is lost as I said in the early in in in in in the first 10 minutes was that you know that disk is really really cheap and RAM is cheap there's no reason to throw away data anymore I mean all this classic sequel databases right they use induced in place update there was a reason they did that because memory was a very constrained memory but I think that's fundamentally the wrong way to approach the problem nowadays there's no reason to throw away data I mean Pat Helen said it really really really well then he said the classic you're like a relation database is just sort of a snapshot of a cache of the event log no the cache of the snapshot of the event log right so it's a very limited view or it's actually have happened in your application everything is there in the event log why not persist that instead and then have have per sample of you right reading the event log updating your database for query that's CQRS for example it's as freeze actually literally free to put in like you've you listening to the event log for auditing or for or for replication if you want to replicate the event stream into something over here and so on it gives a lot of a lot of flexibility it's easier to test the systems I think as well and you don't have this this like this orb object relational mismatch in which in which you have your advanced streams they are they are materialized as objects and then you have this clunky thing you need to persist when you have this beautiful event stream and the interesting thing is that databases they are using event logging men append only log really really efficient structures on the under the hood that's how they are implemented it's just that they're all the tools that you are giving are in place updates where I should lose data every time you do an update they need to go up all funky stuff to keep history around right I think that's wrong and I think that acha persistence is one nicer way of solving the problem I don't know if I should go into detail I'm running a little bit out of time Patrick the Akademi did an excellent webinar on this in depth so is it recorded yet yeah so you should you should you should go and see that instead it's like Lana or in depth the lack of persistence rather than me just rambling about the same things yeah you have to read this paper though I mean it's it was extremely influential paper bye-bye Patel and Patel is a guy be how you know he worked with team gray on tandem systems he's like he's he knew he knows this stuff he knows transactions really it was wrong part of the old of the system to really build these sort of pioneer transactions but he doesn't believe in transactions then they're longer distributed transactions or just broken right if you something like Pat hell and even goes out to record and says that and argues for in lucky doesn't this paper form model where you have these share nothing entities that have that forms this or the consistency boundaries right because because some people ask about what about transactions in the system like this your transit the way you sort of defining transactions the way the way you the way you model your system is that you work with with in DVDs languages could make will aggregate roots and aggregate roots is like a facade to one entity or a set of entities and within this you have four strong consistency right but it's too expensive to have full consistency in your whole application then you can't stripe things then you can put things to multiple machines or whatever so then you have eventual consistency between these aggregate roots and that have implications on design and things are not always easy right if you want to the scale systems and and and then you need to think right tough luck right and to think about the problem in a slightly different way but I really believe that that if you do this there's basically no limit is to scalability because you get to the scale as much as to many as many notes that you have because you can fully stripe things and each aggregate route can have their own journal we which use the single writer principle to minimize contention just writing like crazy to this journal I'm contended right there's a lot of good things coming out of using the right Packer and so I think if you have the solid platform supporting these patterns okay yeah I was actually planning doing some blog Cody I'm not going to do that now because we were running out of time and I know she's planning on doing I'm gonna persisting this game of ping pong first using a processor and then using an event-based processor but I'm gonna leave that as an exercise for the listener I can give out the sort of the the right way of doing it later on if you can we can compare right so anyway so if we please think we're still going to look at this just briefly how it looks so if you have a regular actor you can just swap the actor to not extend actor of aesthetics and processor if you want to use command sourcing or swap two events source to processor if you want to use event sourcing so the difference between an event sourcing if we go and command sourcing if we're gonna do explain it briefly is that with command sourcing you persist ahead of time it had before the actual event enters your system you're persisting it right and this means that if if there is you actually do it even before validation right that means that that when the see when the event then enters the system things can actually fail and things might actually do side effecting right so so so so so either use the idempotent messages and command sourcing but if that's not possible then it's usually rely on only the own events or single event sourcing decouples stay changes from from side-effects so in with event sourcing yes you'll receive the outside stimuli some sort of event then you then you do your state change and then you create an after-the-fact event this has happened this is the fact right for example the ball here if I if I would receive that I would first then update my state then I would if that went well I would generate an AB or received event I would persist that event and then Ryan my run around my side effect in this case printing threw something out and then also extending the ball to my body again so they are very different approaches and you can use them together if you like and and and they have very different purposes but but but depends on what more the application need to do so so so if you use a processor the only thing you need to do is you have to wrap the message in persistent and then send it off it means that you can actually distinguish between persistent messages see at the bottom here I'm sending a persistent food to the to the to this processor that will be persisted journal if I just sending your regular string is not persistent and then it will not be journal so you can distinguish between these with event sourcing you don't need to wrap things with persistent there it will just accept regular messages and and and then generate events on the under the hood so so in a way I sort of like event sourcing better but-but-but-but from because it's cleaner I think yeah the application yeah you generate the events you're in full control they could the question was if the events are generated on your under the hood and so that's not possible because I mean it might be that one command coming in can generate one event or ten events I don't know that depends what you want to how you want to like split it up and how you want to define that this the semantics of the application right so that is very different so yeah the view is also adjusted just an actor and and and it subscribes on this on the same processor ID as a processor he wants to read from and then it reads directly from the journal so an overhead no overhead at all it's really listening in completely free the this channel we can we can skip yeah so let's give me the demo responsiveness necessity nothing this would be would matter if our systems can't be responsive at all times I think when things will go well yeah that's easy right but especially the responsive when things fail when there are unpredictable things right when when they're spikes or whatever when things happen that we didn't know what's going to happen and and I think that there are some nice ways some sort of well-established Patridge duty to deal with this like my back pressure is what is one of them a NACA has that built in in its mailboxes first you can define having set up you can actually define which mailbox your actual actor should have if you should have like an unbounded mailbox that is actually a default but it's not a really good general purpose thing because it doesn't have this this little T's boundedness right you can actually run out of memory so it's it's better to like think things through what kind of guarantee should I have and and so on and and and and have it and have a bounded mailbox you can also built in things like like we have a circuit breaker pattern for example when when things go haywire you can actually back off and and and and do something you would there acceptable situation and then read and then we try and and so on the dispatchers that our scheduler have lots of chemical configuration options as well that helps you deal with these things yeah I usually try to apply little littles law if you know about that the so you could you can look it after that that's a really nice small sort of a mathematical model II wish you in which you can sort of measure latency and measure response time and and and and c-word you should actually cap your your Q's in order to be able to maintain their Sal ace that the the application requires yeah go to the reactive manifesto there's a link there right so so so we just googled it little slaw little slaw yeah yeah so yeah here's a reactive manifesto so most of the things I've talked about here I mean that talks about principles or actually defined in this reactive manifesto it's we we built this we actually we we we started this reactor manifesto just just to get a comic book vocabulary to have one single way of talking about these things to have a way of categorizing what we think needs to be done to be able to solve problems today right because as always I mean when there's a need I mean solutions pop up and and different solutions have popped up in different themes or communities in sub communities or larger communities if it languages or whatever and all that is good right diversity is a good thing but-but-but-but from from the users perspective for you guys I mean you might get really confused because they're they talk about things in different ways reacting manifestos right it's a way to rock bring the communities together one way of talking about things that make sense and also advocating I mean trying to educate people about what we believe in at least so typesafe is is a reason excellent as soon as you probably know I mean reactive reactive platform Ark is one piece in the past all the wheels I play with as reactive all the way down gives you like we actually composition all over the over the web of the web request it has it test like nice nice support for streaming in a fully non-blocking fashion and so on it's all based on both scholars well that's why we all here right the scala meetup scholars of secret sauce it's really what makes this happen what really enables us to do all this really cool stuff I think will be a lot harder in job actually so and and if you wanted to try things out you should you should download the types of activator we have around eight two different templates now covering basically everything you might you might want to do I mean with covering the latest like a persistence stuff covering like persisting with or really war with anything they're covering or writing system and applications using knockout J as angular and then and there were like spark for big data and whatever and it has to write really nice UI I don't know if you've seen it but gives you this interactive tutorial thing right where you can actually edit the tutorial in a way you're right and see see changes happen and if you're really happy with the result and want to go on you can just take that pro user really solid real SPT scholar property you can just continue working on so it's also like a templates for bootstrapping applications so yeah thanks for your patience was a long talk I didn't go through my future's converged how do you see our car vs. that combination and on the other can we see spark coming up which is kind of big data but it's also going in the same direction but it's already resilient yeah and you know if our streaming which is kind of responsive so I wonder what is your take of how these three things kind of yeah this is the question University repeated was was every week we have our car we have we have other like like tools Sollux similar problems like finale or Twitter's finagle and and and likes likes likes Park for example and I think I think one one thing that we try to work with I mean it was what's bringing for example Twitter rain like we did with with it with the futures implementation they used to have their own futures like and that's that doesn't make any sense my future is like a solid things should be in the Scala library so we started this specification and and and and Twitter was part of it and and and we we actually agreed on something that that we think are good and we all use it now and that's that's that's in Finnegan now I think is at least coming and they're always like one version behind the scholars I don't know exactly details there but but I think more more of that right so same thing we're thinking hard now and how we can do streaming in Anaka for example and then and and we and and we're not really there yet but we're talking to to to companies we talking to spark I mean matei there and we're talking to people and they they're I mean sort of it tends to be about the same time that the similar problems pop up in many a minute if different players play places I just want us as a community to talk more and and and join forces on on single solutions rather that's reinvent the wheel all the time because that doesn't help with interoperability so so so we really try to encourage that I mean and and and try to be open in this type sapin and in the aqua community as well for for rockets standardization see if you may call it right makes life easier yep [Music] yeah the question was if there's a if there like an SPI for their for the photo journal right if you wanted that to be an implement and yes there is and it's fairly small I'm not going to show you the code it's just a few methods and this is what have enabled the community if you go to ARCA dot io / community you can see as you can see that's like it's like more than 10 different journal implementations of already summery I mean all written by the community and we even have customers that written proprietary once for for their custom backends and it's really easy it's really it's just delegating it formas forma from a fixed set of methods to whatever the back and they have and of course I mean in the end this is the guarantees that the backend give you how durable things really are right but it should be fairly easy yeah that's a very good question last year we acquired spray yeah spray if you for you that don't know about it is you see so he says according to the latest benchmark is the fastest through HTTP library there is on the Java platform I mean not just call up all the other on the full Java platform and so we're really excited when it when it's when we were able to start taking advantage of that so what we've been doing the let the fall and now is to have Matias in your harness the guys behind spray to improve what they have and it to make it fit very nicely interact so they've been working on that for the last six months and it's it's gonna result into into the new akka HTTP module it's going to be released in a few months basically spray but even better now this is for example where we're looking into consolidating things like like like like streaming for example a kayo used to have a pipeline abstraction that we deprecated now and because we know we're onto something lot better and that that's that's something that we're gonna like fold into like HTTP for example in a car I oh and a lot of other places perhaps even play and and so on so it's a lot of good stuff coming no more questions okay thanks for listening [Applause]