SF Scala: Krzysztof Otrebski, "Beat Aliens with Akka Cluster"

SF Scala: Krzysztof Otrebski, "Beat Aliens with Akka Cluster"

Recording: SF Scala: Krzysztof Otrebski, "Beat Aliens with Akka Cluster"

[Music] okay so title of this presentation is beat onion with a cluster because I have a investor here running on raspberry PI's and on my laptop and basically we are defending Earth from alien invasion ok so again the short introduction about akka cluster who knows what is a cluster ok so almost everyone then going to live demo and then we'll be killing or disconnecting JVMs and testing if application is resilient and self-healing okay so this is my cluster if you are not able to see it I have free raspberry PI's here and as you can see we have icons the magnet rep and demand which is important because we'll be able to match what what is on the screen with piece of hardware and few words about me I am working for a kado technology company which is building online supermarkets and automated warehouses we are based mainly in UK but we have offices in poland and working in poland in bulgaria and also in spain okay so we're gonna skip the slide because we don't have much time it's about the knowledge so it's like iceberg if you're reading blogs you're gonna just scratch the surface to you learn only twenty percent of something mo or ten percent if you want to learn more you have to play with technology you need to do some stuff and you need to fail on production as you need to learn what you have done wrong and then you're going to fail again and if you are not fired you will fit again and learn again okay it's a normal process okay so what is akka based on acha documentation the tool kit to build resilient message driven application on JVM and i'm using acha here I assume that everyone knows what the zakah and okay actor model okay i'm going to skip some slides persistent actor you know actor can recover from failure on another machine or can be restarted emigrated and we are using persistent storage some external database i am using cassandra one node cluster on my laptop very simple but efficient for this presentation okay acha cluster it's supposed to be fault tolerant and have automatic failure detection and we are going to check this on this presentation charting are you familiar with sharding okay so basically sharding works in this way that we are sending message to actor and you don't care about physical location charting distribute actors on some nodes you don't know which one exactly you just send message for example message a to actor a to chardon coordinator and it takes ID of this actor based on message you just provide some function so message I for actor a and forward it you don't care about physical location you don't care on which node with actor is running and that's why I have these icons on the Raspberry Pi and on a screen so we can match it and the same with message to actor see and message to actor be okay cluster singlet on it's a similar thing like a sharding but you have only one instance of specific actor it sounds like bottleneck and we will see how it behaves if you killed sharding in a cluster or you killed actor which is run as a singlet on as a cluster single actor okay it is similar thing so it runs in a cluster only ones you don't care about physical location you just have ID of this single item okay and this application reactive miss will defend I have created this for fun after some training advanced akka training so it helps me to visualize what is going on in a cluster in terms of failures and my class is so I have free raspberry PI's and I have two nodes running on my mark it's a common line GUI and go in a swing which is running also game engine and I have a Cassandra as a persistence so basic message flow for the game engine and this game engine is sending message to towers defense towers okay you can see many defense towers and game engine sending message to this defense towers about what is situation about enemy missiles bombs and send the same message also to status keeper which is responsible to get all the data so it can be represented on you I and defense tower is reacting okay I want to fire a rocket Justin firm game engine about it and also sending it states to start to skipper like just like pink so I know that this defense tower is not responding for few seconds just for visualization and go is just asking status keeper Artur about state give me actual state what do you know and render on a screen a detector is very simple okay you can skip it okay so live demo let me start game again ok we have cities like an mr. defend we have different hours computers is trying to defend our cities and as you can see we have icons below defense towers like where the diamond and we have a diamond here so it means that this defense tower is running on this piece of hardware and what else do we have here we have lists of raspberry PI's green it means that it's okay orange it means ok this note is unreachable from cluster perspective and red it's dropped I'll have also information how many different style actors are running on this note so if i highlight this raspberry pi I can see towers running on this piece of hardware and one more thing towers are persistent actors so as you can see this defense tower is on level 4 because it's learning how to intercept enemy missiles and have angle error twenty percent and experience is also expressed as with blue lines below defense towers so the fatter is the more experienced so this is another for with this range and this defense towers on level one range is lower okay so we can move on so this is one of our notes yes as you can see it works perfectly network is stable power is stable yes and so this is a network cable you don't expect that something happened in your data center yes but if lady is cleaning and just hit something it happens yes so what happened as you can see defense tower now I read because i'm not receiving status updated updates from this defense tower and this note is was orange and now is red because it was dropped right now and shouting things ok i have actors running on diamond for example this but diamond is dropped so now our car shoot me great these actors two different notes okay it's done and it's blinking how many messages were lost because there is no delivery guarantee okay we are always expecting that it will be delivered but sometimes it's not okay so this was dropped because of network issue but this node can have problems with for example power yes it happens and situation is the same okay as you can see this note is arranged so cluster nodes that something is wrong and you can see different stour are showing counter how how long we don't have update and as you can see this is a level of this defense tower it should be the same after recover we have to wait a few more seconds I know how many depends on a cluster its internal like I think you can tune it okay let's give it a few seconds maybe few more as you can see cluster can be working in a different way that we are expecting it we are expecting that it's fault-tolerant it works but as you can see with some delight ok I think I have overloaded this small Raspberry Pi let me connect again the power to mark net it should reboot and start and I will connect again died not what do you think we did recover will it connect quickly to the cluster after just reconnecting network may be the same now it will not connect ok I will not connect because acha will do not allow you a caster to connect again if you are dropped from a cluster we are dead to it because it's a situation called split brain I have defense time around here i have the same defense tower run now on Ruby ok so it was like it is this is persistent actor yeah and I didn't kill the sector and just disconnected network but from clutter perspective the doctor is missing in action so after specified time it will recover this actor on different node from last non-state but this actor have us as now two separate lives yes so this is complicated situation where is split brainy resolver to do something with it we are not using it so I'm not the expedient and I'm not believe in this solution that it will be automatic you need to provide something how to recover from this kind of situation ok a very limited time you start again ok so diamond is not connected again i will just restart it with removing power as you can CI co has some issues right now about how to distribute actors maybe it will recover it should recover from it but what as we can run as i said you have a cluster singlet on in this cluster this is actor which is getting all information about situation state and it used to render you I so I can kill the sector yes this is cluster singlet on and what's now we're asking the doctor what is the state of gain right now and no answer so in a sharding if one note is killed in this example one field of customer will lose functionality will lose one filter function of customer force short time but if you killed a singlet on you will lose this functionality for our customers yes ok now it's recovered I am now it's back down okay sometimes it's surprising okay how much time we have 20 minutes plenty them ok I have lists of things were what we can do with it but we don't go with all points this project on github so we can experiment with it so what else you can do you can click a Sandra for example clip persistence and observe what will happen so basically for example sharding will collapse because it will know it will it depends on persistence you can configure sharding to use distributed data of course so but in my example I'm using Percy's Cassandra as a persistence for sharding okay and you can restart for example is Cassandra and check how long this application needs to recover from because you have some behind use to fail fast in in case of failures so it's what you can do we can overload acha clutter and look how messages are not delivered in time and [Music] yes so when you have these three raspberry PI's and say like over the course of the life of this program which is supposed to run forever these things are going to go up and down something's going to happen right yes so you're saying that the programming that will run forever will sort of diet some point right there's a program well if like you know say this is running for like three years and you're putting too much power to raspberry PI's eat up the ER now yes it can't die and you can replace a that we've an overwhelmed but you have to restart and or so you say I mean you said when you disconnect it is considered dead from that point right yes this instance of actor system is considered as a debt for a car cluster it doesn't think okay this is IP no it's if actor stem is created it will generate some random you ID as ID of this instance as and this instance of actor system will not be able to connect to the cluster again which are now competing to be one of them can be restarted started and rejoin the cluster and the situation is resolved but it cannot be done manually automatically you need some money all steps we need to have some kind of algorithm based on your business logic in order to the situation to situation for you you pull the network from cluster perspective it's a new node joining no butBut IP is it's the same in our example okay so question is how it is decided so I have slight about it so we can build it so what is going on if something goes wrong this example with some data taken from one of recording so basically at the beginning thanked 0 too much something goes wrong network disconnected of power down or JVM does were killed for a few seconds we are not aware that something goes wrong and cluster knows dr. system are gossiping with each other have you heard something about Mike have you heard something and no response okay we assume this note is missing in action yes it's not unreachable and this time is specified in your actors team configuration so in this example it's something like 20 seconds so cluster knows okay this note is on your table now I'm counting to 20 and if now this note is not coming back I will drop it at this point and decision was made act actor system was dropped now I have to recreate actors which will run on this note on over one so it's recreating them for a few seconds advanced situations back to normal operating like it was fine no issues okay rolling restarts ok where is a case when you can say ok I'm living and I have tried to do this and I don't think that I have done this perfectly good but we can try ok let's give it a few seconds so actor distribute themselves on all nodes for example this Robbie let's call leaf cluster method and when I do this it will call cluster live as you can see it work very quickly some of messages were dropped I don't know if I'm doing this correctly because cluster is now unreachable and will be dropped so I am doing something wrong because I expect if I exit cluster it should work instantly action and without this application I will do not learn about this because who looks for unlocks and production just without issues yes and question is if I'm persisting events yes with actors towers are persistent actors and I'm persisting events like okay you gain some experience last message ID stuff like this and I'm able to recreate actor with the same state and because I'm also saving last message ID this is an incremental number so it's easy to count how many messages were lost ok and this result depends on your cluster configuration in my example I have used something called auto downlink time it means that after specified seconds minutes Note will be dropped but you can say ok I don't want to do this automatically so let me summarize failure detection ok it works it needs some delay you can lose some of messages and if you want to tune it you can make it more sensitive but you can go some get some false positives when I was running similar example after training on localhost or running this application on localhost i can t on very aggressively the setting so it works very quickly but if i move to network environment i get a lot of most false positive and application was not disabled so you have to remember about this trade-off ok so fell over scenario how it looks cluster nodes are start to losing heartbeat gossiping each other about have you seen this node and if you miss some of messages you can configure it how many and how sensitive is my name is nose becomes an which table after specified amount of time note is removed from the cluster you can also do this manually and when sharpening can recreate actors on different notes and for singlet on the same so what we have learned so reading documentation is not enough you should play with your applications on production is the best because this is the same environment you will not recreate this on localhost okay thank you guys if you have any question I will be here [Applause] [Music]