Effective Actors

funconf 2013, Jamie Allen: Effective Actors

Recording: funconf 2013, Jamie Allen: Effective Actors

so this is going to be effective actors all sorts of best practices I've learned from using actors in development with Scala since 2009 I have been programming with the originally scholar actors because akka was pretty immature at the time and switched to akka probably around 2011. anybody who's worked with scholar actors knows why but um who am I I'm Jamie Allen I'm the director of Consulting for typesafe I've been with typesafe since uh 2000 and well last year and uh I'm the author of effective echo which will be coming out from O'Reilly media in I think the end of this month so hopefully pretty soon so effective actors are best practices based on my several years of actor base and really asynchronous programming development how many people here have done actor-based development whether it's with AKA or Airline okay great a really large number so you guys probably know a lot of these things as well for those who haven't just a really quick introduction to actors they're just concurrent lightweight processes that talk to each other by passing messages you should never ever be calling a method on an actor that's your number one rule if you find yourself doing that stop don't ever call a method directly on an actor does anybody know why you're introducing concurrency the calling thread is calling into an actor's method at the same time the actor is getting a thread from some thread pool that's going to allow it to iterate through its cue so you don't want multiple threads inside of your actor you have now defeated the purpose of using actors right and you can organize them in hierarchies so you can have fault tolerance and that's really nice isolation of State no internal concurrency really simple example here is just setting up a pinger and a ponger so in this case at the top I have a pinger actor it's a class it extends the actor trait and all I'm doing is defining a received block for the messages I get if I get anything I print out that I'm pinging and I send a response to whoever I got it from next I have my ponger and it's going to say well I've got my own receive block here and whenever I get anything I'm going to send a pong after I print it out and at the bottom is how to get it started I've got a bootstrapping object here I create my actor system which is the context in which all my actors will live here and that I create the instances of the Pinger and the ponger and then I send a message to my Pinger and I say well you know ping I could send it anything it doesn't really matter but I have to say who I'm sending it from so it knows to send the message back to Palmer I'm sort of overriding who sent this message here and I'm doing something you never want to do I'm just doing a thread.sleep because otherwise my actor system would shut down pretty much right away based upon messages being handled inside mailboxes but you know you might not see any interaction it's always synchronous here so I'm just going to let this thread sleep for a second just so I can see the ping pong ping pong ping pong and you know stop supervisor hierarchies based upon Airlines concepts of OTP all we're doing here is defining our way we're going to handle failure inside of our actors right we can Define supervisors that allow us to say what to do when any one of these fail whether we want to start all of them or restart just one depending on the kind of failure that's occurred any non-leaf no and leaf node here can't be a supervisor any non-leaf node by definition is a supervisor if it has actors under it it is a supervisor supervisors have a very simple way of defining the behavior we want to do here we just say how we want restarts to occur we can only do this once we say in this case that for any one failure we get this is the behavior we want applied to that one actor where the failure occurred not all of them under me in this case I've just got arithmetic exception which is resume keep doing what you're doing I've got no pointer exception in which case I'm restarting which does not clear out the mailbox right it does get rid of your internal State inside of the actor but does not clear out the mailbox it will continue to handle whatever messages were there for that actor before note that whenever I create an actor under myself I don't say system actor of I say context actor of that's how I get the supervision local to my actor as opposed to the system now there are two kinds of actor systems I see built one is the concept of sort of building a live cache where I'm going to use actors to hold State and represent real data live inside the jvm and this could be maybe all the customers that you have that's not going to work for Twitter for example they have way too many but a lot of organizations don't and they can create an actor system that has all their customers and all the accounts each one of those customers has live inside of memory and they can interact with it worker supervision is different in the case of the previous example you had State encapsulated inside of your actors worker actors should not really have state state is passed to them in messages and they apply some Behavior to that state that they receive and then send some response on to somewhere else to allow the processing to be handled now parallelism it's really easy to scale up our actor systems maybe a little too easy because I know speaking for myself sometimes I think I know where I'm going to have a really hot piece of code and I know I want to scale it across multiple machines or just multiple actor instances and so I say I'm going to do that up front I don't recommend doing that in this case we can create a bunch of actors underneath ourselves very easily to handle work they can be all the exact same type of actor handling the exact same type of messages you distribute the work to them and actors have another feature that's really nice they're location transparent that actor e could be on one box actor F on another box actor G on another and you're Distributing work across your nodes very simply so routing is you know a very simple concept here I have my class at the top which is an actor which is going to handle some message it receives in this case we're just printing it out but my parallelizer allows me to say well I want to create an actor of this type but I'm going to use a router and I'm going to create five instances and distribute the work with round robin semantics one two three four five one two three four five very simple so now that we've defined some simple actor semantics and terms let's talk about the rules of active Development actors should only do one thing it's a really bad idea for us to confuse or conflate the responsibilities of our actors we want them to be very Atomic in nature handling just one specific type of work or one specific type of data that way we don't end up with actors doing too much handling too many different kinds of messages and having a harder time trying to figure out what happens when things go wrong supervision becomes more difficult the more things you have an actor doing because it's got to handle different types of Errors right we just want to follow Uncle Bob's simple rule of single responsibility here make your actors simple they're lightweight and they're very small they only take up about 400 bytes before you start putting stuff into them so use as many as you want you know that said keep in mind your HEAP size and keep in mind the effects from garbage collection that you may get from having an extremely large Heat supervision every non-leaf node like I was saying is technically a supervisor we want to create explicit supervision we don't want to have an actor handling multiple different types of actors underneath it and therefore not being able to Define specifically how failure should be handled for each of those different types so what do I mean by that here's an example of conflated supervision I've got two customers under my root node customer One customer two and under customer two I've got two accounts and three devices okay well if I have failure under accounts or failure under devices I've only got one supervisor to be able to handle them and they may need to be handled differently so it might be better and actually it is for us to go with explicit supervision where we say that we have an account supervisor and we have a devices supervisor so that we can handle the way failure occurs inside of our accounts you know differently from how we handle them with devices I showed you earlier an example of being able to do one for one restarting or resuming of an actor well you could also do all for one and if you define a conflated supervisor my C2 here to be one for one what if I need all for one for my devices I can't switch that we want to keep our error kernel simple the error kernel is the top level of our actor systems the very root nodes that we have and we don't want them to have a whole bunch of work inside of them we want to distribute work downward through our supervisor hierarchies and Define work in very Atomic chunks so this will help with your fault tolerance a lot if you're building up a hierarchy of failure how to handle failure upward then you can be very explicit about the way failure is handled and what failure needs to be elevated so we also want to use failure zones anybody ever heard of the bulkhead pattern yeah when we build a ship we have bulkheads so we know that if you know any part of the ship is compromised the whole ship doesn't flood only that one bulkheaded area does right and then we seal off that one bulkhead and the ship doesn't go down same thing goes with actors we want to bulkhead them off so that they don't get affected by anything happening inside of other actors that they're not related to so here's an example of you know the the exact same thing with my customers and my accounts and my devices and my applications underneath them I may want to separate them out to have their own thread pools and resources so that if threads are being utilized over here in a very you know uh blocking fashion or they're just under a lot more load the thread pool over here is not being starved these actors have their own resources through which they can get a thread now this isn't entirely free you have to think about what's happening on the machine in which it's running you can define a million actors with them you know huge amounts of threads in your thread pools if you're running on two cores is that really going to help no because you can't schedule the threads right so you have to think about the hardware on what you're running the resources at the physical level as well as what you can control here how many you should be applying for every single one of your dispatchers here your thread pools in use so always keep in mind the hardware and the amount of threads you have available experimentation but generally I started doubling I double my number of threads at you know if I have 80 cores on a machine I just start working with 160 and I see how my application runs every application is different what your actors are doing is going to affect this how long they're holding onto their threads doing their work whether it's CPU intensive or they're doing blocking operations and I'll show you a couple tricks that you can use to handle the fact that threads who are doing a lot of work or not really doing a lot of work but they're doing blocking operations how we can make sure the resources are still available because a blocking operation at the core is still going to get pulled off core right if it's waiting for some sort of i o like hitting a database or something like that that thread is not going to sit there and spin on the core it's going to be pulled off in another thread it's going to be scheduled for Execution so you know how do we take advantage of that do we want our thread pool to you know not have that thread available what can we do to make another thread available I'll show you a trick for that in a minute so actors are cheap we want to use them you know if you're loading a lot of data inside of them then yes they are going to become heavy pretty quickly but generally speaking actors are lightweight take advantage of that create lots of them optimize later yes you want to write code to be as performant as possible while you're going and be iterative about how you are applying your performance analysis performance tests but don't go crazy trying to be you know performance-centric with your actor usage first make sure that you could build something that's working the way you need as well yeah both so what we're doing here is we're saying that threads are shared across a pool of actors if actors are using a dispatcher in the Echo world then they have a pool of threads from which they're going to reference there is a pin dispatcher which will apply a single actor you know single thread per actor now that said pin dispatcher unless you're doing busy spinning yourself doesn't mean that the core is pinned right we don't have a lot of control on that inside the jvm so if you want to have your pinned actor pinned to a core you've still got to do busy spinning right when you're not busy you've got to keep it on that core doing something so that it doesn't get pulled off by the kernel um but that's how we make actors Lightweight by having shared resources relative to having a thread for everyone I mean every thread's got to have its own stack frames you know its own stack its own stack you know for execution whereas if we have a pool we can share those resources across all of them but don't use just one one of the things I typically see people doing whenever they're first starting out with ACA is they start using the default dispatcher that comes with the actor system that you use to start creating your actors and then inevitably they start creating more actors inside those actors they may be doing asynchronous work with Futures who knows but they start running out of resources and they start seeing timeouts timeouts is your first clue that you're running out of threads so you don't want everything pulled off the default dispatcher the default dispatcher will start with eight threads and it will scale up to 64. but that may not be enough for your thousands of actors right you want to Silo them under their own dispatchers so they have resources specific to this set of actors and resources specific to this set of actors block only when and where you must one of the things that we tend to do when we're writing our code is you know especially when we're first defining how it's going to work especially in this agile world where we're not sitting around and maybe doing as much design as is necessary to Think Through the problem it's very easy to get yourself in a blocking situation and it's also very easy for you to say that I need guarantees of stuff right don't don't block unless you absolutely have to there will be times in your application that you do limit them to specific places this when you do blocking think about what's happening again at the hardware level you are going to lose those warmed caches right that allow you to take advantage of high performance at the core level where the data is available at the time of execution blocking pulled off core another one gets scheduled it's a contact switch not good not good for performance now what about Futures and actors are they a good idea how many people write Futures inside of their actors yeah I've since come to realize that I don't like this Paradigm I'd rather not that's not to say that they're not the right solution for every case I mean they they might be in some but I'm starting to find that I don't want to use Futures as much as possible they're more heavyweight than being fire and forget we don't want to ask for a response we want to fire something off and we want to create a Handler for when the response does come back it's better to use our tell semantics than ask so I say I want to use a transient child or actor to handle the response what does that mean I have a very simple example here where I have a worker that is going to receive any sequence of numbers say 1 to 100 or something like that and when it gets it it's going to reduce it down to the sum of all the numbers I passed in so if I send in 1 to 20 I think I get 210 well I've got a delegator here and first of all I Define that I've got a worker reference a way to talk to this actor who's going to do the work for me right whenever I receive a message I'm going to create another actor here and by the way this is not the most recent version of my slides there's a bug here I'm not setting this equal to a vowel a vowel of the actor so that I can reference it later and I say well okay give me a props of some new actor this is like an actor literal right it's not bound to a name or anything like that like a string literal my name in quotes this is an actor literal and I'm saying that whenever I get a message all right well you know I'm going to um eventually handle the response from this actor with this one and it's transient it's only going to live as long as the work is being done and the response is sent back what you also don't see here is I'm supposed to be sending the message to that actor to go do the work right and I have to tell it to do it with this actor I apologize for that the slides are better and they are on my GitHub if you want to look at them the most important thing here though is if you're going to use a transient actor remember to shut it down once you get the response you needed you need to make sure this actor goes away otherwise you have an inherent memory leak for every single message you're going to be creating a new actor that's going to add up over time so blocking when do we do blocking an example is database access if we're doing anything with jdbc we have our jdbc synchronous drivers that are going to be blocking anytime you call to postgres or something like that right we want to minimize the impact of blocking we want to make sure that the actors are going to do that blocking are running on their own thread pool so that they are not affecting other actors the other actors if they're waiting to get a thread because all these other ones over here are blocking well they're gonna be starved over here they're not going to have the resources they need to handle their messages we have a concept inside of Scala it's not really akka anymore called manage blocking and manage blocking allows me to do something kind of funny watch this I say I have an area of my code that is going to be blocking now first of all I said well I want to create my work or actor here using a dispatcher right and I'm going to have specific thread pool information in here so that I'm applying this by configuration and again this has the bug that is fixed inside my GitHub anyway I am defining that I have blocking Behavior here to ask for this work and I'm using the question mark this is a case where I'm going to use a future but then I say a weight result this is where the blocking is occurring but I've wrapped it inside of a blocking block when this happens your thread pool is now allowed to create one more thread so that other actors using this thread pool are not starved right increase my thread pull size by one now that you know it's an asynchronous operation it's going to take a little bit of time measurable time for that new thread to be added to the thread pool but you will have actors not affected by this one blocking uh FYI one thing to keep in mind with manage blocking it's not a free lunch there's no way to cap how many threads can be added to your thread pool so keep an eye on that there may be times you don't want manage blocking okay oh and note whenever I use that with dispatcher there that's me saying I want to use a failure Zone a bulkhead for my actors we want to go with push and not pull semantics we want our interactions to flow in a single Direction this allows us to let go of the idea that we need guaranteed messaging or something along along those lines don't try to think in terms of guaranteed anything inside of your system instead truly resilient systems will never assume that anything successfully got through until they know the world is in the state they need so what do you know systems like rabbitmq do they act you're going to have to put some ax inside your system just to let this actor know that something was accomplished but you know you may have to repeatedly fire the message don't fire it just once and hope that it gets there so find ways to ensure our actors remain asynchronous as much as possible this way we can leverage them you know the machine's resources on which they're running and we want to avoid making our actors wait for anything while handling a message you know doing that await there are going to be times where you do have to do that but don't do it unless you have to do not optimize prematurely I got in a little bit of trouble when a mailing list recently for saying this because they're right yes we should be thinking about the right algorithm in order to write our code and we should be thinking about how a data structure might be more appropriate for what we're doing and you know we might even be trying to do our performance testing from the outset so that we know when our algorithms and our data structures and the code we're writing is going outside of our non-functional requirements that said don't perform additional optimizations such as trying to distribute work until you know you need it you've measured you need it at that point in time find ways to you know distribute work more efficiently but not before because I always get this wrong and maybe it's a failure on my part that I don't get it and I don't see exactly the right place but I just can't predict it very well and sometimes I'm really surprised where I find my code is really hot our initial focus when we're writing our code should be thinking in deterministic terms we do want to think about our algorithms and our problems such that we know that this is going to happen in this order because it's the only way we can reason about them but then focus on also being declarative focus on writing your code not so much about the how things get done but what it is that needs to be done Scala is pretty good at this actors are a little less so because the very concept of an actor means you've introduced a bit of how but Futures are generally speaking fairly declarative you don't know how the thread pool is you know being applied to that future unless you've looked above to see how you've defined the implicit you know execution context for them right they are generally speaking pretty declarative stay immutable as long as possible I don't think I have to tell anybody in the scholar world that very much I mean everybody tries to right start with functional programming concepts of immutability referential transparency first class functions and work from there right there are going to be times when we're optimizing that we have to drop out of this some things do not lend themselves particularly well to functional programming some things do generally speaking allocation is pretty cheap on the jvm and if we size our new gen reasonably well we might not pay a price for that but there are some applications of algorithms where you may want to drop down to an imperative Style just to be faster nothing's gonna be faster on the jvm than a while loop right so keep that in mind advice from my boss Jonas Bonaire he's the person who created akka and he thinks that when we're building an actor system we layer in our complexity as we go think just in simple terms at first and then start putting in the complex parts of your system as you go this is the complexity of mutability in particular we start with using compare and swap semantics non-blocking non-locking data structures there are some libraries out there that are pretty good boundary has one I don't know if anybody's ever heard of boundary.com Cliff Moon and his crew have a non-locking data structure Library that's not bad click click has one we're checking out STM don't go there you might consider using it if you want to do something that's not highly contended and you want Atomic transactions inside of your data you know the classic case of mutable State first name and last name I want those two values in memory to be committed together right that way I don't end up with Jamie Allen being changed to John Doe and somebody coming in at some point and getting John Allen right STM can do that but it's really bad under load you'll get a lot of you know inabilities to commit and I don't know that anybody's really proven STM to truly work I could be wrong about that but I've yet to see it at the absolute worst case scenario add your explicit locking and threads is you know last resort you don't want to go in the into explicit locks they don't compose well very difficult to have multiples of them so but sometimes you may have to prepare for race conditions things are going to happen out of order in the asynchronous world it's just life that means that you have to be able to resolve that in your system you have to be able to withstand the fact that things could come in out of order and you need to be able to recover from it code that way write your actors to say all right well I'm only going to handle these kind of messages for now but what if I'm not getting any of those messages and I think I should be maybe I should have some sort of system check to look around and say what's happening over here in these other actors so that I know that I've got to recover I've got to move back to another state because of the rest of the world thinks things are different than what I do actors can become different received blocks they can change the way they handle messages to handle a different set of them right it's a great way to do sort of State machines but that means you also have to be able to handle the fact that what if your state is inappropriate to everybody else the Thundering Herd will occur when you build large actor systems be careful with it you're going to have what I call event storms where you send a message in and suddenly all the actors start doing all kinds of work and trying to figure out the flow of what's going around can be very difficult well we can sort of mitigate that by not passing in generic messages if you find yourself sending a message that says something like accounts updated that's bad you want to be sending in very specific messages and lots of them representing everything individually that changed that way you can handle it better without having all of these actors you know respond to this big generic message we can use dispatcher tuning with external configuration using backup policies and queue sizes but we also have this thing called circuit breakers and I don't think I have an example inside this presentation I should add one um but it allows you to start giving a bit of back pressure by saying I'm not going to accept any messages anymore I'm overloaded this actor has handled enough so now I can say for a time period I'm not going to handle messages and then I'll start handling some more this will also become very relevant if you start doing clustering with ACA and we have a a really good activator template out there for distributed workers that allow you to figure out how to deal with work overloading your actors by the way has everybody seen activator a few people it's a really cool tool and it's not just for people who are new to Scala templates are out there they're doing very significantly difficult work and we'll show you how to use clustering with distributed workers in ACA so our takeaway start by thinking in terms of an implementation is deterministic and not active base just so we understand what needs to happen now think about how you can make that asynchronous leverage your resources and then finally start thinking about if you need mutable state where and how you're going to handle it be explicit in your intent so this is something that's interesting about akka currently we cannot create an actor from another actor without closing over this which is like one of the worst things you can possibly do technically because when you reference this from an actor what are you talking about you're talking about the specific class you are not talking about the actor rep abstraction that is so wonderful about akka Scala actors didn't have this actor ref allows us to say that we don't know where the actor lives which box it's on or you know which data center it's in we can just send a message to it and declaratively you know not handling all this how to get that message to a specific endpoint you know it just gets handled for us but this is the the concept of this object oriented this open recursion this is being closed over when you do a props of a new actor inside of akka and we know this is a problem there's a new sip coming out called sip 21 about pickling and spores pickling being for serialization spores is a way to define what gets closed over you have to be specific about it before you do your work this would be really powerful for apis like how we create actors in akka so what do we do about this for every actor that we're going to create we create a companion object and there's been some debate about whether we should use an apply method you could I don't like using the apply because semantically when you use apply you're expecting a response of a new instance of my worker but I'm returning an instance of a props here by creating my properties inside of this Factory method in my companion object there's no way to close over this and now when I create an instance instead of saying props worker here I call the companion objects props method and safer this cannot be closed over does everybody get that because this is something that's fairly new and if you're writing your actors you know it probably hasn't affected you much but this is a safety mechanism until spores go in which is you know still an open discussion for scholar 211 scholar 212. so another thing that I want to mention here is remember how I was talking about my Anonymous actor that I was going to use to handle work I can't do a companion object of an anonymous actor type so this is a lot like a Lambda anybody who was in the last session I gave knows my opinion of lambdas I don't like this pattern generally speaking where I create an actor and it exists only here I don't mind the fact that it exists only here I mind the fact that I can't test this actor in isolation I mind the fact that I can't re-reference it somehow and use it in another way because it only exists right here I want to have this abstracted out as a separate class so that's actually my next slide here Anonymous actors they're literals right they're tough to debug because they've got to come up with names for the instance of the actor it's gonna be like dollar sign a dollar sign B you know their name mangled that's where the compiler Engineers call it they're not testable in isolation and their intent isn't as clear an actor that is a type that is defined outside of that scope has a name which can describe what it's doing whereas this actor right here does not have a name and you have to read it to figure out what it does so define specific actor types you know don't do this Anonymous stuff create transient actors of a specific type you've defined you'll get more information about your message flow as well anybody here played with the typesafe console historically we've been really terrible about getting this out to developers at the end I'll show you a link to get it now it's much simpler to install than what it was before because I mean it was so impossible we told people it was free and then we didn't make it available very well we're trying to get better about that we want to name our actors when we create an instance of an actor don't do what I've been doing for the sake of brevity inside of my slides always give your actors a name that's important it allows you to apply an external configuration specific to them right the lookup isn't quite so relevant now with aca22 where we're doing this uh we're not doing actor four so much as actor selection so but still having a name will help you from the perspective of output of debug and from the perspective of being able to pull on external configuring create specialized messages as I said this is good for avoiding event storms create specialized exceptions lots of them don't rely on these broad exceptions out there to handle stuff create new ones and throw those that represent exactly the type of failure and in what part of your domain it occurred this way you have a lot more explicit information about failure and you can react to it quicker when it goes wrong inside of production if you have generic messages it's going to be very difficult to figure out what the flow was that caused this with very granular exceptions you know it's only in this part of code that it could possibly have happened so our takeaway be specific in everything you do don't be generic that's what's going to lead to the event storms that's what's going to make it harder for you to debug in production do not expose your actors I alluded to this earlier with the concept of this never ever talk about actors in terms of this no direct references never if you see in your code that you do not have an actor ref but somehow you got reference to the type underlying the type of the actor specifically you've got a problem you want an actor ref and you should always be talking about your actors in terms of actor refs instead of this you should be talking about self in akka right we want to be able to reference ourselves and send messages to ourselves that way we don't do work inside that is holding onto a thread especially if you do any kind of looping has anybody ever written a loop inside of an actor good that's awesome it's a bad way to do things you've got your thread saying they're spinning doing this looping work right much better to send yourself a message to Loop because then if anything else needs to be coming in to tell your actor that the world has changed that can be handled right never ever published this so the example I use for this is how many people have used jmx yeah how do we register an mbeam we give it an object name and we pass this and so initially whenever I was sitting down and writing my Scala actors I was registering all of my actors through jmx and it seemed like a really great idea except I was sending this which meant that whenever the m-beam server was going to be doing the operation to get the data was calling in with another thread for read operations you know it's probably not the worst thing in the world but if you put an operation inside of your jmxm being interface now you've got concurrency don't do it The Observer pattern is a pretty well known established design pattern we've all known for years but it's bad in the context of actors if we have to reference it from the perspective of this we want everything to be an actor ref use immutable messages we should be immutable in general anyway but sometimes in actors because we know that we don't have concurrency we have you know vars or mutable collections or mutable state if you're going to send anything like that out make sure that you stabilize it first into a copy and send that right that way you don't have to worry about anybody changing the data that was inside of your actor you don't want mutable State escaping past copies airline does this implicitly with copy on right semantics in the jvm we don't really have to worry about copy on right quite so much because of the distinction between our value well our identity and our value on the Heap right our identity is our variable name our value on the Heap is this what is inside the specific memory location inside of our Heap and it's abstracted over through the reference to that value right we can pass information about where that data is right there but we don't want to have anybody change it for us and if they could copy that value into another space in the memory and pass that that way they get a snapshot of the data akka has STM references again you can use them but don't do it in a highly concurrent environment the reason it works for ACA okay is you don't have to do inside of an actor because you don't have multiple threads but you could apply it in some other way outside of an actor we had agents for a while anybody try that one out they're going away they were a bad idea what's that I might have you know he's on his honeymoon so he could be doing all kinds of crazy stuff yeah we don't really like the idea of passing in behavior and STM itself doesn't really work well with highly concurrent systems and agents in STM are sort of bound together so we think we should just kind of get rid of that before we all shoot ourselves in the foot so by the idea of passing Behavior with an agent you could sit there and say that you know an actor encapsulates Its Behavior right you have Behavior whenever you have a receive whenever you receive a message you do specific things you can pass in a received block that it could switch to we don't think that's a particularly good idea it's much better to have that wrapped inside of the actor how it knows it should behave in any circumstance but with agents I mean it's sort of implicitly how that works behavior is passed in to change the state and we think that doesn't work well under load with STM we're we're sort of anti that avoid sending Behavior huh I just explained all this I guess it makes it very easy for state to escape when you do this by the way because you're closing over stuff to send right you have to be explicit to make sure that that data is not you know somehow mutable through reference or through value or through identity by identity I mean it's a bar by reference by value I mean you could change what's in memory there at that specific location so keep everything about an actor internal to that actor how am I doing for time okay good uh and be very wary of any data passed in through closures if you see braces inside of your actors and you know inside the receiver or something like that you see yourself writing braces for something you want to express think about the data that's going to go inside those braces typically those braces are being used for some sort of closure be very careful with them you don't want State escaping make debugging easy on yourself I have to update a part of this slide because I think I or not this slide but coming up I do mention jmx you can use jmx just know how you're using it don't ever expose the operations we want to possibly externalize business Logic the reason I say this is because it used to be extremely hard to test actors it used to be prior to the ACA test kit that the only way to test them was to send the messages you would have to create an actor system you'd have to create an instance to the actor you'd have to send in a message and then it would have to do the behavior and you have to test to see if you got the value back generally speaking what I wanted to do was just test the business logic of what I was trying to accomplish whenever that message was being handled so I would pull that out in libraries of functions libraries and methods and use and just test those by themselves because I don't care about all the actor semantics whenever I'm just trying to prove that if I go over 50 000 frequent flyer miles that I'm now gold status right with test kit you don't have to worry about this as much because in a test context only you can get a reference to the underlying actor and test the methods on it without having to pass messages in but I think this is actually kind of better anyway I like the idea of pulling stuff out into libraries I got so used to it makes my life easier the only thing about it is then you're always referencing something from a live as opposed to being you know inside encapsulated inside the actor but Keep It In Mind as a rule of thumb if you find the actor becoming too complex use semantically useful logging you know it's one thing to sit there and say I've got a collection inside my actor and I'm just going to print it out whenever you know I'm in log debug mode and I you know want to print out information that's very uh inside of that collection but I haven't formatted it and anybody who knows that they're trying to look through a large collection of data that's just sitting there you know wrapping line after line after line trying to find that one piece of data inside it to see what was wrong it's hard format your output and do it at the debug level only obviously you don't want to do this for info level or uh you know a level that's going to be for regular running only for when you know there's a problem we want to use LINE breaks and indentation to make that data very clear so you don't spend a lot of time trying to parse output we want to use unique IDs for messages this doesn't have to be uuids universally unique maybe at Twitter it does actually because you know they have so much happening inside of their system on a daily basis an hourly basis like a second most actor systems generally don't you can create much cheaper unique IDs for which you have a fairly reasonable certainty that for six hours there will be no collisions no messages with the same ID because you know the failure occurred at 4 am on Monday you know that if you look at the logs for that time frame and you see the failure and you can trace through it by a message ID you know that's good enough matter of fact you just grep the output and only see the output for that message ID and you don't have to look through everything else monitor everything find the tools out there that will help you monitor your actor system we're trying to get a lot better at this but typesafe is not a monitoring company we are not trying to be New Relic app Dynamics or you know any company that's doing low-level you know monitoring for your system what we want to do is build you know tools that allow you to write great software the typesafe developer console free the developers during developer mode and very helpful it gives you a visualization of all the actors in your actors systems in their hierarchy so you can see whether or not they exist which is always a good thing to know right what they're doing if they have errors inside of them if they're mailboxes are growing this is one of your biggest things to keep an eye out for mailboxes that are continually getting larger and larger as opposed to being able to withstand bursts and run down through the messages that they're handling if they can't slowly handle all those messages downward you're going to have system failure as they start handling you know more and more and they're just you know their cues are getting larger and larger uh visual representations are great typesafe console is really good for that yes um so here is what it looks like I can show a real quick demo if people want but I can't because my machine's not working um but what you will see is every dispatcher you define inside of every actor system and the dispatcher is your thread pool abstraction it'll tell you the number of threads the max latency for handling a message which is the time it from when it was sent to when it was handled by the actor who received it right it will also give you an event log of all of the message flows through the actor in the last 20 minutes or so depending on the window that you're defining um and it'll tell you when your mailboxes are getting larger and larger here's how you get it typesafe.com if you go to typesafe.com just look at the console through you know the menus you'll find it but this is the URL and there's just a downloadable fat zip that'll allow you to get started yeah so build your active system to be maintainable from the outset give yourself as much information when things go wrong as possible uh that's it I want to thank Jonas and Victor and Roland and Havoc Pennington for helping me put together this presentation the questions yes some companies have said well we tried active and we didn't like it yeah I think that depends on the the requirements of the system if you don't need fault tolerance you can probably go asynchronous just by using things like Futures right that said Futures don't allow you to batch operations so one of the really nice things you can do with your dispatchers for your actors is say how many messages you want to handle in sequence before you release that thread from the actor so that it's not entirely fair you don't want your actors necessarily to be purely Fair where they handle one message give the thread back to allow another actor to do its work and then wait to get another one to handle another message you want to handle by default right now it's five you could set that up to being a hundred if it's an actor that has a lot of messages coming in that way it can you know handle a batch take advantage of the fact that it's got a warm core you know with all the warm caches inside of there and then release and be somewhat Fair if you don't need that Futures are a perfectly fine example that and fault tolerance yeah so the other extreme is to visually suppose I have a actor you know basically acting as actor's service to you know kind of different methods come in so should I just break a busy into 10 20 different actors and each one only handle one messages so where where are this balance like the two of the streams well I think generally you can Define messages as group types of messages and flow them forward them down to the appropriate Handler for the grouping of messages you have if you have I mean you're probably going to flow through an actor who's going to act as sort of a traffic cop right and he's going to say well I know who to dispatch this kind of message to right um generally I haven't seen a case where I've had messages that I couldn't say okay this should go over here an actor can do these groupings right I haven't seen messages that couldn't be separated like that can you give an example you know I'm just wondering the design what should be like if I have a this my think about actor is a Posada which is going to providing the service with all kind of services in a different methods different messages right of course all the message you're gonna handle by this actors I mean this you know and then sort of you know it sounds like you're saying this this actor actually just a Gateway yeah and this is actually you know part of what the cluster design will do if you look at distributed clusters and in in Access 2-2 um you know you're going to be receiving messages into sort of like an entry point to the to that node of the um of the cluster and then it can delegate work to a child responsible to grouping of messages right but uh yeah I mean you just have to look at the Domain and see it could be by failure types of failure that could occur it could be why whether they're blocking you might want to have database interactions over here and as one grouping of actors and you pass all the database messages there um and it just varies um yeah some results from those so what's better to use for this kind of result collection should it be after or should it be mutable can correct connection um I like actor so what I typically do is I will dispatch to all the places I need to get my information from right and then I'm also going to schedule a task to compete against them for a timeout so I say all right well I'm going to send messages off to these three services to get information the example I typically use is I've got to get my checking account information my savings account information and my money market account information for this customer I Define the transient actor that is going to be handling the responses right and then I also schedule a competing task for a timeout to occur so that I know that if all three of those don't come back as a completed data set then I've got to be able to time out and tell whoever requested that I wasn't able to get your data for you in the acceptable amount of time um I can't really do that with a collection so much collections are difficult because you would have to look at the collection and say do you have the money market account information do you have the savings account information whereas that transient actor I could say I have an option of my checking information my savings information my money market and if I have a value for each one of those I'm pattern matching on each message I receive back to say all right do I have all three at this point okay then I can respond or if I got a failure I notice in that response and I'm using my actors Q it's mailbox as you know a promise of sorts because whichever one gets in there first the timeout or all three messages is the one that's going to win okay yeah I can talk to you offline yeah you had a question yeah in your domain to act up directly but put something in between but in some cases where the external domain really seems to be very natural yeah can we talk about it afterwards I'll unhook and let somebody else get started and get married thank you thank you