SF Scala Evan Chan 2 27 14

cool so um my name is evan i'm from uyala uh i'm sorry i need to give this 30 second pitch we are also hiring our most of our back-end big data stuff is in scala we also have an an old rail stack but we definitely um you know we have multiple teams using scala so uh also if you're interested in doing a really cool uh both real time and a lot of big data systems working with spark cassandra and other really interesting technologies um feel free to see me so um akka there is a really big term now um called reactive programming you might have seen that coursera has a course called reactive programming i actually tried to take the first few weeks of that but got too busy what what is reactive programming so um and and that's a lot of what underlies the aca even reactive programming has to do with writing event-driven scalable and fault-tolerant applications and reactive in a sense that the real world has a lot of events going on and you want to react to them in a sensible way and a lot of the components you might have heard about in the scala stack especially the types of stack solves these problems so we have akka we have play which is based on archive play is probably the most popular framework who here has not heard of aka no one and just out of curiosity how many people are using using akka at their companies awesome how many people are using play yeah so if you use play you're probably also very familiar with aka because play is completely based on aka underneath uh so we're going to talk about reactive programming with akka and specifically about how we use akka and scala at uyala and some of our stories with productionizing such systems and how we can uh really make uh productionize uh how we can productionize them effectively and we will i'll talk about a technique called stackable traits that we use with their actors to to instrument them just a brief overview of of oyala we are i'm sorry about this projector we are about five six years old and uh we basically are like you can think of us as like a youtube for business so um a lot of our uh biggest customers are the likes of espn pax wealth networks victoria's secret rei people that like to serve a lot of videos on on their site and everyone who watches videos on their sites like ends up with events through us so we process a lot of data over two billion pings a day um how do we get started uh using scala well we were actually mostly a ruby company even mapreduce jobs written ruby and uh what what we discovered was that uh you know that was before i i went there right and um now what we discovered was that ruby was a good language for a lot of things but it is not a really good language for doing big data stuff it is it is slow and it doesn't integrate natively with hadoop we're not able to take advantage of a lot of input formats and things like that so we started experimenting with scala for uh doing real-time analytics for which uh we're pretty sure that ruby wouldn't work uh as well as for mapreduce pipelines and what we what we realized was that we could get about a one to two order uh magnitude in performance boost just from switching to scala so we were pretty happy with with that uh made made a lot of sense for us so um now we write hundred percent of we have actually like a banned uh ruby map reduced jobs um or at least we've been writing new ones unfortunately we still have some old ones running um and we use scala and akka with storm with spark with mr with cassandra basically all of our big data stuff is written in scala so i want to talk about the problem of ingesting two billion events a day a little bit you have a video player this feeds into a front end of say nginx servers we feed them say in the kafka and then we feed them into other pipelines what what what is an effective way of processing all this data right and uh and we'll we'll start with a couple places where we use um aka the first the first place is it's a place i'm sorry that reads a live log see we have a daemon called live locate and what this does is it is designed to get um new um log files which have been written by nginx and forward them on to kafka and this was kind of like one of our first forays is a production system using akka it's very simple it's simply tailing log files it picks up basically what we do is we have one actor reading each file and actors um you know the good thing you don't really worry about threads or how like um logs or anything like that this is pure message passing so one actor is reading log lines from the file it process them a little bit then it passes them on to another actor which uh you know forwards them to kafka so it's it's pretty good use case for for akka because it's completely event driven as you're reading files you get new files that's another event or where rather when you detect a new file we create a new actor actors are very cheap to spin up so you can so we can handle many thousands of log lines and allow us to do some coordination on top of that so that was our first uh foray that works pretty well we still have it in production um the second system where we started to introduce akka was is funny enough in storm sure but what what does kafka do oh sorry um so kafka is feeding uh the logs on to other processing systems that uh where we do a lot of our analytics for what i guess to back up uh what kind of analytics we need is that someone like espn wants to know uh where are people watching video you know what kind of devices are they watching they want to like slice and dice their data so after stuff goes to kafka then um then that's where we do all this processing with like spark and storm and other kinds of frameworks can you just describe the companies so many supports okay thanks thanks um so for those of you that don't know apache kafka is a message queue and it's kind of like um uh rapid mq it's it's used for very high volume um message handling do you are you using any kind of a front and web server are you just and nginx is just taking this traffic and then flushing it to a blog you're saying nginx is taking the traffic and flushing to a lock yeah so nginx is our is our front end yeah everything basically yeah everything is encapsulated in uh http rests kind of um yeah in requests yep so um after kafka the first uh we started getting into real-time analytics so we started building a um this is really cool you can get uh the number of people actively watching videos in just a matter of seconds and we did this at that time storm had just been released so we started writing this in storm i'm actually the author of the scarlet dsl for storm um so storm is neat uh how many people have not heard of storm by the way um okay so for those of you that are not uh that don't know storm is a real-time uh distributed processing system it's used for stream processing you're getting a lot of events in and you write um they call them spouts in both you can picture like a water flow system where like spouts are the source of the data and i'm not sure where both comes from but basically that's the kind of like the processing for the data and the way that it works is that you define parallelism so both are the processing unit and you can have multiple of them they are distributed across machines and do some work um now um strictly speaking storm actually doesn't have anything to do with akka you could and most people will not use storm like actually with akka um so uh forgive me this is uh this is slightly like advanced topic but um we use so you can think of like boats will send messages to each other it's kind of like actors but it's more of a like a javascript api we actually but what happens is that these run on one machine so when something fails and and one machine goes down the whole boat one boat might go down and then come back up for us though we we wanted we wanted more granularity than just one machine we want to have some current currency within within a boat so we used actors within storm for that and the reason why we do that is because you don't really need them however um the good property is that there are some things that you want to process together where like if it fails you want to bring them up together and it might be more complicated to treat a system where you have a and b and one fails but the other doesn't fail and so when you come up you have to take care of a lot more cases for us it was a case that these two guys did work that happened together so it was easier for them to fail to get to fail together and to come up together um i can explain a bit more but basically what we learned from storm we found that storm was uh too complex to use in production the reason is because um inevitably what happens is that you deploy a system into production then you have people on call at 2 am the phone rings or you know get a page and storm has a lot of moving parts it has supervisors on each machine then it has zookeeper it has a central managing agent called nimbus this is all good but we found that it was pretty hard to figure out exactly what a problem was you have a lot of logs all over the place so it was fairly complex after this we iterated some more um how many people have heard of aka cluster a few people so um starting in version 2.1 i believe aker released a clustering layer and this is pretty cool what what akka cluster is it's a peer-to-peer um it's kind of like cassandra if you guys know cassandra or amazon dynamo it's a system where there's no central master but if you have n nodes and they're all beyond our cluster they will know about each other and message each other and know when one of the notes dies through peer-to-peer gossip it's extremely cool technology we thought we would try this out so we're going to replace our storm processing with um uh with an aquacluster-based pipeline and the way that this works this is distributed is that um you have a coffee consumer this pulls messages from the kafka message queue and what this does is this will forward to everything in red is a it's an http service so we're trying to create an ingestion service as a rest service and we use we use spray because uh spray at that time spray had the best aqua support out of all the scala http libraries the next part is really neat we have something called a cluster router and what the cluster router does is it's it's an archive so sorry if i'm throwing a lot of terminology out uh in arca you have a lot of actors and a lot of times a very common pattern is that you want n actors to do uh work for you they might be this on the same note they may be distributed so the way that you spread work out to these actors is through something called a router and there's different kind of routers there's ones that do round robin there's ones that send them the random messages there's a lot of different types so one type is called a cluster router and that is aware of different nodes in a cluster so if i send a message here it is able to spread out its work across the cluster and in our case we did some partitioning so each of these processing actors is responsible for certain shard messages so this the cluster router allowed us to redirect and and do um uh consistent hashing such that messages that belong to the same chart always end up in the same place um we actually wanted uh we actually wanted it um we actually wanted a particular kind of sharding so it wasn't just for spreading load yeah um those stuff there was some state encoding going on so this was good but it was it was still too complex we had you know there's a lot of notes talking to each other um the aqua cluster in 2.1 wasn't quite ready it was uh still kind of experimental phase since that since then it's actually been like it's no longer experimental so now it's a lot more stable and there are actually a lot of people using it um one thing we found was that so scala is a dangerous language there's like akka is a as a library there's a lot of different concepts you can use there are actors and there are futures so just out of curiosity how many people um don't know what difference between actors and futures are or not quite aware so okay so i won't go over that um so one thing is that http request requests are our request and response model but actors are more of a message passing the actors say let me tell another actor something um futures say let me go and do some work and come back with an answer so the natural intuition you have is that for http which is request response you want to use futures because you want to send out some work you got to reply back then you can send back the http response so that's a pretty natural pattern so what you end up doing is you start mixing actions and futures i might have an hp endpoint this will send off a fire off a future to an actor to do some work then when that actor is done comes back with an answer then um we would respond through the xp response this is fine but we found out that when you had a lot of actors in futures mixed together it still became hard to understand it was hard for our engineers to conceptually wrap around you know when things transitioned and it was hard to keep track of both in the end what we found was that using mostly actors was a lot easier for most people to understand it we had very few trouble like most people got the idea of actors right away maybe because it's it's almost like pure like small talk like object um object pass with message passing and futures were much harder for people to understand for whatever reason so the final system that we moved to is a pure actor actor system where you have independent nodes and we basically used kafka to partition our messages so kafka is a really nice property that um kafka has what are called topics and so let's say that i'm ingesting a like twitter feed and that would be one topic and maybe my facebook stream would be another topic and within each topic you can then set a number of partitions and when i'm sending messages into kafka i can actually i can essentially shard my messages in kafka and push to particular partitions based on hashing so we decided to partition in kafka which means that each of our processing nodes is now completely independent and we are just reading from kafka pushing to one actor pushing to another actor pushing to another actor it is super simple it is completely linear it is really easy to understand um so that's a system that we are very happy with because it's very simple to debug at two am in the morning there's only one application and only one log file yeah question messaging yeah that's a really uh interesting uh topic so um we only do a kind of a we've thought about this problem there are actually some frameworks on top of akka that provide the message guarantees i think it's built into the new version of akka now at the time they didn't have that and um our plan was actually to um we don't do so much the message delivery guarantee at this level it's more of a system-wide level basically these guys write to cassandra so we do a check we we check the data um based on statistics here and and we loop it back basically it's it's we do guarantee message delivery but it's based on the whole system it's it's wider than just the scope of this thing it's how we do it but there are definitely advantages to doing guaranteed delivery within within this system itself you get much faster response times and yeah i think the common way is that if you detect a failure you keep track of the so so kafka is different from most message queues and that it uh the way it works is that you read off of offsets into a file so the typical way to do guaranteed message delivery is that you keep track of the latest offset that has succeeded so i have an offset here it commits here and um if any failure happens i roll back to the latest commit and i replay messages that's typically how the pattern that you use i'll take one more question reinventing the will but you can achieve the same thing just within akka and then you have one last moving part just can you say a little bit about the deliberation of why i still use kafka and you know oh like so why use kafka and akka together or was it just akka and do the message delivery daca you could do the message delivery part with aka like so kafka is well the reason why you use kaka kafka is for other reasons it's because kafka stores messages on disk so if things go down like you have a you know however much buffer that you store with kafka you can it's more it's more of a it's it's a really good way of buffering messages storing different disks and like if if are you talking about like instead of using kafka you have something some service that feeds directly into akka or you probably could um yeah it probably would be even some sort of used uh akka itself but i think you would still have to implement what kafka does for you which is disk based very fast uh message processing or um as a message queue uh so um you could use akka for that but you would have to basically all the stuff that kafka is really good at such as writing this quickly and you know all that stuff you would have to do anyway so kraftwerk does replicate yeah so it gives you kafka basically gives you like this buffering what i mean it's like this right let's say you have a huge bike of events right on like for us it would be like the pope got elected or some event like that right so um you might not like you might not be able to spin up enough new aws instances or whatever to scale that scope so kafka is really nice they can buffer a whole bunch of messages for you and allow you to absorb that load right so you could implement it yourself in akka but honestly kafka has been one of the easiest to run pieces of software around it's extremely solid it almost never goes it pretty much never goes down so yeah but but it's starting to blend the line because like now akka has aqua persistence which allows you to store stuff on disk oh also none of aka per system wasn't around when we started writing this that's the other thing you know so it's it's it's it's in 2.2 this was like 2.0 201 or you know so uh so jake is the creator of kafka yeah we're going to do a meetup in april with him so i think the difference is half has been in production um since 2011 yeah it's basically a distributed consensus system if it goes down right there is a very rigorous process to define the state and they tested into introduction rigorously and nothing in our stores was tested to this extent right basically nothing i think naka was tested in production at scale to the extent which has become a specific purpose so you it's very fair to use it for essentially guaranteed uh at least once delivery right so it's kind of very initial but yeah i i think now it wouldn't be too hard to write a a kafka using akka although i would say that the trick is really making it like reliable and all the other stuff um oh cassandra that's probably like that's probably the topic for another talk but like really really briefly yeah cassandra because we can um allows you to easily um do uh immutable like it allows you to do uh to rewrite records uh extremely well uh effortlessly like if you were to do the same thing in hdfs like you can't append to hdfs you can't like rewrite blocks through the file right so yeah one more question yeah um so this you have these processing pipelines right so it comes into account yeah it's going to flow down right yeah so a couple things so from these actors are actors capable of message concurrency like can they process multiple messages at once in a single ah okay so the way actors work is that you have one mailbox and you process one message at a time so so the pattern for concurrency is that you have multiple actors and and you distribute messages amongst them are there do you have challenges because i am i guess i've done actually a bit of cbu design and there's a very similar problem to this when you have a pipeline cpu and you want to make sure that each stage takes about the same amount of time to execute in order to maximize the throughput otherwise you get a stall like in the middle so is there a similar problem to this where if you have one stage so to speak of this pipeline it takes a long time to execute and then another one is really slow like how does it deal with like message contention or backup or you know or those kinds of problems ah okay i think i probably like answer the question a little bit so the way that actors work under the hood is that you have a thread pull and actors basically they can run on any empty thread and so let's say you look at one thread at a time you could have actor a running one chunk and then the next moment it could have active b and it gets like mixed around so this works really well when um so actors are designed for the message uh event processing uh paradigm where you process small chunks at a time so it doesn't work as well when one actor takes a long time uh and that might be the case for example if you need to wrap um synchronous like say jdbc or thrift or well in our case an example is the cassandra wright writer um these cassandra uses thrift at least the old cassandra did and thrift is synchronous so we would need essentially one act essentially it becomes a bunch of actors becomes one actor per thread and they they would take a long time to do the work because it's it's blocking so so that kind of that you would run into some issues like that typically the way we manage that is that any actors that wrap a synchronous code we would spin off in their own thread pool so that uh that we can manage those resources separate from the things that are not blocking like that's like a node style callback and an actor right you're doing it so you can actually spawn a promise with it so that's one thing the other is actually going to be you're going to spawn multiple actors rather than burden one actor with a lot of work so there's various policies for how to allocate actors and how to the callback is basically a future so you can send a future to an actor and an actor can do a reply and when it does reply it will send back a message to the original guy who sent it and and then that future gets completed right and so that would be a kind of callback yeah no no so um cool so this is really interesting architecture i'm gonna move to the second part of this which is something i call stackable actor traits of course everyone loves traits in scala like you can mix in different functionality so what actor traits well one common thing that you do when you productionize system is that you start adding logging then you start adding instrumentation like metrics um tracing code it starts getting pretty repetitive and ugly and pretty soon like your code you can you can't really see the original business logic anymore because there's so much like logging a lot of stuff wrapping it right so um stackable trades is a way to work around this and to add standard behavior to uh two actors um so since this is a scala meetup we got up some code right so let me see so a if you haven't seen the actor code before the comment pattern an actor is that an actor is a class and there's a method called receive and what what is inside the receive is a pattern match the actors are essentially untyped so they can get any type of message so what you do is you do a case with the type of message that you want to act on um so we'll go into this pattern a little bit where like we're going to wrap the receive method and that will allow us to add behavior to it such as logging and tracing and other things like that sorry i don't know if i've heard you correctly did you say you're doing uh you're going to do a bunch of case statements um yeah so let's suppose that your actor has to handle uh two types of messages like register and um i don't know give me another verb unregister okay so so you would have a case register and a case unregister and each one would you know have some have some code so it's a standard scala pattern match so it's uh whatever you're going to give it to match it's going to try to mention that and if it's not matching it's like you know maybe another one so you can actually because it's a partial option and by factoring different uh behaviors into aggregations of partial functions you get that kind of composition of behaviors so self-polymorphism which is composition of passion functions thank you thank you one more thing would you compare this to aop with before and act after this looks like a much nicer way of doing it uh i have my opinions on that if you really want it i can tell you my opinion is on aop but i i honestly like personally i prefer no magic so i prefer to explicitly know like what kind what is in um you know what what my classes are doing so i like explicitly calling it out so i'm gonna so we're gonna start with a base trait uh by the way the latest the latest akka has something that is actually a bit nicer than this but um the way it works is that we're gonna say for the for your actors instead of implementing receive you should implement a wrapped receive method the reason is because we're going to redirect the receive method to to point at this other guy so that we can uh stack it if you don't do that then you um it gets very tricky the inheritance gets extremely tricky so could do super not receive but if you end up stacking a lot of stuff um there's a whole bunch of edge cases that don't play very nicely uh with that but akka has something built in now to to get around this problem so i'll give you an example of like two two traits let's say that you have one which is uh instruments before and after and the other one does untas uh displays this this simply calls super.receive and um what what you what you do is that you would say extends actually with instrument one with instrument two and then when you send a message to this actor you'll see that this gets called in the order that it is mixed in we'll go into more of these traits later and what uh how like specific examples yeah that was just a um really quick example so so this is our akka stack we use spray which uh is a pretty high performance http library which is all like aqua based with yammer metrics um 2.x yeah kind of fairly fairly standard stuff otherwise one thing is that for distributed systems visibility is extremely important right so although you guys all know this um we um we like using uh logback uh because it it has a bunch of really nice characteristics like you can adjust the logging level dynamically at runtime which is pretty useful for debugging and we take advantage of that sorry i think it's the default for akka um well akka doesn't uh akase has sl4j you can implement any backend you want the only thing is like um there is a trick but it's it's very hard and basically archives immutable configuration so there there's um there's no dynamic way there's no easy dynamic way to adjust your login levels because aca has its own login level which is part of the configuration although there i found out there is a trick around it apparently so um so for example we would have a a trade a stackable trade that implements uh lock lock back style logging and we would override a receive function and so before it caused the main receive method we would do an mdc output so what the mdc does is that so it ensures that when i do my logging i have a variable that expands out to the actor path because one one problem with traditional java logging is that it's thread-based but in aca because your one actor can run on many different threads through his lifetime like the thread-based logging is doesn't help you at all for for debugging what actor was running some code so instead what we do is that we put the akka path in an mdc which is a thread-based thing and when you do logging it will you can then you can see which actor did some work we also do something nice where like we will automatically print out that an actor started which is pretty useful for debugging like initialization sure what is the difference between the stackable trade oh yeah so he's talking about in scholar partial functions you can compose them um the difference is that the stackable trades allows you to wrap code around the receive function you can execute it before and after when you when you stack partial functions what you're doing is you're running one um and then the results get fed into another one so it's more of a chain where you do something first and then another whereas this allows you to wrap it like this example doesn't like this particular example you could probably have done using the partial function composition but i'll show you another one where you wouldn't be able to do that um which is with performance metrics so we define a trade that adds two metrics for every actor one is that one is we want to uh measure how much time is spent in the receive block so we want to see what the latency is of a particle actor another thing is the frequency so that these things can be fed into a chart and you can see how fast different actors are moving right and so this is something that for example you wouldn't be able to deal with the partial function approach is that we're going to wrap a matrix a timer around the receive block and we have a try finally and basically this will measure the time of the block that how long it took to do that logic in that block and when you instrument this way then you can get like pretty nice charts showing like you know these are different actors and then you can see like how they vary over time like as a response to load um which is which is pretty useful for debugging okay i could ask this afterwards so uh i'm coming from a java background so i'm just wondering if the stackable trees is kind of like aspect-oriented programming like were you right um well it's it's quite different um it's quite different yeah thank you so one cool thing that we can do is that we can trace how fast these actors are changing over time as well as you can trace different nodes like you know which ones are doing what and there's um we can see that um let me see for example like these are different stages in the pipeline and like there's a spike in the latency at one point and this kind of corresponds to like so you see that there's certain parts of the graph where it's a little bit hard to see but but there's certain certain stages are falling behind the other stages because um the the rate is a little bit slower and for example like uh let's say the cassandra is slowing down so you could then visibly see that the cassandra the rate of message flow in the cassandra actor is slowing down and and that corresponds to spikes in latency right so that's how we would use these metrics in production um by default the mailboxes are in aka or unbounded this is really interesting because what happens if you keep feeding messages and you don't process them then eventually your mailbox will will blow up right so this is pretty interesting there's different ways of solving like basically you need a flow control mechanism when you are sending messages back and forth one way to do it is using the cheapest way to do it is to or the easiest way is to use bounded mailboxes what that you know it's just like a bounded queue in java when the mailbox is full um aka will have a setting where you it you can define like how long it waits until it's not full and if it waits the whole time out and the mailbox still full the current implementation of of aka is that this message will go to something called that letter mailbox the old versions it actually used to you used to be able to set it so that it blocked forever but then they got rid of that now there's a major disadvantage to this is that this only works this only works when your actors are all on one node because obviously when you have when you do send messages across then the bounded you know doesn't i mean there's there's no easy way for node a to check the mailbox size of node b right so um if you wanted to work for distributed systems and overall like what the guys recommend is that you work on a real flow control system for example you can do a pool based model where your workers pull work off of the first guy right or you can do push with like ax so there's a lot of different patterns you can use um that takes more work but whereas the boundary mailbox just takes one config change but um like the realflow control system lets you design distributed systems basically not let that overflow happening just reappear well i mean i think it's yeah we can we can talk afterwards but if everything is local you can simply use a bounded mailbox that will slow everything down well it throws it away eventually eventually it got thrown away so so basically you you need i didn't say that you get like you know message guaranteed message delivery right so you still need something that will make sure that messages get delivered but i think i think the real answer is that you really want a flow control system that can detect like when you've dropped messages like when when your system is like uh can't handle anymore and will slow down the source right so um so you could answer this later if you like um so i was just wondering for the wheel flow control distributed case uh what does that look like exactly do you have multiple actors getting some like name or you have multiple we can we can talk later there's a lot there's a lot of possible designs so yeah so um i'll show you like an example the flow control system that that's working is that this is an example when we first turn on kafka and you can see that the actor that is consuming from kafka like gets a huge spike and then these are the later actors that are processing the messages after that and you can see that they start converging the kafka one slows down and this one shoots up and then they converge right so that's the kind of graph you want to see if your flow control system is working right um how many people here use visual vm it's a pretty useful debugging tool it's free it ships with some version actually you can download it from oracle or whatever and if you use bounded mailboxes you'll discover that your actors would spend they might spend a significant amount of time in this nq method where it's waiting to end queue messages so yeah um visual vm is is really nice for the for java like threaded programming because you can see like threads doing work my dream is to have a plug-in that allows you to visualize actors instead of threads that'd be really nice it offers the types of console which is actually free if you run it um in within svt um it's not quite the same i mean i think it allows you to discover what the actors are and stats about the mailbox for each one uh vision vm is different allows you to see like all of the work going on at the same time so it's a little bit but yeah that is a neat tool we just you know you can't use it in production so um so one thing that is hard about arcade development are stack traces because you see something like this you see a few messages and then a whole bunch of academic dispatch or gsr166y or something something like that basically you can't see past when you've received the message because um it's the that's a fork drawing pool and dispatch stuff and you realize that it'd be really nice if you get an akka message straight a stack trace where i can see um you know what i've gone from this actor this other actor this actor to that other actor will like that kind of thing so we can actually use the stackable trade pattern to solve this problem and get really nice message flows like this this was a hackathon that we did using d3.js where you can see uh kind of a message flow and the width kind of tells you how many messages have been sent you know kind of thing pretty neat so this is not that hard basically we have a we have another trade called we call it tracker for tracing akka and all this does is we have a partial function called the message id extractor and what this is designed to do is given the message it will extract out an id from it you can define it depending on the semantics of your application and what this will do is this will send the message id along with the source and destination actor paths to a collector and then the collector can build up a graph and create those beautiful flows so this is another good use of of traits um i think this is some random development practices um i'm not sure i really have that much to say about it um some some good patterns that we use is that well what one hints is that like unlike traditional classes where uh you put a bunch of initialization in a constructor and then it fails you get a stack trace for for akka um that's more dangerous if if you uh the default so all actors are supervised so something happens like an error then the supervisor can restart the actors which is a really nice thing it's a fault tolerance by default if you have an error in an actor's constructor then it will stop it what you end up with is an actor that seems to be live but is actually in a suspended state and is not doing anything so my advice is that for any complicated initialization should put it uh have an initialized message and have that work be done in the actor itself so that you can use the normal fault recovery mechanism so if i put it all together all the traits that you've seen this is something that uh you see that you know we have uh we have for example an instrumented actor trait this mixes in our logging trade our actor metrics trade or tracing trade and you can just add whichever one of these you need right and um then we have the class extend this and this is the very different from the aop basically i know exactly what is being instrumented in my code which i personally prefer and you get all the other stuff for free without clogging up your business logic so the next steps well this this talk uh honestly i gave this talk at uh the port uh pacific northwest scala conference uh end of last year and uh unfortunately we still haven't open-sourced this but at some point we are interested in open-sourcing it so talk to me if you're interested thank you very much