SBTB 2014, Ivan Topolnjak: From code to dashboards: Monitoring a Reactive Application with Kamon
hello guys can you hear me well fine I'm I am even doable yeah I'm one of the core developers of come on are any come on users are on here yeah thanks for using our tool for those of you who don't know what c'mon is reminiscent officers project is aimed to monitoring applications made with a cat spray and play and it's meant to be used in production it's not just a toy thing that you use well your development is something for using on production it has been developing under development for over a year now we're type-safe technology partner since May May this year so there is collaboration to make this even better and it's already being used in production for some of our users so that's something that you can rely on the motivations for doing this is that when we started doing some really cool stuff with spray anecca we got to the point with where we had some apps that we wanted to take into production but we couldn't do that without having proper monitoring and there was nothing available that we could use the only thing that was available at the time was the type safe console and it wasn't like the best thing to use at the moment and also we could not use that then so we saw what people was doing at the time some some guys were wrapping every single message and their applications or their futures or doing some sort of metric collection stuff and everyone was doing something different very similar things but everything was different and we decided to do something that could be shared with other people that works well and like try to solve these problems once and do it for everyone so why do production monitoring is is very surprising to us that many people have applications in production but they didn't monitor that and there is a sad thing that is there is a gap between what you think about what your application does the way that's that M production at the way that's that and development it's always different and do you need to know how your application is going on when your users that are using I mean good one happy user Judum when users waiting for your page to load so there is a talk made by Cola hell where he spins very well why you should do production monitoring and i highly encourage you guys to go and click that link and see that talk so you know what i'm talking about next I'm going to talk about the two biggest challenge we have when developing this tool and the first the first one is keeping context across the events so if you look a at a traditional application like you know the old guys doing java ee stuff where everything is blocking and there is a thread that is always touched to the execution of a very specific request so if you do some jdbc calls or you do HTTP requests or you have some business logic to do everything happens sequentially in the same thread so it's easy to keep context there because you can you know put something on a threadlocal and you can access it everywhere when when you're doing the application so some people move a little bit beyond that and started submitting things to thread pools so you can have some sort of parallelism while working on a request but usually you do that but you still have one thread that is attached to the transaction and when those things complete and come back here then you keep doing something sequentially so if you have a threadlocal and maybe some other tools you can still keep context around other things but when you work with reactive applications when you working with that guy would spray when play it's not the same thing there is no longer a single thread attached to your transaction and everything just flows through the actors or future show this in Kyle backs and futures and you need a way to keep it context context across all those things so what we did was we created something we call it dress context and that thing has name has token and has some other things that we use for gathering metrics around that and we propagate that through all the synchronous events on the system so if there is a trace context available when you send a message to an actor that message goes with that trace context when the receiver with processes that message that same trends various contexts will be available then so if it that generates more events than the trace context is going to be propagated across all events so we do that with you know regular tail when you ask when you use pipe also when the actor fails and you know the child fails and it sends a supervision message to the parent and that also carries the trace context and also we do that with features so when the barrier for future executors the trace contact that was available when the feature was created is also available when that executes and if you transform the viewer into other stuff then all those callbacks will also have the same trance context available so basically every event that goes from the moment you started at race to the moment it finishes will have the same Chris Torres context available so we have a very simple API for that you basically grow up a part of your code with this when you trace context thing and it creates a new one and it gets propagated all the way down to the moment you do a trace recorded at finish so the next challenge is well now we have context we know when something starts and when something ends and now we want to measure now and when I mean measure and measure everything I don't mean like having too many things being counted when you count the right things you need to know all the values about that very specific thing if you're measuring latency is if you're doing sampling for that and you're like something one out of each 10 requests and you measure that basically we're throwing away ninety percent of your data and that can be a lot of people that is falling in that ninety percent that is not happy about the service you are given them so you need to know about everything so if you're measuring latency for this specific request you need to know all the latest measurements for that specific request and also you need you take statistics about that so you need numbers that tell you things and we don't know what is important for you maybe you like the mean the median the maximum it's obviously not a good way to go about measuring stuff but we don't know what you want to do so how do we solve that well we keep all the data all this all the measurements we take from your application we're keeping them and you decide what you do with your information so it's kind of easy we just need to be able to store millions of measurements per second which can happen in many busy applications we need to have a limited and predictable memory footprint for doing that and also do it with a very low overhead it's pretty easy stuff I guess you know it is not and many you're asking yourself is it even possible to do that and yes it is there's a thing called the high dynamic range east orem it has been created by guilt any form a social systems I don't know if you heard about him is a really nice guy and this data structure is like a way of having a histogram where you can put data but it does with a configurable precision system so if you restore a hundred milliseconds there it it is not actually storing 100 milli second city will correct the value and the precision is configurable so by default we have one percent so every measurement we take restore it there and it won't be more than one percent away from the valid you actually stored and it does that in a single data structure is basically a big longer ray that is in memory but it's only allocated when you start your a band so see if I start an actor then there is an ester I'm for storing things about it it gets allocated only once and there are no allocations when you actually do their recordings and also the recordings itself are very cheap because it's just about doing a couple bit bit shifts to get the index you need to put data on and do plus plus 1 and that index and the conscious array so it's very very fast you do measurements with this thing so now that we have this thing we are we instrumented the Akamai code by using aspectj we're doing that with scholars all the aspect goes is in Aeschylus there in github we are measuring measuring trace matrix so traces for us is like if you have a request coming to your application you do a lot of stuff to process that and then you've replied back to the user the time taking for the application to do that that's what we call a trace metric we have segments which are like little pieces of your trays maybe a database call and HTTP request to and other service and so on we have actor matrix so processing time mailbox size and timing mailboxes all kept in a HDR a theorem so you get all the values also we have error counters we have dispatcher matrix and we also have a new system matrix model which is still experimental but we also can give you system matrix information also we have user metrics because but we did all the work to make this store a lot of data so I guess you guys may want to use that for your all things so even if you're not using knock on door you're not using spray and so you can ask him one for a histogram account or main mexico hunter which is a thing out we used to monitor queues or gouge and then all that information can get flushed to the same metric vacuums we already supporting come on even if you are not using a core spray so we also have a spray and play integration that means that that little piece of code that they show you that tells you hey here's where at race starts and here's wait where it ends you didn't have to do that if you're using a spray of play because we already know where play starts something when it finishes so we have instrumentation for that we have segments so if you do a client HTTP call using of the spray client or the ws client in play we already record the temp the timings for that and also we have automatic twister in propagation which means that if you have a traitor trace turning one machine and that machine mixing HTTP request to another service we are putting a header there with the trace token so when it gets to the other servers it will use the same trace talking so basically you have a universal ID that can go across all your notes and you have a way to for example if you're logging that you can correlate all the loves for a single transaction across all your servers and that's really helpful when you're doing this kind of obligations we're currently we currently have modules for stats d data doc new relic we have a lock reporter and when we're currently working on a influx be module do your report there if you guys want you get your data anywhere else you can just talk among hasten the actor metrics to this actor over here and it will periodically flush the metrics to that guy too and in all rodman roadmap we have well the next version which will come out some time in the near future like two or three weeks from now we are playing to stabilize what we currently have and make it more user-friendly have a better integration when you rarely with that add up like make things we already have a bit better then the next release release we're going to have cluster and remoting support some of our users are already using a cluster or remoting so we want to make that work too and then it is not clear when are we going to do this but we have plans of queer first create or on metrics Dashwood it turns out that we have a lot of informational of a lot of very valuable information but most of the vacuums either downsample the information or are not able to show the kind of rest graphs that we would like to show about these things so we want to do our own thing and let people enjoy about that we want to create a new Jacob module I don't know if you heard about Jacob but it's quite interesting that it can help you detect how hard the garbage collection and other things that things that passes your system are affecting your app then do proper Evan float racing like you can get a Gantt chart something like Zipkin death but with actors and features and synchronicity operations across Europe and support other non reactive stuff I think the first thing we're going to do is cisco otra which is being used by some of our users but that's it so far so that's all I have for now if you want to get started with Cameron please go to your side there there is a simple guide I know we have a lot of documentation but we still need to do a lot more so if you will go there looking for something and you cannot find it please let us know and we will have the help you there there's link for the mailing list the Twitter account so please reach out and we will help very gladly so are there any questions yeah elasticsearch has the back end we didn't have any plans but I guess it is absolutely possible estill you just need to subscribe the actors and send the metrics there so yeah it's possible we didn't have any plans to do that yeah in yeah well the Kota hell matrix has like I guess the only thing that is worth comparing is the histogram itself and the koala hug i am library it has i think its reservoirs the thing that has to store the things and one of those keeps all the data but it basically has a unbounded list of lungs so you're going you can go out of memory by using that the other one is a exponentially the kind of reservoir that throws that away it makes the assumption that the distribution of what you're storing there is normal and based on that it will throw away some data because it thinks that it will fit the normal distribution and if you have been measuring latency you will know that latency is nowhere near and normal distribution and I think there's another one that i can remember now but basically with with the high dynamic range chest gram we store all the data points we have all of them and you have it available so you can report it anywhere you want yeah context all the whole different actors can you please repeat the question a bit louder synchronously and it is moving to 40 reductive this context yeah yeah of course you in that case you will get your things you get the accurate matrix itself so the actor you know how long does it take to process messages in the actor and if you say well my tracer starts here and and finishes for actors away then it will measure how long did it take from this point to that point okay that's essentially what happens with spray there is a bunch of events that go from the server connection all the way down to the response and you can also do that manually actually one of four apps it's like a crawler and we don't have external request but we are generating load and we do have some management like a manual setting for saying here is what it start here's where it ends and come and we'll measure that for you I cannot give a strong opinion about influx to be because I just started reading about it it seems really interesting but we will still need to explore that further from the ones we have I guess stats d it's going to be the default way to go because even while it down samples some data you can tweak stats d to get more data out of it and actually we have a darker image if you go to the side and Tim block there is a boss where we publish it a darker image that has stats the graph anna and graphite so you can just start that because installing that is really painful you can just start that start your application would come on until it you send the data there and you will see that with actor metrics and trace metrics and so on any other question it was one around no I actually a guy asked that a few days ago we don't but I think there is a bridge between stats d an open TSD be that you could use to get the data there any other question ok thanks