Scale By The Bay: Michael Armella, The Danger of Implicit Blocking in Finagle
Recording: Scale By The Bay: Michael Armella, The Danger of Implicit Blocking in Finagle
thank you my name is Michael our mama I'm a senior data engineer at Credit Karma so I Credit Karma we provide credit reports and credit help for over 80 million members and what I work on is the recommendation system and what we're doing is providing credit products and advertisements on the site for our members and what I want to talk about today is the danger of implicit blocking and finagle so finagle is an RPC system written in scala and it's designed to for high performance and concurrency and it was created by twitter and we use this as part of our servers and client architecture so I hope everyone should read this blog about block party the Twitter came out with and it really explains what the problems are with blocking and why it's such a big problem so there's two types of blocking there's explicit blocking where you call a wait that result or a wait that ready and this will completely stop your thread from doing anything but except for waiting for the results of the future that was called but there's another type of blocking that happens and it's when your code runs some synchronous operations it's doing a computation and it just it's waiting for it to finish so this would be like running a very long computation or doing something that that doesn't return right away so something like calculating a factorial or some other mathematical operation will be a good example of that so I'm going to talk about what what what happened and what our problem was so we have a service this cuz consisted of multiple finagle services so each one of these is a finagle service that's running on a separate machine some of these services are backing or fronting my my sequel instances and others fronting on a cluster and another is fronting a BigTable instance at Google BigTable instance so we have the orchestration service which actually takes in the request and then it will imperil l call the configuration service and the data retrieval service and then it will pass this information on to the data processing service which would do a long computation and then to the ranking service well what we noticed is in testing in our staging environment I'm very low load there were no problems this ran in like under 200 milliseconds everything was good and we deployed it in production as soon as we saw peak load times we saw lots of problems so immediately the latency jumped up and sometimes it was up to 10x taking up to two seconds to process a request and so why was that it was because somewhere in our data processing service we were actually running a long computation directly in the apply method of the finagle service and this stops it from being able to properly respond to requests so let's take a look at a simple service with no latency problems so this should be a boiler plate service that everyone who's familiar with finagle should have seen this is basically just creating new service it runs in constant time has a constant response the only time this would take would be the network time so what if I change this to actually compute something so here we have an await dot result does anyone know what happens when you do this anyone so this will actually run very badly so when I call await our results and then I sleep it's going to sleep for one second and it's going to completely stop responding to requests for that one second so let's change it to do something else that's more sane that someone might actually do so that previous example I hope that no one would ever do that but someone might do this they're writing a service and then they write some function like calculate the Fibonacci sequence and they say give me the hundredth Fibonacci sequence and return that as as my response so does anyone know what happens if you do this who thinks this will perform better than the previous example who thinks will perform worse yes this will actually perform worse or at least just as bad so what's gonna happen here is this is going to start calculating this Fibonacci sequence and it's just gonna Bach just like the other one did and it's not gonna have any way of responding to new requests let's take a look at another example where not only my calculating the fibonacci but I'm actually making an outbound requests at the same time or just previous to it so this should be fine it should seem like this client request should go out it should run async we'll see on the other machine and then I should be able to calculate Fibonacci on this machine but when you actually run this it's actually it'd be worse and so the reason why it's gonna be worse is because when you're blocking and you're wasting all of your threads time calculating Fibonacci it can't actually make the quest call so when you have a very high load on your machine the when the new request comes in it's going to try to make a request out and the request it makes out it's going to be using a similar thread that is actually processing Fibonacci so the reason why this is is because underlying finagle is is is run on the nettie framework so the navy framework is the niño framework that is used by finagle to handle all of its IO operations and what's going on with that is that every I Oh channel has its own event loop so the event loop for incoming requests is is on the left and the right are all the outbound requests so every clients your service connects to and they all have one central worker pool so when an event comes in our request comes in it will get picked up by a worker thread and then it will be processed and it will it will try to make a call back it will also those same threads will make an outbound requests on on the event loop on the other side so what happens is finagle expects your your service to immediately return a future and it will register that as a callback if you don't immediately regice your future you're wasting a worker thread that cannot respond to pick up a new request off of the stack of the event loop so a good analogy of how this works is it's like a switchboard the operator to receive your call and then immediately tells you to hold and transfer you to someone else in this example the switchboard is your event loop and the operators are your worker threads so well what's happening is like imagine if one day that on the switchboard they tell them instead of when someone asked for their bill details don't transfer them to the billing department just calculate it yourself and tell them even though it takes a minute or so well we've quickly happen is under peak times all the operators will be busy calculating this bills and telling the customers rather than transferring calls and eventually there'll be no operator available to actually transfer calls so and what happens we actually balk so what's supposed to happen is the worker thread is supposed to register call back immediately so it's supposed to just take the request start the asynchronous work and return that as a call back but what happens if you don't do that instead you're just busy processing some computation you're not returning back and there's no one picking up the new request off the event befool the event loop and so if if this happens then more and more requests just sitting on the event loop and not processed and it causes a lot of problems and the number of workers is by default limited to two times the number of logical cpus which would be the pores of the machine so the easy way to identify this happening is to look at two metrics that Venango provides that's blocking MS and handle time microseconds so blocking time ms it tells you when some part of your code has explicitly called a wait that ready or wait that result so if you see this time at all in your code it means somewhere there's an awaits that's actually blocking your code you should really find it and remove it but handle time microseconds will always exist in a small amount and it's the time takes for a request to come in a future to be chained together and return back to the hauler so there will always be a small amount of this but if this is growing very large it means you're not responding to the request fast enough and you're not building your future chain fast enough and you to debug why this is happening another problem to look out for is this this does not just affect you processing inbound requests but it also affects you processing outbound requests so if you want to make an outbound request using the same thread pool so you won't actually even be able to queue up outbound requests if all of your threads are taken so how to solve this problem so the problem isn't just that outbound requests are bad everything is fine we make an op on request it is perfectly asynchronous but you need to minimize the code that is actually doing work inside your your apply method that is not asynchronous you need to avoid using wait and you need to run your code in a separate feature pool so future pool is what is the thread pools how they're implemented in in finagle so let's look at an example that doesn't block so in this in this thread in this service all I've done is I've wrapped the previous one in a future who thinks this will perform correctly does anyone think it will perform correctly so what does anyone know what the problem with this is what it's not that's not actually the problem no so yeah so the question you should be asking is what type of feature is this this is not a Scylla future in a Scylla future this will automatically run in a new thread in a Twitter future this will run in the same thread that spawned it so this is a very big difference that a lot of people don't realize between Scala threads and Twitter threads so this will do nothing you're basically just wrapping a constant operation with a future in the same thread the proper way to do it is like this III knew future Poole and what so this is basically a new set of threads and then spawns of the future this code I wouldn't say is very production ready because it does have an unlimited number of threads but it's much better than the previous code so now if we do this for a previous example what's gonna happen here is I'm gonna spawn two asynchronous operations and then I'm going to map on both of them this will run perfectly fine but there's one catch that when you do the mapping at the end where I have a response to a set content string what's gonna happen here is it's gonna return back to the initial thread so if you do a lot of work and when you map on the threads in your PI method that is also going to block so be careful with that as well so basically in conclusion latency problems are not always obvious there's sometimes hidden things you don't know about going on in your system you should always investigate oh why you're seeing these weird latency problems and you need to pay attention to the metrics especially handle time microseconds and blocking m/s how you execute your long-running computations really matters you always need to run them in a separate thread pool and this is implemented in finagle with the future pool so the best way to do this is to actually tune your your future pool to be the size appropriate for your cert your architecture and your your machine and that's about it thank you any questions [Applause] yeah how do you what so it's basically more memory independence of usually unlimited thread pool what's gonna happen is if your requests are coming in faster than you can process them you will essentially make an infinite number of threads and you'll fill up memory and you run out of memory so you need to balance out when you're gonna basically start rejecting a request by or slowing them down and delaying them you could use like the basic JVM profilers and things like that because this is all running on the JVM so everything that normal JVM Pro would would tell you any other questions no so there's no like I said if you when you compile this you won't see it when you run it under low load you won't see it it only kind of appears when you have a very high load so it's very difficult to statically check this I don't have any preference I usually use like artillery or things like that that are or are things in the called Gatling that that actually will put a heavy load using these will will really show you what your system looks like under load at least in your staging environment okay no more questions all right thank you [Applause]