Scaling Reliability

scala.bythebay.io: Moses Nakamura, Scaling Reliability

Recording: scala.bythebay.io: Moses Nakamura, Scaling Reliability

I you so I'm closest the subset of this talk is so you want to add 9 here when we're talking about nines we're talking about nines of reliability where we say like generally we want your service to be alive most of the time talking about how what percentage of the time are you alive question the ends up being are you 19 which would be ninety percent of the time which is sort of pretty bad are you 29 99 percent of time still actually pretty bad three nines now we're getting somewhere so okay I'm most mcmorran i work at Twitter I work on finagle which is an extensible RPC framework follow me please I need more engagement I actually get paid poor tional to number of followers I have so this is that's not true but okay so sort of the core idea of this talk is that as you scale your organization up it becomes actually much harder to maintain your reliability this is true both as you scale your organization up and as you scale your architecture up so we need to change the way that we look at reliability as we grow our company in order to keep up with those problems in scaling your reliability so who cares so Twitter cares a lot probably some of you will recognize this image this is the fail whale it used to be what Twitter was known for basically our reputation was if you need somewhere to go look at a 404 page we have a great 404 page that wasn't really what we wanted to be known for our current brand strategy is pretty different actually so that was like eight years ago and since then we've made a lot of pretty significant improvements a lot of these improvements have been on these metrics things like up time when we're talking about nines of reliability this is often we're talking about in terms of like minutes per quarter dollars like when your website goes down often you're losing dollars whether it's ad dollars or if someone's going to pay you directly for some service or whatever your success rate which is a number of successful responses you have divided by the number of queries you have the mean time between failures and the mean time to resolution these are all things which are going directly impact your users and things we care a lot about I work on finagle finagle has a pretty big focus on robustness and resiliency in the face of this class of failures likes failures and distributed system twitter is pretty much all like the online parts the parts that you interact with are pretty much all micro service architecture right now so something we're thinking a lot about as we were building finagle and as we scaled up Twitter was how can we continue be reliable as we increase the scale so a lot of the techniques I'm going to talk about are built in to finagle so really what I want people to get out of this talk is not necessarily what finagle does although that is sort of what this talk is going to be about but more when we ran into a problem here's how we looked at that problem here's how we solved it and here is why we think that this was a good complete solution to that problem so when you're first starting your company sort of the first thing that you do as like you have your big made your first big major failures you think okay we need to get we need to start owning our own failures and we need to really like how people take responsibility for it and the first core idea is team on their own failures this makes a lot of sense as a company we want teams to be aligned correctly and one of them is if the site goes down you get woken up them opener sort of the kinds of failures that people end up owning are going to be like background noise failures is the things that just like fill up your logs that you just start ignoring great failures highs and bugs temporarily overloaded hosts crashes then the other kinds of things are bugs if you just introduce a bug which just takes down the website your team going to have to own that one interesting thing is that pretty much everybody writes bugs I mean I don't write bugs anymore but most people still write bugs so as you hire more people that's also going to introduce the rate of which you increase bugs to your organization one technique that people have to stop writing so many bugs as they introduce process to reduce the number of bugs things to increase your code quality like code reviews or unit tests engage in tests all these kinds of things and so when we start running into failures one of the ones will offer into is a probabilistic failure now suppose that this remote sir post has a ninety percent success rate that's now I would say not so good one simple and easy solution is caller says hmm how about we just keep on trying after we fail once instead of saying I failed I guess that's just a mm that's it how about we add wheat rice suddenly from a knight something like there's ninety percent success rate if you add two extra tries you can get it up to 39 s discuss what if there's a bug in the release suddenly you deploy and your success rate drops down to zero on the other hand if instead you make it a canary and you say oh look this accessory on that one host dropped to zero you can avoid the massive catastrophic tell you what if it hit otherwise so those are sort of like early attempts that people usually end up making as you scale up a little bit more you end up thinking hmm like maybe if we had a team we could start solving these in a more generic way for example finagle we added a retry filter suddenly instead of everyone having to build their own retries and retry policies and exponential backups and jitter and all that stuff into their own services could have a single place where you could put all of it so if each app is a ten percent chance of failure and we start having this big micro service architecture then as we scale up things can get more and more tricky so hmm okay that number is wrong but that number is supposed to be sixty-five percent we're basically the idea is if each tier has a ten percent chance of failure then your overall success rate is going to be sixty-five percent which is pretty bad I would say it would mean that if you if someone goes in like once to go three times on your website one of those times is going to be a failure in the after we load the page if instead we added retries to each of these edges suddenly it'll go to 99.6 percent which is still not amazing but much better all right less common failure modes so you can run into all kinds of operational failure for example even just when we do a rolling restart often you'll find that there are lots of edges where you can start failing one example is when a server is no longer serving traffic but your client doesn't really realize it yet and keeps on sending your server traffic or for example when your server is starting up but it's not yet warm enough in order to serve a response successfully bad remote hosts like if you're with the remote host suddenly has bad hardware or it's gotten into a bad state and your client has noticed yet or if you're meantime to recovery is slow like your operators don't have enough tools in order to make the right decisions so when we have caller talking to a receiver sometimes all we think is what's happening now this is a chart that might suggest what happens when I call or talk to that receiver so actually there are three services here there's service a service be and service see service a sends a request to service be serviced be sends a request to service see service see is not able to serve it correctly and sends a failed response back in service be sends a sub subsequent request to service see which is a retry see is able to serve that one correctly and then comes back today but lots of things happened here so what kinds of information might we want to know here here are just some examples you might want to know like what was the duration of this request was the duration of each of the requests from b to c it it's pretty valuable to know what failure happened what time that failure was and also it's valuable to know that you did a retry these are the kinds of things we end up putting in tracing and logging and metrics so yeah so some of the useful things it can be you want to know about your server are going to be tracing logging metrics also config information and which remote host you're talking to so failure is well rolling I talked about this a little bit before basically imagine that instead of just one remote host this receiver is actually say like ten percent of your notes suddenly like you're not able to talk to that post because it's being rolled right now so start sending requests and some of them are successful but then the rest of them it's like oh wait that server is now dead I just didn't notice so you're going to get this sort of unhappy like connection closed error or something like that instead if we build more specific more sophisticated information to our protocols like go aways for example in HTTP to or t drains and MUX then your server your client can know much better oh well I now know that I'm not going to get any responses for my other requests so these final requests which I asked for I'm just going to assume never going to get a response and I'm going to send them to another row host so basically color goes through receiver the receiver says hey chill out and then the caller chills out and go somewhere else another example of something that can happen is when you have one bad egg for example the case where one of the remote hosts has bad hardware or orange of being slow codes in a bad state doing a lot of GCS something like that instead of just sending bunch of bad responses back or yeah when you send a bunch of responses back the caller can say oh that one is like failing seventy-five percent of the time maybe I shouldn't send that went so much traffic maybe I should send it zero traffic until it recovers so as we continue sky scaling up our organization we begin to run into problems which become rare and which basically individual teams can't be expected to solve things where it's like you have a one-percent chance of running into it does it really make sense to ask every single team to build a custom solution to worry about this one problem no but when we're talking about a hundred teams if that means that one team is definitely going to run into it and it's going to definitely take down the website maybe it is useful to build it but in a generic way so going back to what we were singing about people introducing bugs if each person introduces a bug every month you double the size of your engineering team you're going to double the rate at which you're introducing bugs so some examples of failures which you begin to only see a much larger scales end up being retry storms environmental pressures overloads caused by the grand color where it's not your collar is causing the problem isn't who's overloading you but it's the colors color so for retry storms you can see the caller is sending requests the receiver and the receiver is sending back a bunch of bad responses and so the caller responds by saying okay the whippings will improve will increase until service improves and so this just overloads the remote host even harder instead if we have retry budgets where you say okay and I'm only going to allow some small percentage of requests to be retries I say okay only up to twenty percent of requests are allowed to be retries then suddenly things can get much better so overload this is when maybe you've already figured out your reach highs from scenario but the remote host is just overloaded you didn't expect to get so much traffic and so the servers having trouble one problem we can run into is sort of a queuing theory problem where you're shoving requests into your queue which is your server now and your server is not able to give you back responses as fast as yourself your requests if the server realizes that it's overloaded and start setting max back then that can be a really good signal to the client that should slow down we have other signals like for example the server just slowing down but those can often be slower or it can be a little bit trickier for the server to respond to those we actively so the caller sends a bunch of traffic to the receiver and the Reaver sends back next now is not really the best time client stop sending as money another example that can happen is if any of you cook or if we've we use mace and aurora which allows us to co-locate applications on the same host sometimes because mesa samora don't have don't give us perfect isolation we can run into problems with noisy neighbors like if you ever been scheduled on the same host as like somebody who's doing a ton of disk i/o you might have seen this kind of thing where the server ends up slowing down and it's very difficult to tell them till you go and sort of look at the stats on the overall machine you say like oh wow CPU is being really utilized a lot much more than I effective it's being one thing is that if we build in tools into our servers to detect this kind of thing then they can respond much better so the receiver thinks okay I'm being throttled for various CPU reasons and I need to tell my client that I'm being throttled so it will stop saying me so much traffic we can use an act for this also so basically the remote host says I'm being throttled and I need you to chill out and talk to somebody else the meantime so you send bunch of necks Klein says okay don't want to talk to that thing that's lending next and I really care why then stops talking to and talk to someone who's not being throttled then after your route neighbor is going away then you can start sending ascending successful responses or as some people do you can just kill yourself okay so one lesson learned here is that can be useful to have not just one team like the finagle team can be useful to have many other teams that are serving other kinds of similar parallel reasons all focused on resiliency and looking at it from a global point of view we very useful have observability storage because if you're just exporting stats I'm not gonna be useful when you want to say okay something really bad happened five minutes ago what was it if all you have is the now information you're not going to be able to figure it out automations that you know like okay like this is something I would have restarted by hand like maybe I don't want to be woken up I already know what things look like when they need to be restored by hand just don't do it by hand anymore deploys how can we make our deploy is standardized and like not surprising anymore how can we coordinate during a site wide outage which might affect multiple different services we should have a single team which can help us with that how can we spot a spread liability best practices when the generic solution might not work for every single problem another great big things that you should make your failures loudly fellers are really valuable information and if you're not sharing them with sharing this gift with everyone else then you're depriving information the organization of institutional knowledge we need to be able to have a global view so that people aren't stepping on each other's toes it all seems coordinate for the same reason we need to empower our teams in order to give them the actual ability to solve these problems completely and we want to be able to keep a history of these things this is one of the big reasons why you should make your failures loudly so that when you when you personally leave your company you're not taking that institutional knowledge with you also I'm making them loudly you set a good example for other people and allow people to not be so concerned about being blamed for making failures and trying to paper them over instead of being clear about what the problem was so you can solve it completely alright any questions what's up okay so the question is how do you solve the problem of institutional knowledge it's really hard I would say there are good sort of process improvements one example was having like a recurring sort of war stories event or you can serve spread stories about awful things that happen so people can remember these kinds of things it can also be really useful just to keep mailing lists around and to have most information like this nice searchable way a lot of people use slack for this now I here as long as you have a slack which can search for a really long time back that should also be fine but yeah it's a hard bro wiki's are okay the question is how many 9s do we target it depends a lot on a service of service basis like some things that it matters a lot when it's not healthy like ads things or the timeline some things it doesn't matter that much so I would say on the like somewhere between two and four typically okay I guess that's it [Applause]