scala.bythebay.io: Oliver Gould, Finagle as a proxy

scala.bythebay.io: Oliver Gould, Finagle as a proxy

Recording: scala.bythebay.io: Oliver Gould, Finagle as a proxy

I all right i think i'm gonna go ahead against started if you just plenty of room up front if you want to get close to me you don't have to okay so let me start with a quick show of hands who here is a software engineer you write software all day every day great it's good start of you how many people have heard of an eagle okay it's another good start how many people use finagle a bunch excellent how many of you are on call in some way are you and things break you get paged yes okay you're my friends how many of you have been in an incident in the last month or you have been paid for something okay some of you have had super nice that's great how many have you been woken up for some reason because we could be here excellent okay this talk is for you you're my people the rest of you you're welcome to stay but and so my background is i started at twitter and 2010 I was started in the ops team is one of the first software engineers in the ops team and my first on call i was ops on call for all of twitter com and i was woken up at 2am for am-6 a.m. for things i didn't know how to fix and if i'm being a hundred percent honest the first thing that went through my head was i could just quit I don't have to not to wake up like just go back to sleep and I didn't and I thought that every other time I was woken up for the next four and a half years and I didn't quit for those reasons and then things got a lot of bed over time and so this is the story of how things got better from my point of view so there's a lot that you you probably understand in a golem from a programming model but as an operator I think there's a lot of value that isn't why they understood or talked about enough and so I want to talk about first of all how it was to work at Twitter in 2010 then I want to walk you through a number of the I think more novel failure as we saw which might help give some insight into why we made these decisions and then finally I want to introduce you to linker d which is a project I've been working on for about the last year alright 2010 let's all get a time machine go back to 2010 you don't have to come back if you don't want to i understand who is on twitter and 2010 is a very different site right there are no pictures there was a star you can click under tweets but nobody knew what it did you had to manually refresh the page it was a very different product and beyond that it wasn't very reliable one of the bigger events in 2010 is certainly one of the things that convinced me to join Twitter was the world cup anecdotally there are a lot of engineers who were hoping for drawers 00 draws in the world cup because every time there was a goal the site we just fall over and we saw a lot of this and so Twitter was actually kind of famous for its failure for its very visible public on reliability it showed how how what a great platform it was and everyone to use it at the same time it was great for events but it wasn't all that reliable and so if we peek under the hood Twitter and 2010 it was one big service for the most part called the monorail the monolithic rails application and there are about I don't know a couple million users and a couple million tweets per day about a hundred plus engineers the company maybe 10 or 20 of which were in operations so there's one deploy team managing every deploy at the company making sure things go out if things don't go out they all get rolled back together so it was very and everything got rolled back basically so he it was hard to move quickly at little at all and we were down all the time for various reasons like really we were down several times a week just full down sites down everybody's pulling her out but we are growing quickly and so the engineering leadership got together and really identified that this was a real problem that we had to fix one we've had no a real reliability story it was kind of a mess and two we weren't very flexible we couldn't move quickly teams couldn't ship features we couldn't actually grow the business and add all the things that we have in Twitter today because it was broken all the time so we could move so we realized that we needed to really think about building a platform internally to build an infrastructure layer that other teams can work on without having every operational detail impact them and the other thing is that we needed to embed the ownership model in a much more concrete way which is you know what DevOps I assume means and we were using this dirty word SOA now I hear it's microservices that's very cool but what that means is that we wanted services that were owned by one team they could make all decisions related those services deploy schedules on call rotations all that can be much less Monell if they can distributed so that we can move independently and move quickly so I want to before I go into more of the details I want to start with one of the overriding principles this quote is actually from the funaga Khan keynote last year for Marius resilience is imperative our software runs on truly dismal computers we call data centers besides being vain ously complex they're unreliable and print operator error I think about this for a second if we could go back and run Twitter on like one huge computer and we gave it perfect power and perfect cooling and all its Hardware ponents could work perfect and would never break we would do that right that that would be ideals we have a huge mainframe and it would never fail but it's really expensive to do that these computers basically don't exist and so what every other company does now is they buy warehouse computers they bite cloud resources and you have all this commodity hardware and it's going to break at some point and we have to deal with that in our application because we can't rely on the hardware to do that for us and so if we think about this for a second especially in the cloud world this means that a kernel you don't even know exists a vm layer you don't know exists a container layered our know exists who knows what the networking topology is all of these things are of your control as an application owner and we still have to do business so when we hit a kernel bug in the middle of the vmas at the Super Bowl or the election hopefully people can still tweet and exclaim whatever they want to explain and so this is a real challenge it and engineering challenge that we had to face and now you really don't see the fail whale anymore so I want to kind of talk through a bunch of the failures i saw these are not all of the failures that are saw there's a lot more people that can talk about other things this is from my perspective on the platform team so in 2011 we started investing a project called mezzos is anyone heard of messes here then when using mezzos here excellent and so mrs. idea is that we don't have to worry about the hosts we can just have a layer that sits over all of the hardware and so that as an engineer I'm not memorizing host names as SS a Qing into things and doing curl deploys and things like that I just want message to deal with this so that when I host fails there's no fire drill and also there's another piece of this Aurora some early marathon there's other schedulers these sit on top of mezzos and make sure that the configuration is managed so i can describe my service in a predictable way deploy it and it all works and that kind of looks like this so we have host down at the bottom I have this message layer that manages host resources and Aurora I deals with services so i can say i want a user service with 300 instances go run these I don't care where they run they just get run on the data center and so as a service owner I just focus on the top I don't have to worry about mostly details about the bottom this means that when one of these things catches on fire has is known to happen only this team has to be involved we don't have to have a fire drill so at the whole organization and in fact was a great anecdote and I think 2011 time frame we hit a kernel bug where if a host was up for 396 days system load would go up to twelve hundred percent wouldn't schedule any work on the box everything would just kind of meltdown and so we had a huge spreadsheet of up times for all of the hosts we owned and as we started getting close to 360 days we'd have to kind of schedule a fire drill and say okay this team has to move all of their software off this with a scheduled maintenance and a downtime and all this stuff and it just touched every team at the company and like waves of pain and so we this I think is what really motivated internally and sold the engineering teams on a why something like message was important because we didn't want to have to worry about the sort of stuff we want to focus on features and so once we're in this nice you know distributed orchestration world we have this problem of service discovery has anyone had to deal with a service discovery scheme of some sort and so this basically amounted initially to us putting zookeeper clients in every single service of the company and this works by at startup the user service will just announce itself and i'll create a node say hey I'm running over here if you want to send thing that user service I'm ready to take those requests same time a client is going to settle watch and say hey I want to send something to the user service where do I send this so this replaces dns and a whole bunch of other schemes that can be used to manage this when we're in this world where things can be scheduled anywhere we need a way that's kind of very dynamic and reliable to type this stuff this is all well and good I can now send a request an RPC or a thrift message or nature to be call to the user service great where we have a highly distributed you know lock store to do this super reliable right no I was on call for the service discovery for a long time so I I have some pain around this zookeeper goes down hopefully never but it will and when it will it'll hurt and so we have this big io moment zookeeper goes down and nothing sending requests no traffic in the system what do we do well first easy thing is well maybe if zookeeper goes down we don't have to just you know go out to lunch I can keep the results I had previously in the client side and continue to serve requests okay now and zookeeper goes down system still up and running we're good no we ship that feature a few months go by and we're very happy any problems here oh of course when zookeeper comes back up when it's been in an incident if all of these servers can't register in time and when we're back up this might take you know a couple minutes or even longer everything's reading empty result from zoo keeper so zookeepers up and it's saying yes I know the user service has nothing in it and everything at the company is just sending traffic know where this is probably one of the more painful adages I was ever involved in and then when we fixed it it took i would say months to get everyone to upgrade it and get the library out there because again all zookeeper clients in every application that have to be upgraded across 100 teams or whatever and so we started to think fundamentally about the api is for service discovery new tree to the moors advisory so that we only treat additions from service discovery and we treat them as something that might be available and we rely on load balancers in the client to be smarter and keep things around until we can't actually send requests there so this is a kind of novel learning we had but it took staff years I think to really get this right and so now we have a connection between our services and I have some way to send traffic between them this kind of gets us to wear for Nagel starts so finagle as you all well know is an RPC library it's asynchronous its functional strongly typed and it's pretty well battle-tested this started being developed in 2010 by team on the front end of Twitter and then within a year to every service of Twitter was using this because we were all inventing these solutions to the same problems over and over again it's a huge waste of time for most the org I was implementing load balancing connection pooling and timeouts and retries and Python somebody else is doing this in another language Nettie jetty every framework had the same sort of problems being solved and so finagle came through is the one solution that we could all kind of implement around finagle fundamentally is a session layer framework it can be used to build things like Finch which are kind of higher level concerns here but the way I think about this is it the application code your code should not be talking about how do i format a request and put it into a load balancer and worry about a bunch of the operational details it should be focused on tweets and users bike rides or cancer or whatever the nouns in your domain are down at the bottom of the stack we have all our hardware and cloud providers and the orchestration systems and really you know I think this gets us up to service discovery here where the job of the data centers to get your process running somewhere and listening on a socket job of your application is to just receive domain events and and do some things with them and do something else right send some data store returns in response and this middle layer is something we haven't really talked about too much in the past the RPC layer the session layer especially this is really where finagle comes in and while we have this nice functional programming model there's a lot of operational benefit we get here and I want to talk about some of those some of the benefits we get there this is a great quote from a former Twitter engineer Jeff Hodges it's slow is the hardest problem you'll have it's down is general easy right move things away off that box bring it back up get the connections established again it's kind of a binary thing for the most part it's slow can be a lot harder so in the CEO comes down and says well and Kim Kardashian tweets I can't see it for 20 minutes what's going on like well I see it what I don't know how do I debug this right does this work for me and this is kind of where finagle really i think shines so first and foremost before you worry about any reliability concerns we need some way to measure things we need visibility out of the system finagle is instrumented with staff so we get counters number of failed request number of successes number of retries we have histograms for especially for latency so i know a distribution of how long things take and then we have some tracing system so Twitter open sores epyon and I think 2012 this is an actual tracing diagram from Twitter in 2012 it's very complicated this is the user service down here and so we see how much connectivity there is between things the reason we can't do this just threw a spreadsheet or through some other system is because these things change all the time I may take in a new library and not even know what services it talks to and so when I'm in an incident I want to be able to go to one place and say show me with actually happening in the system right now descriptively and so what with finagle this is all instrumented so Twitter engineers don't have to make these decisions themselves they get this tracing just by using the library so once we have this kind of distributed connected system we have a whole bunch of latency type of problems to think about and so the first thing we tend to do is think about timeouts and retries timeouts being an SLA and retries being some amount of grace to recover and so typically you know we have three or four teams here they all kind of agree on what the SLA czar for the service maybe we have to serve requests in an about a second and so we pick timeouts that kind of fit that anyone see any problems here things don't compose at all timeout up top is really the only one that matters if anything timeouts and retries below the timeout up tops in a fire and just not going to matter so these things are actually really hard to do across a larger organization you might be able to get these three teams to sit together and agree on timeouts but when I have multiple pass through the system for different features with different timeouts this becomes combinatorially hard to figure out so we change our approach instead of using something simple like timeouts we move to deadlines and the way that we do this is up at the top of the system hopefully at the feature level when a request comes in we say this needs to be served by this time then as we go down the system I can simply look and say is this timed out do I actually I think I have enough time to solve this today answer this question and if I don't I can just not do the work and this becomes a much more composable thing where I don't have configuration at every level the stack to deal with this that brings us to reach runs so we typically start with some number of retries if it times out if it fails for some obvious reason let's just try it again so we can get the job done so we say maybe retries three maybe we set a back off to leave sometime between retries or some jitter so they're not a lockstep any problems with setting retries like the well what I've done is when the system is not healthy I've made an error cannon when we're failing requests i'm going to put more load on the system this is a terrible failure mode this is the way to melt your down stream's you're just never you know once it's failing you're just going to keep putting more work on it it's never going to recover so finagle took the novel approach of thinking about this in terms of budgets how much more load am I willing to put on the system below me to satisfy the request and so we now infant nails and say I want a 20-percent retry budget and that limits the upper bound of work I'll do when we're in an incident and so this gives us a nice mechanism for loadshedding I brings me to load shedding and so when we do something like a timeout right we cancel a request here there MIT there's still work going on downstream you know this they're doing database reads and maybe they're hitting many services and this is a big fan out query and that's expensive when we're in an incident and we're timing lots of things out we still have lots of excess work that we're never going to use results from and so we need a way to harvest that out of the system reap that and so this is done through a mechanism of cancellation we can if a client times out it can inform its downstream say hey I'm not going to get the result of this I don't care do whatever you want with this you can keep doing the work you can fail it I'm not I don't care what your answer and this allows us to tear down connections or a tear down pending requests and really harvest a lot of load when we're in one of these incidents similarly back pressure is I think a very interesting thing to think about so let's say we have lots of requests coming through the system but down in the database layer may we have a lock contention or something that's actually just generally slow what's going to happen the user service well if it's right on the JVM it's going to go into garbage collection and it'll never recover right we're going to have so much memory pressure coming into this thing if it's holding on to thousands of requests that I can't satisfy it'll just melt furthermore you won't be able to send it back up because it'll just crush in the load during restarts and so we need to think melt yeah and we need to think about how we can limit this and constrain this so that we actually press back up to the client so the clients can make better decisions whether they go to another data center where they relax their polling cycle and ask for less data and so we do this in a simple protocol like HTTP one we do this simply by not reading if I don't read from the socket TCP will handle the back pressure for me we just need a way to be principled about not reading from the thing until I've sent the response upstream are now you in HB 2 and MUX which are multiplex protocols we actually have explicit flow control mechanisms to acknowledge the amount of request data / or data per request to satisfy the back pressure this is again another very I think overlooked piece of what it means to have a well-conditioned system for instance finally I think one of the things finna go really shines out as load balancing and it's maybe with a sharpest tool for reliability we have in Finnegan's tool belt if you are familiar with other load balancers like engine X or h8 proxy or Hardware load balancers these tend to operate at the connection level and so they can only do things like round-robin for every connection i get i send it to a new host or i can use a queue depth one and say okay i'll send this I'll establish a connection to the host with the fewest connections this is actually a very naive load balancing strategy especially when we're dealing in request responses when I'm not dealing purely in streams so Twitter over years moved off of these naive load balancing algorithms to per request a request level load balancers like Yuma and so in Yuma does it actually looks at the latency of a request how long did it take for me to get a response to this th node and punishes flow nodes so that when a node is maybe starting up and needs to warm up we send it less traffic than we do the other nodes it doesn't get an equal amount or if it's just going to garbage collection we can ease memory pressure by not sending it as much work another load balancing algorithm called aperture is I think also pretty novel if you're talking to a large cluster of a thousand nodes and you're doing you know 10 requests a second it's going to be very hard to make smart load balancing decisions because you don't have waiting information for the vast majority the endpoints and so what aperture does it constrains the number of hosts you consider to fit your load size and so as I get more load I consider more host to load balance over and this actually leads to much better performance results for some services this example here shows us that really how much of an impact these load balancing things can make and so if we have a one second timeout and I'm talking to a tended cluster and one node of these starts to fail that timeout with a round robin or fewest connections scheme this push this into a ninety-five percent success rate in most of my uncle rotations that's been awake me up success rate and I don't like to be woken up so just by moving to a Yuma balancer we can push this over three nines which is usually asleep through the night success rate I hope and that's just by changing this one parameter and a finagle client it's a really amazing difference this is a quote from a speaker at four neocon last year philippa kado was at digital ocean and soundcloud and i think there's a lot of truth to this any sufficiently complicated microservers architecture contains an ad hoc informally specified bug-ridden slow implementation of finagle so I know you're all scala people so yeah just rewrite everything finagle you get all this stuff and you can sleep through the night well unfortunately we also have some things that aren't written skull in the world or maybe we don't have finagle as a you know an option because it has a bunch of programming implications well I've been working on this project called linker d for the last year or so it's open source and on github and the idea here is this beat is a proxy version of finagle so it's written finagle but it exposes just ports you send a request into these ports and we deal with a lot of this complicated management layer the management of requests so back pressure timeouts retries all these things it's very pluggable protocol supports pluggable by default HTTP and thrifter there we've just added HTV to support in the last couple weeks we support a number of discovery and orchestration seconds and anything you want to be pluggable probably can be applicable in it and it just requires usually writing about 100 lines of scholars so why are we called a linker well the idea the linker is not a new one we have abstract names and we want to bind them to more concrete names usually addresses to send requests to and so we can take this approach to services get renaming in a complicated system you may have let's say I own the user service when a timeline service wants to send to the user service which version do they send to or I on the user service and I want to deploy a new version how do I get them to test against my version this can be very complicated in a large organization you maybe have a staging data center and everyone tries to deploy their first but nobody really wants to test their because it's not real production it doesn't get real traffic and it's actually hard to maintain a fake copy of your ecosystem the other option of course is to just deploy to prod and pray which is we did a lot of that too and there's actually a very nice happy middle of via finagles naming system and so what this allows us to do is to deploy a new version of the service and incrementally move traffic onto it separately from the deployment phase this ideological naming comes in where every client in linkwood ease case every request is given some logical name that represents its destination so for instance the user service what we really want it to do is route to some server set some concrete cluster so you know look in zookeeper in the prod namespace and the user service and resolve that or maybe the staging one and we have this language for expressing rewrites so I say when you're looking at an S prefix make it a zookeeper prod look up and I can add overrides here that change that behavior in a very dynamic way this can be used for things like staging in fact one of the last things I ship the Twitter was a staging infrastructure like this and so I can go into my browser and I'll demo this if I have time at the end and just add a header on to request and say well when you're going to talk to the be serviced use the b2 service instead this actually gives us kind of a d'etre style call intercept interface at the data center level instead of in the process similarly to this kind of redirect staging use case I can also do proxies so let's say I want to have a proxy that injects some application level failure that returns a different data structure when certain things happen or just injects latency or something like that I can use the same rewrite rules to say well when you would go to the e service go to the P service first and then go to the e service and this allows us again to do these kind of call intercepts tile things and filters to do either quick prototyping or failure injection or a number of use cases and this brings us to what we kind of considered to be a service mesh that we deploy linker d in a maybe / host mode we have applications talk to regulate a linker d and so we have basically finagle as an outer process networking stack and these can communicate they can do protocol upgrades they can do Transport Security they can manage back pressure and cancellation and acting etc etc and we use another system called named ready to front service discovery which manages most of these rewrites have been discussing and with that I'd like to do a little bit of a demo so anyone familiar with job job is not very smart or classy and he wrote a very named program the show arrested development and so natural i took this program and i made it a micro service that runs in Google compute engine net at a large scale as you do and so the way this works is we have a web front-end these are little go programs I have a word database which is basically dictionary of really dumb words the web service calls that gets a result gives it to the gen service and the gen service just repeats it forever or up to some limit I'm pretty dumb what we're going to do is we're going to deploy a new version of this or we actually of both of them and we're going to use name ready to manage how that rollout goes use this linking and naming system to control and do an incremental control deploy I also will remind you that this is all open source and on github you can go through this number yourself and about 10 minutes f afterwards so let me try to set up here me okay so here is my service like I said it's pretty dumb and here's link your knees bet dashboard so this is the admin interface on one link ready process on the web front end and we see here is that there's a these two clients link rd configuration really doesn't have anything specifically about that i'll show you link reduce config basically has addy tab or policy that looks like this which says when you see an 81 one request look up the hostname as a service in Coober Nettie's i'm using a cooper native scheduler here so that there's nothing application configuration wise about link ready and we're sending a bunch of load through the system i can curl it and it all looks great and so i can even excellent so what I want to do is now I want to figure out I want to stage my new services I have these two bro tak versions of the service and we're going to try to send requests through one of them at a time so i will i'm just going to look up the yeah and so this what this request says is I've added a header on to my request that says when you would use the gen service please use the gen growth hack service instead if it exists and I immediately get a different result right so this is now if i go to just my normal old prod thing i get the old result when i add this one header i'm talking to a new service downstream this is propagated with a request and i can do staging in this way so now let's put some loot from consistent requests we see that we're getting different words back and i want to go i'm going to change the policy here and so by default we're just doing everything there and what i'm going to do is say send ten percent of traffic to each of the new services and ninety percent of the traffic old versions of the service so you know wait with 129 and I'm going to update so this is a I push an update into named Rudy and Link OD immediately picks up the changes and so we see here we see there that now to about ten percent of the load is going out of new version of the service and we see that we've just automatically detected new clients I have success rate information latency information etc we do see this weird thing here maybe that this new version of the service isn't so healthy I may have written a bug into it but we're going to pretend we didn't see that for a moment ignorance is bliss and we now see get in this thing that we actually get different results back now i'm getting about one at every 10 is going to the gen growth hack service and i get that very different result okay now what we can do group let's go we're just going to go to production we're ready for it we didn't see the error and what we're going to say here is use the growth hack versions of the service by default as long as they're there if they don't exist for some reason this or becomes a fall back to the old version of the service i push the update and i get the new results and we see here that traffic shifted over one interesting thing here is that we see higher load on this so here we see there even right here i see one is doing more requests why might that be I'll give you another hint here we see about eighty percent success rate on this client yet at the high level we're doing a hundred percent success rate where is this failure going well I gave you kind of a hit before we're doing retries and so we can look we know these are get requests because they're HTTP and I can look at the response code and now it's at 500 and I can automatically try those and so here this is just using retry budgets and happens to be that are we have a retry budget of twenty percent and we're failing twenty percent of the requests so these just line up perfectly and we're able to retry all requests and improve success rate finally we've we've acknowledged this the ops team is freaking out and saying rude the service isn't going well what are we going to do so we're just going to delete it I don't want to push the detail rules it's going to tear it out of production we tear it out and traffic just falls back onto that new service immediately we're there we go so liquidy is about you know we've been open source since februari it's still a pretty young project even though it's we're standing on the shoulders of giants we're using finagle and Eddie to do a lot of the heavy lifting here so we've just got HTTP two in I'm really excited about it it's going to open up a whole bunch of operational features for us especially grp see we're looking at all sorts of off features TLS client certs I talked about deadlines we're not even forcing them yet we're just propagating with requests but not acting on them we think we're in a really good place to do dark traffic so I can just send a request to the new version of the service without return the results the client and we can give you a little bit of a preview of what it what it might be like to deploy and generally linker d is just a configuration layer over for Nagle it's so that you don't have to write a new service in Scala to get all this value you can just plug this in your application with a little bit configuration and get the same value and so there's a lot of stuff and finagle that we haven't been able to expose configuration wise and we're looking for your help to figure out what the right ap is and user interfaces are for exposing this configuration so with that I'm going to ask you to go to link Hridaya linker di oh it's a very pretty website I didn't make it we have a slack where we're always happy to entertain your questions we have a github page where we love people to click on the star it makes us feel really good and you can follow me on Twitter I'm trying not to use it this week you can follow link ready and with that I think we have a couple minutes for questions [Applause] I'll give you a sticker if you ask a question what is dark traffic is the first question so dark traffic in my mind means i can send requests to new version of the service and the results of those don't go to the client so i still send two versions of the requests one to the new version of service one of the old and we can just compare the results see the latency differences to the differences results without impacting clients at all and it's a nice way to get confidence of a new version of a service before you go to production that's a great question um hopefully it's very incremental the idea here should be that you can do this in a one service at a time thing or just in some subset of the service and change the configuration of the client to instead do services cover your talk to DNS directly send its request to the local host a linker d this can typically be done by setting an HTTP proxy environment that sends requests to the local host but there's a link to die oh there's a whole number of examples on how to set aside all right and I'm being given the the crook I'm being yanked off stage I really appreciate it thanks you