SBTB FinagleCon 2015: Oliver Gould, Buoyant Service Discovery
Recording: SBTB FinagleCon 2015: Oliver Gould, Buoyant Service Discovery
okay how's everybody doing how are you I know running late today and we're all kind punchy so yeah I'm I was a Twitter for my name's Oliver first of all hi I was a Twitter for about four and a half years starting to the 10 and recently about six months ago left to go start a company called buoyant recently left to start a company called buoyant and so it's both an adjective and a noun about that so this talk is about kind of my history dealing with services cavorite twitter i worked in and around that problem space very closely when i started i was the lead for observability i'm so the monitoring is like when you saw graphs and the diff etalk that's the stuff that i worked on and then I went into traffic space so kind of very close to the final team working on operationalizing the names and services Gabry and so i got a kind of front row seat to most of our services Gabry outages over the past five years and let me tell you there are many of them and we're gonna so the talk is about that story and kind of set the context for the things Marius talked about this morning and if I talk to quickly please slow me down so yeah it's a story and we'll tie it all back to finagle at the end but it's really not about finagle that much so we've heard this story a few times here about decomposing model is right when I joined in 2010 even before that Twitter had kind of acknowledged that they had this fail well problem it was very real and if you use twitter in 2010 you definitely saw it that's what convinced me to come join because they had problems to solve and we kind of you know recognized already that we need to get out of our monomyth problem and move to a more nimble what now is called micro service architecture at the time was like here's some Scala kids go launch this somewhere else and so that's really what the story is about obviously when you go to that world you now have communication and a whole bunch of places you didn't before and so you have to do our PC bit of trivia this is the original finagle logo nobody really is seen that before I think um not not everything around it just this thing in the middle and so the service discovery is obviously you've heard about this a few times today but it's kind of the predicate step to do RPC I needs to send my request somewhere right so where do I send them and initially we had not really identified what service discoveries for we had this place to dump host information and so we used it for process numeration we expected to be very consistent and authoritative and over time we really wanted to change as requirements and that's what this talk is going to be about and as you've seen it's kind of a fundamental underlying part of finagle you don't have to use it you can give it finagle a socket address and it will happily connect to that but to do this really in production you want a more dynamic flexible system under there so when I joined there were a few ways to do service covery the most common one was like a huge text file with all the host names and hopefully ports for services and then so like you know if I need to talk the user service I go ask the user service hey which webhost you on and I copy and paste that list and then somebody else finds the file in my directory and they copy and paste it and pretty quickly that becomes a big configuration management problem and we didn't really like that we all said this machine database was great about telling us who what teams on what machines but just because i own a machine doesn't mean i wanted to serve traffic or any of those details so that really wasn't a good solution either and there are some people old DNS with all readiness in a second and there's some people who would kind of come out of google big surprise you're like oh with this there the better way to do this we're gonna use chubby chubby is not open source so we're going to zookeeper and so people had started playing with that the quick note on DNS it's great because it's everywhere right everything speaks DNS to some degree however they all do it a little differently they all kind of lament caching differently and especially Java if you've ever tried to like deal with DNS names and job and production you've encountered these flags and they are a headache like a real big headache raise your hand if you don't think they're heading great and then we want a more live option right we want to be able to get updates about changes kind of quickly and not wait on DNS TTLs to propagate to the system furthermore DNS is really a layer 3 solution it doesn't really address layer for you know we don't really have port information there serve records are there they do cover this client support is not great especially in the JVM again and so we you know we could go down the route of like implementing a bunch of DNS protocol stuff but we were kind of hipsters so use a keeper and that kind of started looking like this so I have a service that creates an ephemeral node in zookeeper it's shove some you know well originally thrift but eventually JSON blob in there that's just like hey here's the stuff I'm serving here are the ports I'm serving on etc and then some other services just watching that directory and saying hey give me all the nodes in there and when that works well we get our PC boom and yeah and that's kind of what we had starting out so about the time I joined there was one general purpose you keeper cluster everybody got to play in it everyone just chose their own directories and shove Jason in there thrift in there there are no ackles it was kind of like hey our engineers are good we don't make mistakes like that and there weren't that many service that how to use up most of them again we're using static host lists and configuration files but still some very wise people actually the people who work on our Orin missus started like they realized that this would be a thing that would have to exist and they started to work on a server set library which is still open source but is not the one that's used in finagle by default anymore that looks something like this instrument your code so to register on the registration side I would copy and paste this bunch of Java I you know creative zookeeper client so no everything has to create a zookeeper client and then I just register my endpoint in there so we see some paths and some time outs and all that stuff okay great now we're caught up to when i joined twitter so 2010 to about the time went public we were really fighting scaling issues in production every day eventually that he declared they killed the fail whale good but it was a real learning experience and to being on call on that time was really formative for a lot of us so a big part of that are these things so that story really wouldn't have happened as well as it did it had not been for things like her own messes happening that they really enabled us to not have to go twist somebody's arm to get hardware which is a big problem at Twitter for long time and so now I can automatically provision hardware great finagles also be poured out story one turn this world you really need an RPC layer that deals with these problems for you and compensates for the complexity so service discovery then no more static hostess nowhere Deena suzuki / oh my god zookeeper this is not a talk about two keeper by the way it's just whatever that services carry back in it is that things tough to maintain in that world we have now a zookeeper cluster per zone per region and a whole bunch of crosses talk to them in front ends discover these things via zookeeper so now we aren't a special very special purpose cluster with observer set up in a you know read-only replicas and all sorts of policy just to solve scaling zookeeper for service discovery we have other zookeeper clusters to deal with other things so if you and we have a lot more services talking to it all right Marius is no longer here but I'm gonna make fun of anyway so we're going to figure out some horror stories I'm with that so my favorite one a manager thought he was talking to his local zoo keeper on his laptop he had a tunnel open BAM all production got removed all the notes that was a subzero that's the worst one in some cases zookeeper goes down Yung she talked before about Pinterest you know zookeeper like the leaders get all effed up things can't right and now we're dealing with stale data or things go really badly and everything this goes down and all the clients losers state they're flapping up and down everything's crazy so in finna go we introduced a lot of caching logic if you look at the service at code nickel server sets it's like all right as long as I have data I won't change it right so if the I lose my connection I don't drop things out it didn't do that initially initially it just dropped out you're screwed so we fixed that with caching but then what happens when it comes back up and it's empty now i have like everything's like hey it's telling me it's empty and you're waiting for everything to stand back up and that can take some time when everything is hammering this the kind of lesson here is that zookeeper clients are very hard to get right and we weren't only dealing with a JVM we're dealing with python we're dealing with Ruby and we were dealing with multiple implementations in each language to make things better who here has tried to use zookeeper at all before like writing a client great do you know what this means this is a state diagram for the session for like zookeepers client session this is a lot of stuff you have to manage to write a zookeeper client it's very hard to get right I've written one and I'm an idiot for doing so but it all of these kind of session expired connection lost cases are really hard really really hard to get right and so we want to minimize that as somebody who is responsible for maintaining that library we have a lot of stuff in people's code that does this right now it's not like just something I can go to ploy and fix I have to go to every team which is again hundreds to thousands of services and say hey can you please upgrade i fix my bug in the zookeeper client well next time you deploy you won't have an adage more than that we have a whole bunch configuration right I have timeouts that deal with liveness I have host names I have paths in my code and so we end up with a whole bunch of configuration just to manage different environments staging in this zone prada map zone all these things end up with different settings and so we are investing a lot of time in like configs which we really wanted to get away from and that's a big thanks to Marius so as somebody who is on the team supporting this stuff in a service-oriented architecture I want to run serve as ask every just like any other team I don't want this thick client that's in every service that I have to go like literally take months to upgrade like I I don't know how many worked in a big company but in big companies things don't move quickly and as they grow especially right there are other priorities so we want to figure out how do we get out of the burden of upgrading other people's code and so I don't know what the future is but I want to kind of talk about some of my lessons here right like I i'm not going to prescribe here's how you do services kaveri i want to tell you how we're thinking about it though it's pretty good huh so really with the first big thing i want to decouple my application from the back and then i don't care if it's sooo keeper or Etsy dior Redis or my seek or whatever like I want a layer in between this so that i can change things and fix things and manage that big ugly hairy state machine on the client side in one place instead of in thousands of places i want to team to be able to own that and they own reliability and they can put cashing in place so they can do sharding her they can do any of these things in one place without doing them thousands of places and I don't want have to scale my back end to scale my service right like if I run twitter is the Nisshin on my sequel if Twitter it'll only try to scale my sequel they never would have succeeded right I want to play the same game in services Gowri so we were here remember where we have these things kind of very tightly coupled with the back end with zookeeper and I want to move to a model that's more like this and this is what we're working on a buoyant right now and I'd love to open source once we've used it in a little bit more anger I'm not going to give it to you until I've been really mad at it and so the first step and this is something Twitter's already done it's a split registration outside of the process right let me put that in a little like command line client that just like hey I'm going to announce something right it's not tight at aliveness of the service which is something we want we want something deserve registers and so Twitter this works directly on zoo keeper but what I really want is a very constrained API there so I actually have a service where i can manage this i can change out the back end without touching any of this stuff and this is again somebody can be on call for this thing instead of an on call for n thousands of these things similarly on the read side and this goes back to what Morris is talking about today I don't want to say give me the zookeeper note i want to say i want to send requests to the user service what's a user service look like and let a name server give me name trees you know these thing things that are kind of these logical names having a process to resolve logical names these complete clusters boom now we can do our pc so the registration API very constrained again hide the back in semantics and the one we've implemented is really simple no Harry state machine on the registration side I don't need like a firmer ality or like liveness or any of these session heartbeats that zookeeper does I want to say hey I'm alive here and their service is great tell me again and that's like a very if this was on the Reid side to be very similar her DNS works but this is kind of how I want to do registration is very flexible on the on the right side ok now to talk about the read side and finagle finally ok so going back to it Mars to talk man i'm so pleased that he gave an intro to this because it's been hard to rush through we have a logical name like a user service and i have a whole bunch of concrete names and in this case I really want concrete names are like this deploy of the service these things should basically be immutable I can scale them up I can scale up my replicas up and down but i want to say here's my new version of this code and weather gets trafficker not as a routing decision not a deployment decision this really freezes up in a lot of ways so to go on a name we've seen this earlier today we basically just create our client like that I want to talk to you service I don't know if it's a staging service abroad service or in Europe or in Sacramento or on my laptop I just express this logical name and this is actually maps to this this is just a nice alias for using names Marius is talk definitely explain that better similarly we have some very simple delegation rules i can say when I talk about service I actually mean this environment right and I can layer these the way it's mauri said earlier so in this case service users becomes this path which is probably in zoo keeper or whatever take a minute into the stack client so finagles new finagle six clients have this very nice stacking module that you let you replace features I really encourage an affinity with clients work that's the place to look and then I want to highlight just three kind of big pieces of it there's many more than I'm going to explain today The Binding factory is responsible for taking this logical name and the client and actually instantiating like resolving this to a concrete name to address is actually and binding to that and so this uses VARs to manage the bottom half of the stack and the bottom half of the stack is rebuilt as these names change and the top half of the stack is not load balancing factory so this and builds a name tree the kind of topology to think about how we're out load balancing factory sits on top of that and chooses which requests go down which path 'try and the endpoint stack is rebuilt for each individual kind of address that gets put on there and so to just briefly show what we can do with this we can use these detail right sand this is a simple header that HTTP that pinnacle HTTP supports today and I can just add this rewrite rule on my client and so instead of going to the fried user service I now go to my user service easy right and this is the cool one we talked about this before it or hack big project I can also do crazy proxying rewrites so I have now launched little process that just takes some residual path like that strips off some number of milliseconds from it and then connects to the remote alright and so here I now have a client that I just rejected latency purely through detail rules this is a very very powerful mechanism I don't have time to really go and do it today but i really encourage you to look into it and i really want to thank this is not just me working on this by myself or just to CSL team working on this by ourselves there's a lot of people at twitter and i'm sure i left some out this is a big team effort over five years to do this stuff and I really want to thank will in particular who made all these awesome images and I stole them finally a point where we're building a lot of this stuff taking these lessons we learned at Twitter and trying to build some products around this Andrea Lisa bunch of this in the open source after we've again used in anger so definitely you know follow us or however you want to stay in touch I would love to keep in touch with you over time and if there are any questions I'm happy to feel them if you're all tired again this one okay so real briefly um so this naming mechanism and I totally lost over this so you're welcome to be confused it actually supports residual path so every node doesn't have to resolve the full name there may be some left over that says like hey I resolved a user service right and I have this extra bit that I didn't know how to resolve to a name and so we end up with his yellow bit that's just sitting on the request and this gets propagated through MUX I've written some ways to propagate this reach to pee that's not really integrated in finagle yet but you can in your protocol or in your client stack have these residuals they get Soudan to this service now gets this thing kind of just sitting either as a header or your eye or something and now it can interpret that and so what this does is just like take the rest of this name the first thing is the number of milliseconds the rest of its my destination and so you could have a generic proxy that's not bound just to the user service that can talk to any proxy and just by rewriting these rules I can cause it to wrap through that so this becomes a very powerful debugging tool you could use you can do DTrace with this you can do you know your distributed prints through this this becomes a really powerful tool the other thing to point out is that this propagates for the quest context so this is not like my request is going into the API here and hop later is being affected and anywhere that talks about the user service and the full request graph even this is like very deep will take these rewrite rules and works in my proxy it's a very powerful mechanism as Marius of the day is how staging a Twitter works now to question first one is dropped yeah because you don't want to live yes yo que ya yes you're right I did just drop that okay so what we want we leave kind of abandoned health checks and Twitter's of bandit health checks because they're not a good indicator of liveness right either I'm like there is some latency in between it reporting itself and what we found at Twitter is that most of our health check endpoints didn't actually indicate health of the service the service if it's sick can't tell you it's sick generally and they don't actually exercise the request pads so what we really like I'm going to hit some health check on points like yeah the process is up but doesn't tell me if they're like request path it's actually go through the code is able to serve if the caches are warm of all the things that I need is a request so we want finagles load balance here to make those decisions work and actually you waiting decisions and if you tune it properly with us at lowes or were three tries you can actually achieve this without having health checks and the early that's the goal this hasn't been totally proven out yet that's kind of the aspiration there the other issue we saw is that when this thing garbage collects if the if the zookeeper session timeouts are small enough you start really flapping up and down and she's like new session new session new session and that session creation is actually very cleansing zookeeper and so that was another big motivator for moving this out does that clarify that so much when you have something for you to watch yes so on this on his name server we need a watching API right on the street side that has to be how it's been implemented so far as long polling which is I think I find me to do this if you turn your server properly but yeah you do need some watch mechanism absolutely is there another question are we good oh yeah the name server has to like just like zookeeper has to live somewhere predictably right so like some people say oh just use dynamic dns which is maybe an option but zookeeper exists on some well-known how sport and this has to as well we could for instance live on the zookeeper servers himself all right thank you so much