SBTB FingleCon 2015: RPC Redux

Recording: SBTB FingleCon 2015: RPC Redux

excellent all right so let's get started first off Travis was being a little bit generous in this characterization of my involvement here I certainly was there in the beginning but finagle is a product of the effort of very very many people and at this point i think i'm more of a sort of spiritual figure than you know anything else but but thank you guys all for coming here today it's kind of accelerating to see that finagle is being used in so many different ways and i'm very much looking forward to you know seeing seeing the talks today and and get some more details about that and thanks again to Travis and Chris and everybody else who helped organize the conference despite the little little hiccup we had I think it'll be smooth sailing from here so we have now been using finagle in production for nearly four years the code base is actually almost six years old at this point in time Nick made the first commit and over those four years we've had a sort of the opportunity to learn a lot from what it means to run and maintain an RPC system in the service oriented architecture and interestingly a little bit of trivia the first production use of finagle was actually a java application which was Twitter's web crawler the first one we had I think and this was a very interesting Java code ragavendra who do this work who is now at Pinterest sort of took the took the model to heart and and there's a lot of Java code with a lot of flat maps and all sort of different things but I worked out really well and so it's kind of interesting to see that that the sort of Java aspect of or multi-language aspects of finagle took hold very early on so the other interesting thing is that while finagle of course today is at the core of basically every service at Twitter the open source community I you guys and others has really extended our reach beyond what we could have imagined to do by ourselves and I think this conference is really a testament to that it's really interesting and accelerating again to see to see it being used in different settings so the the sort of core model of finagle has really remained very constant I think by october two thousand ten you know futures and filters and services were all there and they remain today really to to form the kind of orthogonal basis for programming with with the system would finagle and i think it's interesting to sort of examine how why why it has endured way it has and I think that really boils down to three different things so one is simplicity the second one is composability and the third being separation of good separation of concerns so so first of all all of the abstractions that that finagle offers is really something that's been reduced to its very essence a future is really I think just about the simplest way you can understand a sort of concurrent computation and the service is really not much more than a function that returns the future and filters are really not not much more than a standard way of decorating those you know functions with different behavior and the result of that is that it's almost always kind of obvious where functionality belongs there are ready answers and clear answers to what belongs where and I think that's evidence for this being chi minh orthogonal basis of this kind of computation the other really really important thing is that all of these data structures compose and very important in interesting ways so first of all as we all know futures compose with other futures iva concurrently or serially to form sort of complex data flow graphs that are very simple to express and have very interest very good and predictable behavior and services and filters also combined in different ways very much like functions do you can stack them you can pipeline them and so on and so forth and the sort of key part of that of composite compositionality is that it lets us build fairly complicated structures from very simple parts and that has a lot of advantages we can reason about these individual part in isolation we can test them we can develop them we can reuse them elsewhere right and this has turned out to be very very important property of this way of programming and finally and I think this is a really really important thing and I think without this you couldn't have the others which is that I believe most of the abstractions in finagle itself promote really good separation of concerns and so if you look at something like futures for example we're kind of decoupling how code is structured from the sort of concurrent aspects of of computation and what that means is that you're sort of encouraged to structure your computation as effectively a data flow graph and that allows us to kind of divorce the semantics of computation with all the minutia and details of you know spinning up threads to deal with and currency and so on and so forth and it gives the underlying system i fin I finagle and Twitter you tell a great deal of freedom to do interesting things with you know scheduling those things for example or otherwise and I think it's kind of interesting that that now oh and and and I think another important thing about this by the way is that I don't think it would be possible to have the kind of compass compass internet compositionality of services and filters without also having futures to kind of be couple the concerns of concurrency from from semantics and so in some ways they're they're very much tied together and I think at this point and in sort of twenty-twenty hindsight this design is almost inevitable right it's hard to find a sort of simpler decomposition of this kind of problem i believe if you have found one please let me know but but i certainly haven't and i think that's kind of important aspect of this as well and so I think this model has held up really really well so now for those sort of the bad news I guess which is which is the following so i just presented to you very briefly this kind of you know beautiful Platonic ideal of a structured these systems together and glued them and glue them together and so on but this quickly breaks down in practice so i'll call this kind of style of programming your service a function simply because i published a paper with that name and it kind of adopted it as a sort of umbrella a name for this kind of style of programming so again what I what I just subscribe to you sounds very very simple and in in theory and in fact most code using the style of programming is very very simple but when you start to deploy finagle in particular you you quickly run into some limitations and the the sort of most egregious of which is that in order to use finagle successfully today you're more or less have to be an expert at its internals right and so we don't have a really good way of separating the kind of semantics that we really need from how the internals are implemented and I I think I'd like to illustrate this with using two different examples so the first one being better creating resilient resilient systems so I really like this this definition of us in is by the way which is actually from a a book about structural engineering and a structural engineering kind of emphasizes this notion of strain and so I kind of adopted it I've adapted as the software systems as well resetting software is software that's able to withstand strain in the form of failure overload and varying operating conditions ie programming and in a modern environment and the reason why resilience is such an important property today is that we're really optic truly operating with the crappiest computers of all time which is which is our data centers right and our data centers are really really heinous and complex and that in and of itself is it's clearly the enemy of robustness but they're also really unreliable and prone to operator error however they're only datos there are only computers we know how to build cheaply on that scale and so we're kind of stuck with them and so it's up to us to compensate for the robustness that we forfeited in hardware and our software instead and the interesting thing is an interesting thing about resilience as a property is that it's really assistance property it's not enough that a single piece of software in and of itself has some modicum of resilience it's really the system that has to be resilient because that's what matters at the end of the day it doesn't matter if you know your local service performed beautifully well the whole site was was melting down right and in particular this kind of requires the adoption of the kind of systems thinking to solve in an acceptable way so why is this important for finagle well finagle kind of has an exalted position her because it I like to say that it occupies the boundary between independently failing you know servers or you know mutually hostile States or whatever like the nogales is that buffer basically and Panay go provides a lot of the important tools that you can use to create resilience systems and so I'm these are things like timeouts failure detection load balancing queuing policies and so on and so forth now the trick here is that in order to create a risk truly resilient system using to negril you have to piece all of these things together just so right like there you have to understand how they all fit together how they all interact how to parameterize them in order to successfully use finagle to create a resilient system so just to walk through a bit of an example you can imagine that if you have a system today first of all you have to tune its timeouts both on the server side as well as the client side you might use client timeouts in order to sort of maximize your chances of success while also not getting bogged down by a slower unresponsive servers and so here already we see kind of contending needs we have kind of service level agreements needs as well as liveness needs right and we use timeouts as a tool to address both of those and at the same time you want to make sure that when you do encounter a failing or a slow server that you have room to retry those operations where you can write and so already we see that a simple seaming knob like timeouts a configuration like timeouts are ready deals with a lot of contending needs the same is true of tuning retries so you want to make sure that you retries are set so that you have basically a good chance of retrying failed operations when they do but at the same time you don't want to effectively denial-of-service attack yourself by effectively getting into traffic amplification or something like that which can occur with retries as well this very much shows up and things like connection pools and queue sizes you want to make sure that your inbound queue sizes are not so large that you want just infinitely except work that you're unable to perform but not so small that you limit the throughput of your service right and what we end up doing in practice to tune all these different things is basically to load test them using tools like Iago for example which is another open source project from Twitter that allows you to generate traffic it was kind of designed for this purpose and from your sort of rough estimation of how your system behaves under these load tests you you're more or less pick out of a hat you know what your parameters are and and kind of hope hope for the best and I don't think that's necessarily a sustainable strategy and and finally also another really important thing another important tool that finagle gets you in order to create so that you can create whistling systems is the ability to do kind of failure classification and handling so if you have a system you need to know whether or not you can safely retry a particular failure for example so this is not really to knock on this method in fact we have many many remarkably resilient systems created in just as way and in fact Twitter today is full of them and I believe that at this point in time the sort of Twitter infrastructure is a fairly resilient one and this is how we accomplished it right but it's a lot of work to do and also a lot of upkeep your operating environment changes constantly you might have new clients that talk to you you might have to talk to new servers maybe you enter a new data center which is slightly different numbers apology and different latency profiles and things like that and in order to really maintain this kind of resiliency along along the way you really have to remain vigilant and it's a lot of manual labor and a lot of you sort of upkeep and load testing and and and and and doing sort of implementing good software development practices in order to to to create a truly resilient system now another sort of important aspect of of the wave finagle is tuned today with all these different parameters or the tools that you have available to you is that they're all intertwined and I hint it a little bit out this earlier where you know ty masz for example are used for lots of different purposes but the same is true between parameters as well so for example maybe you want to you know maybe you want to increase the amount of concurrency that your request concurrence in your server doing so almost invariably has sort of downstream effects and all the other parameterization that you have because you tuned your system to handle a particular level of concurrency and now you're changed out for whatever reason you have to sort of reconsider everything else that you do so that's that's one problem with with the style of system I believe and we do have an answer for it but it's not a very satisfying one right it's clearly kind of a breakdown of this model we have to sort of really clean an orthogonal model to work with actual application code but in practice we have to do a lot to make it work in the kind of setting that we want to another big another really big problem using systems like this and by this I mean service service-oriented architectures is that your forfeit basically all the flexibility you have and more monolithic applications and so this is a very rough not representative cleaned up outline of sort of how Twitter the Twitter infrastructure stack looks like and so on the I'm the on your left here we have the front-end web servers which friends a bunch of front-end you know basically HTTP front ends which talk to some middle terror of effectively domain logic and servers which then talks to a source layer so if you imagine all this logic is now spread out all over the place and executing independently all over your data center whereas before you had it all under one code base and in the process of doing that you forfeit a lot of the flexibility you had in the monolithic code base so for example consider just production testing and you and use a new service so maybe you have a new version of the user service that you want to make sure it doesn't break you know in practice so the way you would do it on a monolithic application is that while you you check out your get branch or whatever you make the modification and you maybe deploy it to a few servers and that's it right it's all all under the same code base and so you can do that very easily in a service architecture everything is much more complicated so in order to do that you would have to deploy a few instances of the particular service that you want to to test and roll out and not only that but in order to actually understand how it behaves under whatever you know load you want to give it you have to orchestrate every service above it to sort of rear out or redirect traffic to to that particular service and that's just one very simple example of how distribution complicates things there are numerous others and I think it's pretty well recognized that when you distribute your your application your forfeit a lot of this sort of simplicity of of working with this code base in practice and again this is this kind of monolith versus service-oriented architecture discussion and this by the way extends not only two things like you know testing a particular service but imagine all the different tools you have available to you in a monolithic application first of all you can sort of print up anywhere you like right that's actually useful really really useful thing to do well in a service architecture you can't really do that in order to do that you would have to add your print up to service deploy it someplace hope that you know the right traffic lands there and look at his logs or something but it becomes very very difficult you the kind of impression I'm trying to give is that basically everything becomes more difficult in this kind of setting and print up is just one of the one of the examples serious things like you know debuggers DTrace a lot of these different things that you kind of forfeit so maybe it's worthwhile to think of how you might account for these kinds of the deficits effectively in an RPC system so so far both in this talk and and finagle generally we've been kind of operating with a very simple model of what our PC means namely that the role of an RPC system is roughly just to link up two different components and figure out the dispatch a you know a function remotely that's more or less it right and everything else in between of course is a lot of complicated stuff that happens but that's that's the kind of model that were we're we're kind of adapting and you can kind of reduce that to say wealth in a goal is to either export a service implementation from the server or else import a service implementation from some other servers right that's roughly what the role that vinegar plays right and that kind of assumes this very simple model and again of course finagle does a lot to accomplish that there's everything from you know load balancing connection pooling you know liveness checking and everything like that and I guess what one one other thing that the surf mentioned is that the model of finagle is actually one of kind of replica sets and so it's not just a single instance it's a pool of identical instances that you're you're dispatching to but the that distinction is not not super important for the purposes of this talk so in many ways that model is really the least we can get away with from an RPC from common RPC perspective and it is the sort of traditional interpretation of what our PC means but it also doesn't solve these kinds of problems that I've outlined right in fact it's complicit and creating them because it's the it's the thing that allows us to distribute stuff and now we have distribution and now our world is painful basically and so I think it's worth thinking about how we might serve account for these sorts of things and in the Indus model so first of all I think it's also important to address the question why you know why we need to tackle this in an RPC layer in the first place there are lots of other different ways of accounting for these kinds of problems so for example maybe you introduce software load balancers that can route traffic in particular you know specialized ways that allow you a great deal of flexibility and how how that's done or maybe you have specialized front ends that I like to stage certain certain servers and make use of them and easily kind of integrate yourself into that kind of development cycle I think that approach is very traditional and it's it's it's being used successfully but I also think it's has several problems first of all they're all very narrow in scope right and so I might deploy a sort of staging system that is specialized from my particular infrastructure for you know maybe front-end web traffic and that's it if I need to do something that's outside the scope of what it what that is intended for I'm out of luck another thing is that these kinds of infrastructures tend to operate below the session layer in the kind of OSI model and what that means is that they don't have access to a lot of the kind of semantic goodness that you want to to to do a lot of smart things so for example I don't have access to what my you know framing looks like or what the timeout for a given request is or what whether or not a request is route rible or not because I'm just dealing with bite Supremes basically most of the time and then finally they also just constitute more infrastructure you add more more infrastructures to some more problems but does also more complexity and more things that can fail and so on and so forth and so I think it's worthwhile thinking about how we might instead account for these kinds of problems in the RPC system itself so the way I've been thinking about it is that I think it breaks down into three main components which is one to have a component model and I'll get into what these mean in just a second so one is to have a color component model the second is well call a linker and then finally an interaction model and this is actually a kind of model that we've been working to move finagle towards and I'll you'll see shortly how how that plays out and but first I'll explain I'll explain myself so what do I mean by component model while it's basically your server as a function in this case it's kind of how you structure components and how they fit together and how you you know how they compose and so on and so forth and I think finagles basically works and again I think it's stood the test of time in many many different ways there is one sort of common deficit however which is it's very difficult to interpret failures from finna go and so a future can carry any throwable and a throw will in and of itself doesn't mean very much right so as a simple example if I if I see a failure from a finial service I don't know whether it's safe to retry it you know without having without knowing more about the actual implementation and again it's about sort of coupling or we're trying to decouple the implementation from the the semantics of the API and so toward that end we we recently introduced a failure class which kind of bridges that gap which is a failure is also a kind of throwable but it has sort of semantic provisions for telling what an error is actually about and what you can do with it right and so in particular it has a set of flags and these flags allow you to represent things like well this failure is retrieval or this failure happen because the operation was interrupted or so on and so forth and because we're not how a common way of talking about the semantics of a failure we can suddenly do more more general things so as an example you could have a retry filter I just test whether or not the failures tribal and just blindly retries any retry bowl failure and again this helps us disorder the couple the semantics of a failure from the from the underlying implementation and so that's really a small modification that we've done but I think actually one that is very important and that's really I think all the changes that we have made to the sort of core component model here and now I think for more exciting part so usually the way the naval clients are constructed is that you give it either a static host list so you know this set of hosts or else maybe use the server set so server set is a implementation effectively dynamic host list using zookeeper right and I know many use that we use that a Twitter very heavily and server sets are really really useful they allow you to maintain these dynamic host lists that come and go when aren't you a service become available or go away and vinegar will load balance over you know effectively the current snapshot of that host list but they're also very limited in the sense that they really are just a host list and that's it right there are there really just a dynamic version of the Socialists the static kosis that we we have talked about so one thing that we have recently introduced is the ability to do symbolic naming and finagle instead and what that means is that instead of using these sort of concrete notions of destinations with targets like hostess static or dynamic you instead introduce a purely symbolic version of a destination and then let the environment interpret what that what that actually is and I think I think that's best explained sort of by going bottom up and in terms of how it's actually implemented and I hope to convince this is a very powerful and useful mechanism so first of all the very lowest level of a destination in finagle is this thing called an adder and so an adder is effectively a case class or an ADT that might may I'd be bound and so bound abdur has a set of socket addresses these are things we can actually connect to write it could be failed because maybe the thing that produces the outer failed or it could be negative which means that that destination doesn't exist we know that it doesn't exist right so that's what that's what an address is now on top of an address we have something call a name and so the interesting thing here by the way is that addresses are both concrete and Static right this is just a regular socket address and addresses describe concrete destinations if I if I have an outer bound that represents it can creat host list effectively that I can connect to so names are kind of interesting so names have two variants one is that it can be bound right and the trick here is that a bound name has a variable address right so that's an address that can change but what that tells me is that my name is so knit surname is some that represents the destination this tell us that the name is actually bound to this variable representation of address and the only thing that for the purposes of this talk at least that you have to be aware of is that this bar effectively the notes that this underlying a value of this underlying type can change under you basically now the more interesting thing is this variant called a path and a path again it's a symbolic name symbolic names are hierarchical so as an example of a valid path is something like this maybe it's the user main the main user service in this case right and so will take s to mean service and then user to be the kind and main to be the particular deployment I guess of of a particular service and that has pretty profound effects on how we treat names and naming in finna go so another thing that an important note on top of this is that using a users of finagle with rail or than rarely encounter names by them by themselves instead there's there's a mechanism to turn effectively parse a string into a name for you this color gris arbor and so in finagle we write this is how we write our in our code basically you might create a new client that that derives its it's a dress from or name rather from the zookeeper or service that's stored in the zookeeper path and in this case this is a name that bound this is a name that bound right but this is a name that path right and so here I've instantiate a client that has destination that is purely symbolic and so now the question is if I told you no more that would be basically no op you can that's completely meaningless and so we have to have a way of giving meaning to these kinds of names and that's a process called interpretation and so again interpretation is basically a way of turning a path into a bound to name that path into an end up bound and the way that's done in finagle is through something called delegations and what a delegation is is that it's effectively a way of rewriting a path into another path right and so here's a here's a particular kind of delegation that says well paths that begin with /s gets rewritten to have this prefix instead of the /s prefix right and in the so in this case if I had as user main it will be written to z kz k local tour contouring 181 user may right and so it's a simple sort of prefix substitution rewriting rule and so that is what a delegation is now that's not super interesting by itself because if all I had was a way of specifying how to rewrite a path in another path that's still just you know rewriting in the void it doesn't give me anything and so there are special kinds of paths that give rise to special kinds of interpretation and those those we call system paths and the begin with dollar and so for example the path dollar / come Twitter server sets don't result in rewriting but rather and something called the name or handling that we write and so for example in this case paths that begin with dollar come to our server set take the sort of residual of that path and give it to the server set namer which then can interpret it in this case that neymar returns a name that bound that has the actual value of that server set now the interesting things thing though is that namers can terminate like it just described by returning a name bound but they can also recurse by returning a name dot path right and so that might say well this path is really this other path and reinterpreted in my you know current delegation table or they can return a negative result which is to say this path doesn't exist authoritah bleah so the way and I hinted at this just a few minutes ago so the way so everything is tied together is to something called a delegation table and a delegation table is an ordered list of delegations and this is what we use to interpret these names in totals and this in this particular case oh and by the way rules apply bottom-up so later rules when basically so given this delegation table if we try to interpret our previous name which was s user main it would first get rewritten to serve proud user main which by this rule then into Zeke a local producer named by this rule and then finally to dollar come to our server set and then the rest of that path by this rule and here we terminate because that is some that we return to name bound to me now where it gets really interesting is when when these neighbors return negative results and when they do finagle will actually backtrack an interpretation so imagine you have a delegation table like this and and notice here that there's two definitions for SRV right so this one that goes to Zeke a local and one that goes to Zeke every mode and so by the previous role or the previous rule that i described will first interpret will first try to traverse this path and try to resolve a name given this rule and we can see how backtracking works here so again we try s user main it gets rewritten into syrup Raju sir main gets written as eeky eeky local and then finally enter the actual zookeeper concrete namer which returns negative result in this case so maybe that service that doesn't exist and in that case we actually backtrack up to this step again number two and try the remote one instead in this case maybe it succeeds and the the net effect of that is that we effectively have kind of overlaid namespaces so in this case I can say well try the the local one and if that fails try to remote one it's very kind of similar to how a union process to might work we're at several layers and I traverse those layers in order to find a name that actually our result in a service now the interesting thing interesting thing about that is is that we now have a way of sort of separating symbolic names from how they're interpreted right and so I might in my code just talk about /s user main but I might alter its environment by modifying the delegation table right and not only that but the delegation table gives me a lot of interesting flexibility which I'll get into get into now right and so so again what this what this ends up being is that you know we effectively have a sense of location transparency where I'm just talking about the symbolic names and the actual concrete location of what those names refer to is transparent to the client and some simple examples of where you can use that is for example well imagine that you have several different environments that you're deploying your processes in your service and you can without changing the code just alter the meaning of those names and perhaps in a staging deployment you want to talk to other staging servers or perhaps in development you want to talk to only servers that run on your local laptop we can now affect these kinds of interpretations without changing any configuration at all just by changing the delegation table itself so the delegation table in some ways encodes your operating environment so the interesting thing though is that we took this idea much further which is when you think about what it means to be an environment you might think well maybe it's the process lifetime or or something like that that's kind of a logical the limitation of what you might consider instead these delegation tables these environments actually apply in a per request basis right and so what that means is that i can actually a different delegation table for each individual requests and that'll oestrus really really interesting things and not only that but those environments can escape your kurt your your particular instance into service that you're talking to and so we have supporting our RPC protocols so mux and an HTTP in particular to ship effectively delegation tables across the wire and so if i operate in a particular environment so will the systems that i'm calling right and so for example if I'm if I'm staging then the system calling will also just you inherit the staging mess of my environment and it will dynamically reinterpret its names for the purposes of that single request to to to those those of the staging environment and that turns out to be really really interesting right so here's here's that that example concretely so perhaps I want to deploy this is kind of example that I give initially maybe I want to deploy a new version of user service that is staged and i want to send it traffic to see what you know how it works now i can simply issue a request from our air front end and say well this request a request trees should operate with this amended delegation table this amended environment i'm modifying the environment of the whole request tree and no matter where that request goes / s / user / main will always be interpreted now as / just lets users are staged and not not the original definition and and that shows one simple one simple way in which this thing is very powerful and in fact this is how we now do staging at Twitter and so we no longer have a specialized aging infrastructure you can simply deploy your system somewhere and instruct our HTTP front ends to amend the environment to account for you know your staging server there are a lot of other interesting uses of this this technology first and foremost system the ability to be the ability to define environments dynamically in the way that it described but there are many more so for example we use this mechanism also to the cross data center load balancing and the way you can think about that is that I'm just talking to I'm just talking about a symbolic name like / of / uses josh mein the environment might choose to say well what it means to be a / out / uses our name is to send eighty percent of your traffic is out there to Center in twenty percent your traffic so that other data center right and again it is completely transparently from the actual client code right and another really really really interesting thing that we did is this I think in January there was a hack week at Twitter that we have occasionally and Oliver was sitting right there and there's no longer Twitter and Antoine well i think is climbing somewhere and and myself we we got together and just in a matter of a few days implemented effectively the moral equivalent of D trays for distributed systems and what that allowed us to do is basically define a way of matching traffic so maybe i want i want to see ten percent of traffic that goes to the user service and given that traffic i want to run this you know little scrub so maybe that's script is to just print it out so here's my print of right or maybe you do more interesting things like maybe you want to do a chaos monkey like thing and fail certain percentage of requests or maybe you want to do hypothesis testing about your your system as a whole and and delay effectively introduce artificial latency into your request path and this we could all just do by defining this little DTrace auroch like scripts to match traffic and then execute a rule and this is all made possible by the fact that we have this kind of symbolic naming system and are able to enter precision systems in between other systems so we can do interesting interesting things like this this is a quote from an anonymous Twitter engineer who is impressed I guess by by being able to do this stuff so moving on so the other thing so we talked about flexibility so rather we talked about a component model a linker and so this is what I call a linker basically the system responsible for piecing together all the services that we're operating would and finally we have the interaction model and what do I mean by that basically all the kind of problems that I I outline in the beginning so how services interact with each other so today basically finagle has a lot of this stuff but it's very weakly defined or weekly specified at leasts right and so again finagle does all this stuff with respect to load balancing and queuing timeouts retries error interpretation and so on and so forth but as I hope to have convinced you of earlier this requires a lot of effort and a lot of kind of tying the implementation details or specifics of finagle with the services that you're deploying or the code that you're working with and I think it's possible that if we are able to race the the sort of level of abstraction a little bit we can do a lot better so you can you can sort of think about what what are your goals when you're deploying a distributed system or a component in the distributed system here are a few that that kind of show up over and over again which is first of all your system has to be well conditioned and what that means is that you want to be able to reject requests when you're overloaded so that you don't slow everybody done right so you prefer to reject 20 say twenty percent of your requests versus slowing down one hundred percent of your requests you want to be timely because otherwise you know the work that you're doing is useless right like you basically want to respond in a timely manner otherwise users will leave and also you want to be well-behaved you want to be a good citizen in this kind of distributed environment so for example you want to make sure that you can't overwhelm you know down systems that are downstream from you by you know unintentional or intentional traffic amplification for example and and kind of us I i say alluded to earlier this really requires a lot of vigilant tuning and and deep understanding of how finagle works in practice and I think that's problematic for for lots of different reasons first of all again as I kind of alluded to there really are a lot of conflict or we use these tools so again tools like timeouts and retries and so on to serve multiple conflicting goals right so for example you might use timeouts to you know deal with resource management basically because you want to make sure that you don't have really long with the objects in your in your heap you use timeouts to deal with liveness concerns because you want to make sure that you don't succumb to a downed server that will never respond and you also use timeouts to sort of effectively express your SLA your service level agreement right so that's just one example where you effectively have three sort of separate orthogonal goals all bundled up into this one parameter you supposedly should tune right and and those kinds of trade-offs show up all the time so like let's say you want to the exercise of saying maybe the system should tolerate a higher downstream error rate well maybe you can reach an increased retry budget then but then you might also increase your concurrency level as a side effect of doing that well then you may be you have to create you know larger connection pools but connection pools generally you know this you slice them generally to deal with spikes and now because because of that you can now affect the rate of primitive objects and so on and so forth so this is a really really large or cascading effects and without sort of understanding with the underlying goal is right you're stuck with these sort of low level knobs and again you're sort of stuck tying yourself to a particular implementation in particular behavior and that really ties the hands of the implementers of finagle right like there are a lot of a large class or category changes that vinegar just simply can't do because people are effectively assuming a particular behavior from finagle another problematic aspect of this is that this kind of configuration is inherently static in the sense that it fails to respond to varying operating conditions right and so you tune this at you know one point in time and then two weeks later your operating environment is totally changed and you fail to account for that in your service and you lose you effectively lose all the work that you did but I think again like the most troubling fact about this this kind of arrangement is that it compromises abstraction and modularity finagle is now unable to to do a number of things and we effectively tied ourselves to particular implementation so one way that you might address this is to instead of talking about all this low-level configuration instead talking about higher level goals right so the goal effectively expresses the behavior that you want right and so I think there are roughly 22 sort of large categories or goals one is service level objectives which you know effectively allow you to express what is you know what what is my expectation for an SLA for this request right so for example you might have a deadline or a timeout to express that and and on the other hand you also want to make sure that your system remains fair and that Amanda's managers of resources efficiently so you might also have effectively cost objectives and those might be something like you know the amount of effort you're willing into trying to satisfy a particular request and that sounds like a pretty simple change right in the sense that you change the API to explicitly not deal with these low-level configuration knobs but instead directly with a higher level goal but i think the the implications of that are quite profound because now finagle has to freedom to implement do whatever it can to meet those goals and we have a crisp definition of what the actual expectations are right we actually have a way to converse about those kinds of trade-offs and introduce dynamism into into the picture so I think just about the simplest SLO that you can come up with is as a deadline right and so deadline is basically after this time the requests or the response rather for requests it's no longer needed maybe that's because you're unwilling to wait for a certain amount of time like it's an actual user level goal right and so in finagle we actually introduced the ability to express that and we have a simple case class that basically allows you to express what it you know what an actual deadline is and then a standard request context that allows you to set that for the context of your request tree and just like the delegation tables I described before these are transmitted across process boundaries using our RPC protocols so if you set a deadline in one system it gets transmitted to to your downstream systems and this seemingly simple change actually allows you to do some really interesting things even upfront so here for example is now a very very simple admissions controller that simply rejects requests that have already expired right and so you might you might think well is that is that really useful and the answer that is absolutely yes which because you might for example have a server that undergoes garbage collection or has some other hiccup or there's a network hiccup for example and this allows you to reject requests that are not going to none whose responses are anyway useless and allow you to effectively shun the you know shun work but that is useless so that you can get yourself up on up on your own feet right and this is again is a very very simple very very simple admissions controller but nevertheless a useful one and Rubin and Bing in this room as well are working on on more sophisticated controllers that account for actual runtime behavior so you can imagine treating a service as a effectively a black box and trying to estimate how it would respond to a particular particular requests and then based on that estimation shed work early for example this is similar to how network routers do random early drop in things like that and even in this simple form it's very very useful and again this is very much ongoing work but I think we're hoping to move towards is expressing these high level goals and then do simple but quite effective things in the actual stack to both manage those resources as well as implement the the SLS in whatever form they are which leaves how do you express how do you express the sort of resource management or cost objective aspect of this so we talked about how you how you want to say here's my SLO effectively here's my service level objective but you also have to be able to express how much you're willing to pay to effectively achieve that right because perhaps and to account for things like making sure that you're a good citizen for example and not overwhelm downstream servers and so one thing that we've done in fandango recently as well is to introduce this notion of a fan out factor and this is in essence a simple version of a cost model and what it does is affected a lot is that the ratio between how many requests are asked to finagle and how many requests vinegar will actually try to perform right and to see why that's useful you can imagine well maybe finagle needs to perform a retry in order to try to successfully dispatch a request because the first one failed well this effectively gives you a budget for those retries right and you don't have to now manually specify here that this client has the static retry budget this other client has this other static retry budget and again that is problematic for all the reasons I described before where you're fixing this very static thing but the actual goal is that you want to be a good citizen here you can basically say well this is what it means to be a good citizen finagle you're free to retry as much as possible as long as you remain within the balance of this right and you can also imagine having more sophistication now for example the riving automatic backups based on your budget right so and things like that and the goal is the idea is very much the same we have this notion of a high level goal which frees the developer from reasoning about how all these different knobs interact and how they should be tuned and where instead a balsamic they interpret how to execute on that goal right and I think a lot of interesting stuff is going to come to come with that though that's logic future work being done by the core services team at Twitter so i think in conclusion basically where we're moving towards a model where our pc's kind of operate a higher level abstraction and so instead of treating our pc as a simple remote dispatch primitive we also introduced this notion of having having high level targets so you know symbolic names service objectives and cost objectives and so on and i think the kind of important thing here is that both of these these things allow us to kind of separate the semantics of what you're trying to accomplish from how it's actually being executed it kind of increases the level of separation of concerns that we have in the stack and I think that will be very very useful first of all it allows users to express what their goals are directly and it allows finagle to do more sophisticated more interesting things and more powerful things actually because we now have a semantic understanding what those goals are in order to meet those those objectives and so that's it thank you for coming here today and I hope you will enjoy the rest of the companies I think I think we also have time for some some questions and I'm happy to talk about this or anything else really for right so the question was whereas backtrack or whereas it could get a negative result from a namer you backtrack what opportunity to fail the answer is that you fail so the filler propagates and you don't have a you don't have that name yeah so the question was is there any sort of Kelly rounding strategies you can use on top not once that you can express directly in the delegation table but certainly in code you can but that's that's an interesting idea i mean you can imagine having an ability to basically say this rule is optional for the intensive purposes of failure so doing delegation people doing the biting hard basis on super awesome understand if there's yeah so the question was it sounds really cool to be able to do per request tree or per request delegation tables how do you make it perform the the answer is that we make it perform this works this works really well and it uses a fairly sophisticated caching scheme to be able to do that in fact the kind of resolution strategy or interpretation that I that I showed each intermediate stuff is actually cached and so if you reuse different parts from even different delegation tables we can reuse those results and so in practice it ends up working really well we had a lot of problems initially with that in fact we costs a fairly bad incident at one point in time but but I believe we have not seen any issues at all since done and it's used very heavily and so I think I think we've solved that problem yeah just a sort of practical question is it seems three of this is obviously all powered sub inspector I also magic power becomes possible to create disuse I'm sorry if there's a higher class something must've changed location table County process or who oversees yeah so the question is delegated powers are very very powerful how do you in practice sort of deal with the fact that you're giving a very sharp tool and a lot of rope to you know thousands of engineers and the answer that is basically we limit the places where you can introduce delegations into the system it's in particular we have modified our front-end so the way that works in practice is that if you're inside the Twitter office in your you know if you're a Twitter employee you can send extra HTTP headers with HTTP requests which kind of modifies the route and we are very careful about the shape of those modifications that can happen that are allowed to happen and so we have very strict policies now that's not to say that you can't override that but you really have to know what you're doing in order to do that and so that certainly prevents kind of accidental you know rope hanging i guess but without kind of limiting what you can do so it's kind of like the the distinction between you know cut komsan unsafe and whatever else right like we have we have that back door but by default we kind of limit the shape of overrides that you're able able to do in production and also the rates of them and things like that to make sure that like make sure that we don't accidentally ship a you know bad twitter client that has a delegation turn on by the thought that shouldn't have been there for example the sfo's nature motivating those the simulated like the lower level filters in services yeah yes so the question is how do I sell as well as compose and the answer is that the way they work in practice is that you set an SLO kind of for some requests tree and so you say i want this computation to run or risk under this SLO right now if parts of that computation also set their own ssl owes you pick the the most conservative of the two and so if you look at the implementation deadline for example there's a way of composing an exactly like that where i say i want to combine these two deadlines and what is the resulting deadline right and you can imagine composing other kinds of s lows in a similar way one over that yes so the question was how do you deal with cash expiration the answer is that all of the actual underlying data is stored and zookeeper and arcades right so the the name or implementation explicitly a force for variabilities you can basically say the result of this resolution is actually there you know changeable value right so in the case of zookeeper we can implement that by having watches and you know trigger triggering changes based on those if you're not using that you could use a TTL is up to the implementation but in practice what we do at Twitter at least as we use your keeper for that sure about having sort of heavily to clients and then resume linking allowed back into might have lots of different performance characteristics do so I guess ellos to fully address yeah so the question was basically when you're when you're a server you talk to other servers that are sort of invariably variable in their performance and how how do I solos address that the answer is that they don't directly address them and so in SLO just seriously here's the objective for for for my request right and you can inherit that in examine it and you could have a specialized implementation of an admission controller for example that accounts for you know the fact that you know something more about your server then can be known generically or perhaps we can come up with good enough sort of black box of mission controllers that can't account for that right so perhaps you know perhaps you can so so as a simple example perhaps the admission controller can maintain a queue / method and maybe the variability is kind of contained within a method right or maybe it says well the variability this thing is too great i'm just going to throw up my hands and give up and i'm not going to be able to make a good judgment but that's kind of TBD to be honest we have talked to one more so the question was will GC awareness for your implement SLO so so for this or not familiar DC avoidance is a effectively feature in finagle that you can turn on by a flag that allows you to route around garbage collections and so a server can signal to a client basically I'm about to undergo garbage collection stop sending me requests and the answer is yes you absolutely can because now I actually have a way of knowing what the expectation were request is and so you can say well I expect my garbage collection to last for 50 milliseconds but my response time is estimated to be 50 milliseconds and i have a budget of 100 milliseconds in that case i don't have to actually reject the request right as a simple example so you can certainly account for it and it could make things better for sure and in the sense that right now GC avoidance is an all-or-nothing proposition but if but if it doesn't matter for the purposes of the request that you're serving then you know you don't you don't need to worry about it ok Thank You ru Thank you