SBTB 2014, Jake Donham: Stitch, an Applicative Functor for Composing RPC Services
Recording: SBTB 2014, Jake Donham: Stitch, an Applicative Functor for Composing RPC Services
uh hi i'm jake donom i'm an engineer at twitter i work on backend infrastructure stuff oops what happened here where are my buttons going automatically okay i want to tell you about a project that i work on on twitter called stitch it's a library for service composition um i'm going to start out by trying to explain what the problem is that i want to solve and tell you a little bit about the stitch api and then give you a handful of examples of cool things that you can do with this api and i just want to preview by saying that this kind of fits into what mariah was talking about yesterday about picking a nice functional abstraction to solve some pragmatic problem that we run into in a big system like twitter okay so this is kind of a schematic overview of the service oriented architecture that we have at twitter there are just tons and tons of services and services calling services and little knobs and and things hanging off everywhere at least that's what it feels like so a typical application of twitter is built of many services the stack is is both deep and wide and we have services calling services all the way down we use thrift rpc for communication among all these things and it's very typical to have batch apis in these services so the reason for this is just to amortize the cost of traversing the rpc stack so a pretty typical api would be get me the tweets for this sequence of ids and return a sequence of tweets so that's sort of the setting that we're working in so we have this new problem of like service oriented programming how do you write an application that's composed of dozens of services and how do you structure your code so that it can have good concurrency across all these services take advantage of these batch apis for efficiency and still be clear and modular and flexible and all the things that we would like out of our code it turns out that these wants are often in conflict and particularly i want to talk about the the problem of the batch apis okay so let me let me talk about the problem of the batch apis so this is a pretty typical thrift level interface at twitter what we have here is some request type and some response type and the request type is kind of heterogeneous there's there's a few different arms of it and the response is match up with the requests right so when you make a wreck a you get back a response a when you make a request b you get back a response b and often um this is a much larger uh set of set of instances for the trade so then we have this typical badge-oriented api where we take a sequence of the requests and return a future of a sequence of response so this is kind of the setting that mars was talking about where all of our rpc calls are represented by futures um okay so that's the situation so so what happens when you're trying to call this api so the first thing is that you have to kind of build up a batch of requests and depending on what you're doing this might be kind of a pain because it's sort of a non-compositional thing right so you you may be composing a bunch of functionality but if you want the request to a particular back-end service to be batched you have to sort of extract the different places in your compose requests that you need that you need to put into the batch so then we we make this call against the batch api and now we need to match up the requests and the responses and the normal case is that we get back a response of the correct type but there's this sort of can't happen case which is that we made a request of one type and got back a response of another type and we're sort of exposing that to the application level which is which is not that nice so let me show you how this looks uh with stitch um the big idea of stitch is that instead of dealing in batch apis we expose these single key apis which can be type safe for that single key and then all the batching happens behind the scenes so when we actually run a stitch query it sort of figures out the batches for you and you don't have to do that manually so in this case we would expose instead of a single batch call with this heterogeneous type we would expose two homogeneously typed individual key calls so the request a returns the stitch of response a and request b returns the stitch of response b and then we can sort of work on this thing um in a type safe way so this stitch join is just saying that we want to make these two calls concurrently if you've used the futures api it has a similar future join and then we always get back a type type safe response um okay so then this problem also comes up on the server side which is that we want to implement this heterogeneous batch api so we've got in the sequence of requests and they can be of any type so we want to handle them all concurrently right and it's very typical that what we're doing in handling this request is passing things off to some deeper level of the service stack and all of those the things below us also take batch apis and so what we want to do here is in sort of a clean way process each of these requests individually but we would still like to get batching when we call the next layer of services this is actually really a pain so you can sort of imagine how would you what you'd like to be able to do is just map sort of a request handler of these things and then collect up all the responses asynchronously and return them but that doesn't actually achieve any batching right so you are sort of forced to keep track of which requests resulted in which downstream requests and then put everything back together as you return it up has anyone run into this problem in their own systems does this sound familiar at all nobody okay all right well i'm solving a problem that you don't have i'm sorry about that but it is a big problem at twitter and this problem has led to uh really serious application architecture problems where the entire application is sort of structured around this need for retaining these batches so again here's this how this works in stitch um we can actually write these single key calls and do everything in the really natural way sort of process these requests one at a time and we can collect them up and when we call stitch run what we're doing here is actually finding all the batches and executing the rpc calls and converting the entire thing into a future that we can return up the stack okay so i want to tell you quickly about the stitch api um so there's a moon add for this problem this api looks quite a bit like futures if you've used them it's really almost identical in the external interface the internal implementation is really different and i'll talk about that briefly so we have the standard map and flat map handle and rescue for exception handling and again this looks very similar to what's in future we can make a stitch that's already completed with value we can join two stitches concurrently and get the pair as a result we can collect a sequence this traverse is something which is not in future but it's basically the combination of map and collect and it turns out to be convenient pretty much everywhere and then finally we can take one of these stitch expressions and turn it into a future so what happens when you do that oh sorry i got to talk about service adapters first okay so we have this sort of single key view of the world but we actually have to talk to the backend services which do not have this view of the world and so what we have are a set of service adapters one per service and what we want to do here is implement the single key call method that takes a request to a stitcher response and basically what we're doing here is the call group is talking about how do you batch these things together so it's saying if i get a call with this call group it can be batched with any other call of the same call group and the call group also explains how do you make the underlying batch service call and that's what you what you see up there now we still have this problem that the result of this call might be of the wrong type right so we didn't really get rid of this type safety we just moved it into a different place but it's sort of in the implementation of the service adapter so rather than application code having to know about this potential type on safety that can be in the service adapter so the application code is really clean okay so let me talk about how we actually run a stitch and turn it into a future so the first thing is we represent the whole query as a syntax tree so when you make all these different calls on a stitch you're basically just creating a new syntax object so a flatmap object or a handle object and then there's sort of an interpretation layer which interprets this query so we sort of run down the query looking for exposed calls that we can batch together and make an rpc call and when we get to a join those are two things which are supposed to happen concurrently so we can go down both branches to collect up calls traverse and map or the same way with flat map the the issue there is that we don't know what the sort of continuation query is going to be yet so we haven't actually executed the thing that the flat map depends on and so we might not know yet what's going to happen next so we basically just have to pause there and wait for some asynchronous work to happen so once we've collected up all these calls we can group them all together according to the call groups and execute these batch rpcs and every time one of these rpcs come back comes back we can sort of fill in the results that we got into the syntax tree simplify the whole tree and then repeat the process and that's how things are executed and i don't know if people have looked at the implementation of futures at all or promises but it's quite a bit different it's sort of a a future is like a mutable reference cell that gets updated uh when the result comes back and and this model is very different but what this lets us do is sort of see inside the the production of a stitch which you can't do with the future okay so here i just want to talk about a couple of cool things that uh you can do with this api um so here uh this is something that comes up in in a lot of our batch apis where um these batch apis expose like per key failure so it may be that something downstream of of the call to that key failed and we want to return that up the stack and then typically we want to do some retries or something like that and so what you'd like to do is try an entire batch of requests and then the ones of those which failed you'd like to make a retry just for those ones like so you don't want to have to retry the entire batch because lots of them succeeded and then maybe when those ones fail you want to do another retry with a smaller batch so what i've got here is sort of a per key version of that where we're not going to try to retry the whole batch we're just going to retry this single key so we're going to make the call use the rescue method to catch an exception if that's a retriable exception and we have more tries left then we'll just go around the loop again with one fewer try and otherwise we'll return the exception upstream so this is really this expresses the intent very clearly it's very simple there's no code here to collect up which ones which one succeeded and which ones failed but when we execute this we actually get the thing we wanted which was that the first time we try all the keys the next time we try only the ones that have failed and so on so that's kind of nice okay this is a totally different thing the talk about play yesterday mentioned a little bit this issue of how do you compose a web page out of lots of independent components so this gets to that question a little bit so in one of the applications of stitch at twitter we had mustache templates and we wanted to independently composed these asynchronous components that we're making rpc calls so what we did here is just sort of translate these mesh mustache templates into a stitch query and then we can execute the whole thing and achieve this batching across independently written pieces of the template so again we can just sort of plug together things which really didn't know anything about one another but still get batching across those components and completely concurrent execution um and this is sort of a non-rpc example of how you might use stitch and we do have an application on twitter that's using it this way you want to compose sql queries and this is a case where if you do lots and lots of individual single key sql queries your performance is terrible but if you do a batch query at the sql level you get decent performance so here's a way to do that again without exposing that batching to the application layer you can make lots of single key calls at the application layer but have that converted into a batch select call as i have here okay i see i have a little bit of time left so i just wanted to first acknowledge a huge debt to this facebook project called hacksall basically all the ideas of stitch come from hexel i based this on a talk from last august so the implementation is pretty different from the the hacksaw one but the ideas are or i'll take it from that um oh and i wanted to because i put this in the title of my talk say one thing about the difference between a monad and applicative factor um so this is just kind of a geek out for a second you may know that every monet can be an applicative functor in two different ways like so you can if you want to join two things you can sequence them the first one and then the second one or the second one on the first one using the magnetic composition the flat map but we don't actually want either of those because we want these things to happen concurrently and so that is sort of the key difference here right so like if you just take a monad and turn it into an applicative functor you're not exposing the independence of these two parts but that's really crucial to what we're doing here because we want to be able to batch across those two parts and so we don't want to sequence them okay that's all i have any questions yeah so um let me go back to the interface here so again if you've used the feature interface it's really similar i think we were talking about uh kind of corresponds to the collect interface so we have a sequence of queries and we want to wait for we want to concurrently wait for all of them and then return the sequence of results if any of those fail in collect then the whole thing fails so just like futures a stitch kind of embodies an exception so it can either return a successful result or an exception yeah yeah so um one of the nice things about doing things like one key at a time is that you often can sort of put your exception handling logic in the key query itself like so you don't have to sort of collect up a bunch of things and then sort of handle the successes and failures you can just put the failure handling in line to produce some results or return a default or call a backup service or do a retry or something like that so you don't have as much need for what you're talking about but you certainly can implement that the default collect doesn't do that but you can do something like you can lift a stitch into a try so that that will always succeed but it will succeed with a try which is either return or throw um i'm sorry i should have repeated the question before um you're asking whether these are the definition of sure okay so you're asking whether the request object should contain the number of retries um i think of those as pretty independent concerns like the request is sort of semantically what it is you're requesting and you're not necessarily saying anything about how it should be executed and i think kind of a a trend that's happening at twitter is trying to get away from the sort of like manually specifying retry accounts and things like that and trying to just have some objective for success rate and try to have that handled as automatically as possible so i think that's a good that kind of points out how these are really independent concerns like the application doesn't really care about retries like it just wants a successful response right any other questions stitch is not yet open source unfortunately but i hope to get it there pretty soon i've been focusing on internal adoption anything else great thanks a lot