scala.bythebay.io: Chris Phelps, Finagle Your Own Codec
Recording: scala.bythebay.io: Chris Phelps, Finagle Your Own Codec
I you so can I get a quick show of hands who here is familiar with finagle okay pretty good how many of you guys have written your own protocol or use of protocol other than thrift HTTP maybe Redis and my sequel ok a few of course travis has so i'm going to talk i'm going to break my talk up into three main sections so we'll start with some kind of concepts of what our protocols in finagle and why might you write your own and what is writing your own entail second part will go into kind of a case study of what we've done at tendril with finagle protobuf and at the end i'll share some of the lessons and recommendations that we have out of that so at tendril we do energy efficiency and energy intelligence we're all about trying to understand how people use energy in their homes and optimize for that and we've done that by building our stack on top of finagle using finagle protobuf and we've run that in production for three and a bit years now so let's start with concepts most of you guys raise your hand you are familiar with finagle so this should be mostly review finagle is an extensible RPC system and its protocol agnostic and I highlight these couple things because these are kind of to me a couple of the key things either irrelevant to my talk but be that our differentiators for finagles so extensible RPC means we can add to it means we can use that to make our PC light calls as opposed to something like resti kind of calls and protocol agnostic again meaning we can build on top of it and it doesn't know anything about HTTP and all that kind of stuff so so why are PC and in particular why a binary format like protobuf so our PC sorry so rest right is resource oriented is very oriented around the data we have a very fixed set of verbs that we're used to operate on that data RPC on the other hand is operation oriented we're going to have the flexibility to have many more different verbs and operations that we're going to use so a lot of times when you're writing rest it feels kind of weird to say you know perform the simulation or something is that again is that opposed is that a PUD I don't know I'm gonna have to argue about it with my colleagues etc etc so um some things just are naturally procedurally not resourc in terms of binary formats versus JSON binary if we have a really chatty system we're passing a lot of data we can do that more efficiently in a binary format a lot of the serialization and deserialization tools are faster than doing JSON JSON parsing on strings and a lot of these a particular binary formats protocol buffers Avro thrift etc have a lot of other rich semantics that you can do to build your messages um and those are usually defined in terms of interface definition language which gives us some kind of schema that we can work with to define our messages so protocols in finagle turns out we talked about Finn protocols all over the place in the finagle docs but there's a never really a definition so for purposes of this talk a protocol in finagle has codec for doing the encoding and decoding for the wire dispatchers to figure out what methods you're going to call on each end client and server configuration initialization and set up your kind of stuff error handling and then potentially integrations with co generators compilers other things that are defining your protocol so let's start at the top of that list so codex in finagle based on the codec trait they define the encoding and decoding that we're going to use on the wire and this builds on top of Neddie in order to format up your stuff sent it over the net e channel pipeline to the other side unformatted on the other side codex also let you modify the service filter stack so you can add things like filtering on logging tracing all those kinds of things as filters typically you would have your protocol be symmetric so the same kind of format same kinds of objects on both the client side and the server side and your request in your response are probably also going to use the same encoding both ways but there might be some reason why you want to have things look differently to clients then they look to services or have your request format format things in entirely different way than your response formats things so and codec factory is another trait that's that's there to relate your client and server codecs so technically you end up having a client codec for the request and the service code sorry the other way around client a client codec for the client writing to the service side and a service codec for reading the other thing so Union flow looks something like this right your client code is going to call to your client side service dispatch your client side dispatcher then encode the data for the wire send it over the finagle nati channel pipeline on the other side decode that back into some objects figure out what you're going to call in your service dispatcher call your service implementation coming back the other way reverse all the arrows re-encode it back up to send back to the client client unpacks it matches a back up with requests sends it back to your client code and your service dispatcher and client dispatcher here deal with any errors that happen to arise so when you're building your interface for your protocol what sort of interface do you want to expose and there's kind of a spectrum here where one side is you're exposing finagle things services from request response the other side is you're trying to fit some kind of protocol right you're trying to fit code that's generated by pro to see or by a thrift compiler or something like that and you could do something in the middle so in Twitter's case thrift uses a special compiler Scrooge that generates shapes that are much more finagle like shapes in the case of our finagle protobuf we're using produce ease compiler its generating interfaces and we're trying to fit that so connection handling there's a couple ways you can do connection handling one is multiplexing and the other is connection pooling connection pooling you end up creating new connections to make more requests you typically only have one open request per connection and you only reuse a connection once you've completed a request things fail maybe your connection closes um maybe you Cola is a connection every time but you're still using more connections for more requests whereas a connection multiplexing approach which finagle implements and mocks uses one connection and shares that connection from any simultaneous requests and then you have the complexity of dealing with matching up a response to which requests do I need to go call callbacks on a connection pooling ends up having a lot more configurations MUX has a simpler configuration model but so you have to decide what you're going to build on top of another concept within finagle is service stack so finagle builds up a stack of components that each handle a small bit of behavior and by stacking these things all together then we get a rich service that can do all of the retries and timeouts and logins and different things different points in the stack there's a stack API that you can use now to build that Twitter docs say this is subtle and should be used by experts or there's client builder and server builder which twitter docs say these will be deprecated so maybe don't build on them so hopefully by the time we get to the point that Twitter makes a decision to deprecate there'll be a little more docs and a better way to do the stack API so that it's not so subtle and expert so I want to talk a little bit about ways to handle errors so um I'm thinking about errors two ways right so we have of application errors which things that our eyes out of your software right out of your services you're trying to run a simulation do some physics and you don't have weather data you're trying to get stuff from the database and you don't have a connection to the database something like that something that's application level might be an exception might be um you know some kind of empty response something like that versus framework errors we have timeouts where you have rejections because there's too much work going on where you have an exception escaped from somewhere that you catching you have to deal with so you have the question about do you deal with these two different types of errors the same way or different ways and there's kind of three main ways that you can approach those one way is protocols so if your protocol supports some kind of error mechanism that's a really great thing to use so thrift supports a specific message type which is an exception so if you can model your things into thrifts exception that's really nice if you don't have that available now you have to decide am i using special formatting on the wire to transmit my message or am I transmitting this on this error at user level right so in the user level case you have a certain message type or you have a field within your message that you're going to reserve for populating the error and then you're going to have to decide how do I can catch exceptions and map them to one of these approaches how do I do it on the other side am i throwing exceptions in places and catching exceptions and my just storing exceptions and never throwing them all those kinds of things and do you want to do it the same way on the client and on the server so now let's talk about finagle protobuf more specifically so I'll start with going into a little bit what protobuf ideals like right and this is based on the probe of to ideal so we have a couple of different kinds we have messages and we have services so messages end up being like the the data transfer objects or the value of exit we're going to pass around so here I've got a message he's got an optional string he's got an optional int he's got an optional context so we see here that we can have different types we can have they might be the optionality they might be there they might not be there we can refer to other messages this context is referring to another message my example here doesn't show repeated doesn't show required required in proto three is going away anyway um and when we use the standard pro to c compiler we get some java classes that we can call from scala that that looks something like this right they generate a builder pattern that we can use to build up our messages and they provide some checks for us to see are these optional things there how many things are in one of these repeated lists all that kind of thing so we can interact with these services define calls that we're going to make our PC methods that we're going to be able to call so here I've got an echo service he defines an RPC echo takes an echo request returns an echo response and we're defining these echo requests and these parameters and returns as messages so we'll have a message somewhere else that's defined that says what these things are and that out of pro to see is going to generate us this monster class that has a bunch of stuff in it one of the key things here that I want to highlight is it has an interface in there that provides each method that's in that service and it has this weird shape where it takes it returns void it takes a controller takes the request and it takes a callback of response so typical Java kind of shape not really a great scholar shape so this leads us to you know those questions that we asked before do we expose this do we expose finagle like where do we want to to do our mappings and additionally we get some stubs and we get some reflective classes and we get implementation stuff so we tendril have taken the pattern of using a separate request and response envelopes and then having the data in in a message that's inside that response envelope so i showed you echo before here's our echo request it's got our parameters this is a really simple one and then our response you notice we've got this error wrapper so that gives you an indication where we're going with our error handling and and then our echo which is going to be the the response that we got back from the server that we're going to pass around and do something with right the application level response so so what do I want to say here so this is um this is an example of how we're going to use pro to prote Oh Sees generated classes to call a service and then on the next slide I'll kind of break into where we plug into this with finagle protobuf so this bill deco client takes some some parameters that we're going to pass to client builder you notice here we're talking client builder not stack API some filters and things that we're going to build on top of the request the around the service executor service trace or stuff like that they're going to pass in the client builders so we're going to use client builder to set up our client we're going to get the stub from the protis e generated interfaces and then we've got an RPC factory that we're going to use to call on that stub to get the service that it's an instance of this stub that's got the finagle service inside it and then from a client perspective so the first thing is this this kind of a helper function that we're going to use to set up right and then our individual calling code is going to be like on the bottom will build the echo service with the server port that you want in practice we're doing a bunch of stuff to figure out that server port and whether we're getting it from zookeeper lookup or local for test purposes and so on and so on I will use a RPC factory to create the controller where we'll build our request we'll set up our callback and then we'll call on the stub that echo method with the control of the request and call back and in practice for us are the controller that we build in the callback actually end up being the same class and actually end up also implementing a guava listenable futures so then we can chain these all together is guava stuff later so back to what we just walked through so the RPC factory is how we build a stub the stub is generated by proto see implements that RPC service interface that I showed you guys before and it delegates the method calls to a channel channel that we use is the RPC channel in pole which uses the client builder to build the finagle service the call method that comes from the from the stub delegates to this channel impulse to the finagle service and then when the finagle services future completes we resolve all our callbacks tie everything back together so that our RPC call back that we passed to the stub gets resolved and down underneath that all we go through our encoders to format stuff up for the wire I'll talk about that a little bit in a couple of slides but we're taking the method name that we want to call we're using its hash code and that's what we're passing as the code that the server is going to be able to use to figure out which method should it call on the implementation so on the service side services are a little bit more straightforward we still have a bunch of parameter e kind of stuff we use a service builder interface to create our server and what our client uh what are what our service implementing developers end up doing is pretty much writing this little bit at the bottom that extends the interface that we got from proto see to do whatever business logic we want to call run on the response and and we've provided this kind of build echo server stuff up in a up in a wrapper library so RPC factory built the service service implementation that our developers write implements the protis II generated interface service dispatcher when it gets the incoming requests decides which method to call and implements that sorry invokes that method when that method the call it provides its own call back to that method so that when that callback happens the service dispatcher then gets a future completed and can resolve the promise accordingly prom future into that promise is given to finagle so that finagle then gets that resolved sends it back over the wire yada yada so we talked about these to our PC control and our PC callbacks that happen in there those come from proto see sorry those come from protobuf those are protobuf interfaces that we've implemented our PC control basically does the failure in cancellation stuff and our pc callback does the run method stuff and as i said before in practice we have one implementation that does both of those interfaces so i want to talk about wire formats and the main place i want to go here is to talk about the way that we evolved over time and what that means when you're writing a protocol so we started off with a really simple wire format we had the message code we had message length so that's the number of bytes that the message is going to take up then we had the message and that was it later on we said okay well finagle has this cool Zipkin thing that lets us trace through calls what would we need to do to support Zipkin so zipkin has several spans you have a span for the root of the trace you have a span for yourself and you have a span for your parent and you have some flags that say amongst other things am I tracing this request am I not tracing this request so how do I want to pass those just like I said with error handling we have to decide are we doing that on the wire are we doing that in in user space and requiring a certain shape of message to hold those we decided to do that in the wire well so now we have to distinguish when we get frame from Nettie is it the old format is the new format and one more constraint is we'd sort of like to be able to rev our services without forcing our clients to rev until they're ready so we don't have to do that across the board for every single service just to turn this thing on so so what we came up with is we have a version marker each of these is a few bites that we use to decide is this one of our one of our trace messages are v1 style or is this look more like the d0 style and will fall back and we'll use that if it's the v1 style then we'll go and we'll pull off all of those other traces then we'll look for our method code and other stuff at the end so our service when it starts up is configured as a v-0 v1 service the v1 implementation can handle both as I just said we fall back if it doesn't look like a v1 style message well detective urchin it is we'll use the right decoder to pull the right two bites out of the buffer our clients then are either v-0 v1 clients at this point everything city one but as we migrated that wasn't the case and so the client just writes however he wants to write if he's a v1 client he's going to write all the trace stuff if he's a visa recline he's not and then we made this decision that on the response everything was going to be v-0 so that clients never had to worry about things when they came back everybody could parse that message coming back so an old client speaks the old version the server responds with the old version he can still handle it a new client speaks the new version the server responds with the old version so you can still handle it so this let us do backwards compatibility and we didn't have to update clients until the clients were ready to say you know I need for some reason to rev to newer stuff let me pick up the v1 stuff while I do that so these are the kinds of things that you have to deal with when you're writing and maintaining a protocol and think about how do i add stuff to to a protocol that I've already got running how do I deal with backwards compatibility in the middle of that so error handling and mapping so we chose to do our error handling at application level so we pass as I showed you before on when I showed you the response message we have an error-filled within our response message and we populate that airfield when we have errors so our service dispatcher catches any exceptions that happen turns those into error messages and sends them back out and our error messages have a have an error code that tells us in this case we're not we're not doing that very sophisticated Lee but there's five or six errors that we handle and so each one is assigned a number and then the message is all the texts that we're pulling from the exception itself so both the the error message and all of the stack trace information that we have on the server side on our service side we have an exception handler which goes from exception to message so it catches a tip the service dispatch or catches the exception the handler converts that exception to a message and then that's what gets returned in the response then on the client side when you get that back he goes the other way so you take that on that error you you take the whole response message you decide if it has an error-filled in it or not if it has an airfield in it you send that error to the handler the handler turns it back into an exception and then we usually throw it which I don't like so some of the lessons and recommendations that we have is we went through that so why might you build your own protocol through the lens of hindsight if you need to do one side support like if you want an HTTPS maybe not a good example but it's something that you would expect to see existing services that are not yours out speaking this kind of protocol right if you want to write the client side for some protocol to talk to some other server or you want to write a server and have clients talk to you that might be a good reason to write your own protocol if you need to interrupt with some kind of legacy service that you have that speaks some kind of other wire format that does come some kind of weird that might be a good reason to build your own protocol and there's a couple of good examples in fanatical core about wrapping other things into services so something like my sequel or Redis lookups end up looking like finagle services that might be a good reason to write your own protocol otherwise you're probably better off using one of the existing protocols it's a lot of work to write and maintain your own protocol so if you don't have one of these compelling reasons using existing ones probably your way to go error handling it's tricky to get right it's even worse to test it with with hindsight we think we probably would have preferred to do transport or in protocol level handling the finagle thrift uses protocol level as I mentioned they have an exception type in thrift so that's fantastic otherwise it's doing it on the wire is probably a lot easier than four senior engineers to include an error-filled in their message because now what happens if they don't so now I've got to have code in my protocol to deal with does the user how does the message have an error-filled in it and if so use it if not figure out what a fallback is blah blah blah right if I'm doing everything on the wire don't have to worry about that so support community is pretty solidly thrift and HTTP and Twitter's pretty much focusing a lot of their their efforts into mocks so if you can use these you get a lot of their enhancements a lot of their bug fixes for free there's other people in the community working on them etc if you go your own way you're going your own way right and that means you're going to have to keep it maintained that means when they come up with a new clever way to do retries you're going to have to figure out how to backport that into your protocol so on so again if you don't have one of those compelling reasons using theirs is a pretty good way uh when you're figuring out what to expose in your interface do i use the finagle interface do i use a generated interface that i got out of proto c or a thrift compiler or something like that do i use an existing generator like proto c and deal with all the weirdness of the interface shapes that it gives me or do I roll my own like Twitter did with Scrooge to have a code emitted that fits to a shape that's nicer for them to work with it's more nicely with finagle you have to decide what you're going to expose to your to your developers and what you're going to hide away so we made the the decision that a lot of this stuff gets hidden behind another rapper library so our developers remember i already said guava listenable futures not scholar twitter futures a lot of our developers are using Java on top of this not Scala on top of this so we tried to ramp this stuff and hide it away and that's caused a lot of pain for us over time you know we thought that that would give us some decoupling so that we could rip finagle out but it's a leaky abstraction lots and lots of things are going to have to change fundamentally for us if we move our way from finagle so now I'm paying all this complexity costs for something that's really not going to buy me what I think it's going to buy me we bring the new engineers they're going to have to learn how our stack works anyway why is that any easier than teaching them how finagle works so facade approaches they're very tricky right you've got leaky abstractions you or figuring out how to map those abstraction zeeeee together you have to decide what your hand rolling what you're generating more layers and more magic you have the fewer engineers on your team who are going to understand that and be able to really dig into that and get to the root of everything so with hindsight our recommendation is probably exposed the finagle stuff and fewer layers and more simplicity so um as you build a protocol right you might have things that are based on an interface some of that's going to be interfaces that are generated by your your generators some of that are going to be things that you do yourself like the exception handler interface and now you have to decide if i'm writing to an exception handler interface where am I supplying the implementation for that am I supplying it in as an example in Doc's as a default as in a wrapper library am i forcing my developers to write the exception handler there's all reasons that you might do any of those but i would definitely push towards be sure that you at least have an example implementation for any of the interfaces that you write in your protocol and and have good docs around that right because if your engineers want to try to dig into this and they can't even figure out where is the implementation for that exception handler interface how do i get a service up and running and just figure learn anything more about this so that leads into making things easy for your developers you want to have good docs you want to have good examples real code as well as documentation code right test beds so we've done a lot of work very recently to try to understand various configuration parameters and how they interact with one another and what are the causes of certain kinds of errors and how do we respond from those kinds of errors and having a simple test bed that's not a full running production service has been really useful for us to have small place where we can just iterate on tests and seed projects you want to allow your developers to start from something and not just go read the docs and figure out how to build a new service and as you build more complicated services on top of things having those seed projects be more and more like your real service you know your real build files your real docker builds your other tools and frameworks that you might be interacting with in most of your services things like those so that people aren't bootstrapping themselves up from nothing where nothing may mean literally like no good docs nothing so I mentioned about us just recently doing a lot more of this learning this has been an ongoing process for us we're three years of into running for naval base stuff in production and we're still doing experiments and things all the time to understand how two different configurations affect my behavior my performance what might cause certain kinds of errors how do i recover from those kinds of errors for example the interaction between max concurrent requests on the server wait queue depths on the client timeouts throughout things on all these things interact with one another right and so we're continuing to learn that kind of stuff and when you're building a protocol that's important because a you want to make your protocol fit to the framework in the most efficient way possible but be you want to be able to diagnose this and understand what behaviors are coming from the underlying framework and what problems are arising because of things in your protocol right if I'm doing something weird with a couple of extra exception catchers and I've got three extra futures that are hidden inside of there right there may be some of that behaviors rising from me not just because of something that came from Nagel so a couple of things to check out first of all mr. Nadav sunette I hope I didn't just Massacre his name is speaking tomorrow morning about spark and protocol buffers I'm excited to hear his talk tomorrow and part of his talk I think he's talking about an alternative to proto see so we're interested in looking into that finagle serial is a finagle protocol in in the finagle organization that uses schoo deck underneath so you can build binary format services and finna this is where you can find finagle protobuf all the things i showed you here are we're hoping to have land in PR there in the next couple of weeks so i think i've got five more minutes if anyone has any questions so the question is are we publishing our learnings um yes I think we absolutely should I don't know what the shape of that is we do not have an engineering specific blog so I'm not sure how that will be but that's a fantastic suggestion I definitely like to do that any other questions so the question was when we revit our protocol why did this service not respond in kind so what you're suggesting is the service receives a v1 message and returns a v1 response the server receives a V zero message returns of e0 response that's a really good question honestly I think we just did not really think through that way of solving the problem because I spoke to one of the engineers and asking that question the other day it's like yeah that would have been a great idea did we consider compressing our protobuf message we have not for our scale and and size of messages we found that just the density of the binary format / jason has been enough reduction in message size that we've been happy with that anything else one more so I think the biggest reason for that was around having a place to put the air so something I didn't really touch on too much in the in the body of the talk is you have to let as you go through this kind of process right it's better to iterate on this and let the process teach you what the shape should be if you start by constraining yourself and saying I want my messages to look like this because I have a pre pre conceived notion of how to deal with that kind of message you're probably jumping through some extra hoops to make the protocol fit that constraint if you feel it out as you go you might make different decisions and I think in this case we had sort of decided ahead of time that we wanted to have the errors kept separate from from the responses and done at kind of that package level so we can pass around that envelope and always have the error there and always have the success case there [Music] so we absolutely have we built finagle protobuf back in late two thousand twelve early 2013 we spent a lot of time in a lot of conversations in fourteen as MUX was being built we've still got a lot of ongoing discussion internally about do we want to build on top of mucks or do we want to just migrated from finagle protobuf to thrift entirely thrift MUX entirely so it's kind of an ongoing thing in practice right we built this before mocks and we've been sticking with what we had so I think that's about all the time I have and it's lunch so like to let you guys go a minute early thank you very much [Applause]