SBTB 2014, Greg Soltis: Designing Highly Concurrent, Multi-Protocol, Multi-Tenant Services in Scala
designing highly concurrent multi-protocol multi-tenant services in scala so my co-workers made fun of me for this talk title but i thought it sounded pretty interesting and hopefully you all agree my name is greg soltis for the past two years i've been an engineer at firebase i work on our back end as a service product and it is an example of a highly concurrent multi-protocol multi-tenant service written in scala in case you were wondering where i came up with the talk title so what are we going to cover uh we're going to build in a couple of steps a program that does these things it's going to speak multiple network protocols it's going to handle a large number of concurrent clients and it's going to handle many independent applications at the same time but what's so hard about all of this the main trick is managing complexity any one of these applications or protocols might be simple but when you mix them all together running at the same time things get complicated very quickly and before we start i just want to cover a couple of definitions so that we're all on the same page about what we're going to get out of this the first being network protocol so i'm intentionally being pretty generic here i'm not tying this to like a specific level of the osi stack this could be udp versus tcp or it could just be rest versus long polling over http the important part is that you have to follow a different code path to send the same message to two different clients basically you have you have non-reusable code for that same message large number of clients this is an arbitrary number i picked the main thing i wanted to do sort of with apologies to the node crowd is rule out single threaded solutions we're going to handle a large number of clients and we're going to do it with multi-threading and coordination and finally application um this is also maybe a little bit of a weird definition for application um but it's kind of a worst case in the absolute worst case we're going to need to serialize access to our application state in the real world probably your application can be a little bit more parallel and that's great that'll improve performance but we're going to go through a worst case and see how even with that we can do it and manage the complexity all right round one we're going to go with a an apple uh sorry a program that handles a single application a single single network protocol and to start off with we're gonna uh just require that it's only used by one client at a time uh so for instance maybe we have a static html javascript page somewhere or you know corporate internet portal or whatever and we're going to write a web service that's going to add a little bit of dynamic content to it and since this is a very contrived example here we're going to feel perfectly justified picking something way overpowered for the job so we're going to use a library called neti neti is an asynchronous event driven network application framework and it's it's written in java but we're using scala and so we'll have no trouble incorporating it it's all on the jvm the abstraction that nettie gives you is a pipeline so connections are channels and each channel has a pipeline of handlers and handlers respond to events on those channels events are things like here's some data to read or here's some data to write or this channel is closed and actually these handlers correspond pretty well to the filters that marius was talking about in the last talk they can take some input and transform it or decide to respond to it or preempt it or something like that and these handlers are chained together to form are to provide interesting functionality under the hood nedi is using all non-blocking i o calls and it's mapping channels to a group of single threaded event loops so our first neti application is going to have two main pieces of configuration this here is an abbreviated version of just setting up a server it's going to look a little bit familiar to those of you that have done socket programming before we're binding to a host and port and the second line there is telling nettie that when you get a new channel use this thing to figure out what the pipeline is going to be for it and that's the interesting part so here's our first pipeline where we've got some http parsing to start with uh we're parsing from bytes to an http request we're batching up all the different parts of the request into a full request which is uh kind of an optimization or not optimization uh simplification that i'm doing just so we don't have to worry about streaming for now on the outbound side we're going to expect an http response and we're going to transform that into bytes to be sent back to the client and the so far these are all handlers that ship with neti you can use them or not but they provide a bunch for common protocols so we're going to use these and then we have the first version of our application so for our our first example here we're going to make a bespoke social local big data weather application and we're going to use it to let us know if it's foggy outside the firebase office and we need to wear a hoodie or if it's not foggy and we should carry one with us for later and here's a little snippet from the first version of our app so we're going to use a scala object a singleton that's just wrapping a boolean value and you know there is foggy value and we're directly dealing with our http request we're saying okay if it's a get uh just you know render a response that includes that boolean if it's a post update the state and we'll respond with the current state as well also let's talk about threading for a second we're building this application for a single client and to start with we're using a single event loop which has one underlying thread so right now there's no concurrent access and so our singleton with just a boolean there is fine but what happens if you open this up in i don't know multiple tabs while you're doing development what's going to happen well it turns out we're actually still fine in this case since we have just a single thread even though we have uh two clients now they're still going to be processing in serial the event loop's going to be handling one at a time it's one thread so the good news is we can now use this application on our whole team without having to change anything so what do we have so far we do we're dealing directly with uh http gets and posts we're taking input and returning a boolean we don't have any concurrency yet but we are handling uh multiple uh clients and we're constructing a response directly in our application there so this is all pretty simple standard stuff um let's move on round two we're going to get a bit more complicated we're going to handle a lot more clients and more protocols so our application was a hit and the ceo asked us to roll it out to the whole company uh one thread is just not going to cut it anymore we've got very big company load times are going through the roof and our 16 core server is still sitting just mostly idle so we're going to have to figure out how to scale this up so we're going to add more event loops to do that this is part of where nedi helps us is that when we get a new connection nettie assigns that connection to an event loop in a pool of event loops that we give it so this is great we'll be able to handle a lot more connections by just sharding them over different event loops but we've introduced some complexity here now we have race conditions in our application that just single tin with a boolean is not really going to work anymore and to make matters more complicated the marketing department got wind of how popular our application is and they want some metrics for a blog post they want to know exactly how many reads and writes have happened at any given time so what we're going to need to do remembering that this is kind of a contrived example so we're going to need to serialize access to our application because we need to give them exact counts and for requests that are updating the state we want to make sure we're returning the state that it was updated to and not some later state in the application so we're for now going to use a lock just around our application state we'll get a request coming in it'll block on the lock for the application it will do what it needs to do either just update the read counter and grab the state or update the right counter and grab the state this is not ideal but it's progress and should be able to handle the traffic okay we said we're going to add more protocols in this round as well the head of product decided that refresh is too slow uh he wants live updating weather widgets on the company intranet portal so we're going to add websocket support this is going to be our first persistent connection this is not just request response what we're going to have to do is whenever that state changes we're going to broadcast out the new value to all of the persistent connections so they can all update their weather widget to make matters worse the it department their collective motto is never upgrade they still use ie6 so we're going to have to include some long polling implementation as well just to kind of add some more complexity to our server code here and on top of that the corporate firewall only allows connections on port 443 so we can't cheat and just run all these on a different port we're going to have to come up with another technique so in one respect we're lucky all of these protocols start out looking like http which means we can keep our http parsing in place and what we're going to do is when we get a new request we're going to look at it decide which protocol it corresponds to and reconfigure the pipeline accordingly so this is another strength of nettie is that the pipeline configuration that we saw earlier is not static we can at any time in the course of processing a request decide to completely reconfigure it so if we notice that we don't in fact have a get request what we really have is a websocket upgrade request we just reconfigure the pipeline to deal in websocket traffic from then on um and since we're using http for these we can make the uh protocol detection a little bit easier on ourselves uh we can you know the well what i'm going to do actually is just use the extension the file extension on the end of the url path uh so here's some changes to our application at the top we're now using multiple event loops we're going to modify our pipeline to add this uh stage that looks at what protocol we're using and then uh in that protocol chooser we're going to modify the pipeline accordingly so here's uh just a little snippet of checking for if we have a websocket and if we have an http request like our first version we can just pass it along to the original version of our application that we had all right so our application has to change a little bit though uh before we were dealing directly with http requests and we were constructing an http response that wrapped a boolean just directly in our application code and this is a good example now that we have multiple protocols where abstraction is important right so we have instead this trait here that speaks in the semantics of our application rather than in the semantics of the protocol like previously it was like here's a get or here's a post now it's get state or update the state or here's a new listener that needs to be notified when the state changes which is in addition in this round we've got those persistent connections anytime we're updating the state we're going to need to send to them so if you'll notice in those two methods the adding and removing listener there's a type there that says nothing about what the protocol is so we're going to implement this trait here in our websocket and long polling codes that our application on the inbound side we said it doesn't need to know what the what the protocol is anymore and similarly on the outbound side it shouldn't need to know anything about the network protocol this will help keep your application code simpler and is a good way to modularize your design and then just here's an example of our websocket handler it's one of the neti handlers handling websocket traffic but it's also implementing this trait it knows how to say okay here's a new state let's send it to the the connected client all right so this is what we've got in round two we now have capacity for a large number of clients we've got those 32 event loops that are all processing asynchronous events from different channels that's going to scale pretty well we have a protocol independent application interface which is good we talked about getting the network specifics out of our application definition and similarly on the outbound side we have a protocol independent way for our application to send messages to a client and uh we have a few different protocols now we've got some that are uh request response that's our like basic http get and post and we've got some persistent ones the websocket and even though actually the long polling isn't quite persistent it presents itself as persistent to the application the application doesn't care about the details it just knows here's a persistent connection all right round three things with our app are going very well so one of the engineers noticed that we could apply the idea uh synchronizing booleans to a bunch of different problems you know the boston office wants to know if there's currently a baseball game or the new york city office wants to know if uber surge pricing is in effect and the hr department loves the live updating somo weather report so much they want to apply the same idea to their pdf to word resume converter unfortunately accounting doesn't want to pay for another cluster for each of these applications and they observe that although the latency is occasionally a little high on the weather app the cores are still a bit underutilized it's a very technically savvy accounting department all right so we're all experienced engineers and that means we're all probably a little bit lazy we're not going to overthink this we're just going to reuse the application code that we have we'll just put multiple copies under different urls so we're going to need to make it a little bit generic it's not going to be tied to specifically whether or not it's foggy and soma anymore the hr department is supplying their resume converter code we'll just need to wire it in and because we can kind of see where this is going we're going to put the mapping from urls to applications in some sort of external queryable data store so that we don't have to redeploy our cluster every time somebody wants to add a new instance of one of these applications and here's how it might look so when we get a request uh rather than just uh speaking to that uh trait before with this dealing with the uh state of whether or not it's foggy we're going to look up the specific instance of our application and your implementation of this could do something like talking to zookeeper talking to memcache or some database or flat file or whatever you return an instance of the the boolean sync application and then your protocol already knows how to deal with that we just replaced that singleton we were talking to before with the treat for the boolean sync app and then that same the boolean sync app is again just wrapping the boolean state it's wrapping our counters we still have locks in there all right so we've got a uh generic application trait as a result out of this which is good as long as uh you know your independent applications all kind of have a similar-ish behavior you know you can include some stuff in your external store to customize them a bit we've got some app lookup that we can do so our our set of applications that we're running is now defined at runtime rather than when we write the code and our network protocols talk to a generic app not a specific app so they're a bit more reusable now okay so that's most of what we set out to do but i we still have another round to go and i kind of snuck in an extra bullet point here to remind us of what we need to accomplish still i mentioned back in round two that uh there are some problems that we're gonna come back to and specifically it's a performance problem uh so it has to do with how we are serializing access to the application state i said we were just going to have a lock and when a request comes in it would block on that lock and then do whatever it needs to do so the problem lies in what is actually blocking on that lock and the answer is the thread underlying netty's event loop so let's say we have one application that's extremely inefficient take the the resume converting application as an example and maybe each conversion takes like 30 seconds to do if we get a bunch of requests for that particular application one of them will grab the lock it'll be in there for 30 seconds and the others will all be blocked on it blocking the underlying event loop and what that means is all of the other requests and responses in the system will be blocked because the event loops that we're running them are blocked on that one application and that's pretty clearly not going to work for a multi-tenant service one of the challenges in a multi-tenant service is giving each tenant the sort of illusion of isolation even though they're actually using the same shared resources so we're going to have to come up with a different way of serializing access so we're going to use akka akka is an actor framework it's written in scala from akka.io actors are very lightweight concurrent entities they process messages asynchronously using an event driven receive loop all right so what does that all mean the basic idea is that an actor is a behavior function coupled with a mailbox which is usually a queue of some sort of messages to process the actors system which underlies all the different actors is responsible for successively calling the behavior function with each message in the mailbox other actors and external pieces of code can at any time add messages to be to the mailbox to be processed by a particular actor one of the guarantees that we get from the actor system is that a given actor will only be processing a single message at a time so it behaves in a single threaded fashion although we aren't actually guaranteed that the thread running that actor won't change between messages we're guaranteed that we'll have one continuous run of a message on a single thread so for our for our purposes we don't really care which thread is running it we just care that access to our application state is serialized remember that the marketing department needs absolutely 100 accurate counts and we said earlier that an application was going to be a single threaded state machine it's kind of a worst case here so this is going to work for us we'll just use an actor for each application so akka actors form kind of a hierarchical model you can have child actors belonging to a parent actor and if your application is kind of decomposable a bit and you can run more pieces in parallel this is a pretty good way to do it but again we're going with the the simple worst case we're just going to use one actor per application and how is this going to work with nete so the network protocols are still going to be running through the neti pipeline the the group of event loops that we have but instead of blocking on an application they're just going to append a message to the correct application's mailbox and we're getting that via our app lookup that we added in round three so this is going to let us run a whole bunch of different applications all multiplexed over a thread pool that is the actor system okay nettie and akka sound kind of familiar are similar if you look at their descriptions so i thought i would just quickly cover some of the the differences and similarities between them the real fact is you don't actually need to use both like you could implement this design using just one of them for instance with akka you could assign each channel an actor and then you would have serialized access to that channel and with neti you could actually assign each application to a channel and then you would have serialized access there but it gets a little um a little complicated but definitely doable they each have their strengths though so nettie is very focused on network performance it uses direct memory buffers and can be tuned to do zero copyrights which is great for performance and it also comes with a whole bunch of ready-made handlers for doing net network various network protocols and network related things like batching and rate limiting and stuff like that whereas akka is a bit more generic so you can do networking with it but that's not all that it's made for where it really shines is how customizable the concurrency model is so if you have some piece of your application that can be run in parallel you can use a bunch of different actors for that and then some part that needs to be serialized they can all feed into one actor for that and you can kind of do this you know in whatever way makes sense for your particular application and it's pretty easy to do it's all just sending messages to different mailboxes whereas with nettie it's a little bit difficult to go from a stream of events for a channel and then regroup them into a stream of events for an application it's doable it's just not quite as simple akka gives you that flexibility so we're going to use both of them and try to use the strong parts from each one all right so how is this going to look for our application let's go through the the simple case of just the http requests say we get a get request for the the current state of one of our boolean sync apps we're going to use the ask pattern which is the question mark there and it returns a future uh it's a future like we saw in the last uh in the last talk it's either going to be empty or have a value or have failed and we're gonna that that ask is sending a message to the actor the actor is gonna get that message it's going to say okay this is the current state and send it back which will complete that future and that's all running in akka's thread pool when the future completes we'll have a just a tiny little bit of glue code to kick it back over to nettie which knows how to serialize a an http request and send it back to the client the important part is though the in this entire time the nettie pipeline is not blocked it's not waiting on that uh that response from the actor in the meantime it can be processing other messages which might go to other applications or even the same one it's just appending them to the mailbox so similarly similarly for a post request we're going to send a message into the actor and wait for a response our message here is going to be a little bit different it's going to include the state that we're trying to update it to but in general it's the same idea and then the last thing that we really need to handle is our persistent connections the ones that get broadcasts whenever the state changes so they still need to be registered with our application uh so that when the state changes the application knows who needs to get notified uh so we're gonna send a message in again uh this time we don't need to use the ask pattern we're not waiting for a response this is just kind of a one-way message to the application saying hey here's a new listener and so that message will include that new listener you can do that one a couple of ways you could use an actor for each listener or just a simple hook to get back to the netty pipeline that that part's kind of up to you and what makes sense for your application and so here are some messages that we might use um scala helps us out here if you're doing this i definitely recommend a sealed trait uh just so you can guarantee that your actor is handling all of the messages that you that you can send it uh in addition the messages should definitely be immutable given the actor system is going to be bouncing these around to different threads we don't want to have to worry about somebody changing one of these things out from under us so again this is a nice benefit of using scala for this problem um but here are our messages uh that you know this is how our the things our application can do right it can get the state set the state we we talked about our stats and our broadcast listeners there and this is what the actor or a snippet of the actor looks like so the receive method is the behavior function for the actor it gets called uh one with the messages in the mailbox kind of one at a time and this little snippet here is how we might implement the uh get state message we grab the sender for the message and we log our read and send the state back to the sender which is going to complete that future and kick back over to the neti pipeline one thing to notice here is that we don't need any locks anymore the actor system is guaranteeing that we're only processing one message at a time and since this actor is wrapping our state we actually have serialized access to our application state just by virtue of using an actor all right so this is uh our final program here we've got multiple protocols they're a mix of request response and subscriptions for broadcasts we've got support for a large number of clients and we're not blocking the event loops anymore so that'll let that scale up nicely the messages get just get appended to the actor mailboxes and we have support for many different applications uh running concurrently and independently on the thread pool underlying the actor system um the the full amount that you can scale that up really kind of depends on your workload but i guess the the max that you can have run running concurrently is the number of threads in your thread pool we also got some bonus results these are not really stated goals up front but turned out to be some pretty nice properties of this design so we have a an extensible interface for new protocols right we we can implement that listener and we've got new broadcast protocols um we have those messages that we send to the application so as long as the protocol knows how to turn a message into a network message into one of those actor messages we've got a separate protocol and the application doesn't need to know anything about it so along those lines the protocol agnostic interface for each application is pretty handy for managing complexity and also for being able to test we no longer need any sort of networking or uh protocol code in our application logic's test code which is great we can write just send these messages to this actor and expect the state to look like that and that way we don't have any sort of latency or race conditions or anything like that in our test code and finally one additional benefit that we got out of all this is we actually have an audit log for our application so we said we're going to use one top level actor for each application well the list of messages that it processes and in that order corresponds to all of the state changes for our application so what that means is if or when something goes wrong rather than looking at a log of http requests that came into your system that probably won't have the broadcasts it probably won't have websocket frames you can actually be looking at the list of messages that were sent to that actor and you can if you want reconstruct them and just replay them and you should be able to generate whatever problem you observed and then once you have that it's much simpler to debug all right and that's about it so all this code is uh up on github you can check out the four different versions and see how it works um i left out the long polling and resume converting portions um i also want to say both nettie and akka are extremely powerful and very tunable i've put up code that just kind of grazes the surface in the interest of keeping things simple definitely read the documentation if you are building something to operate at a large scale and also just the obligatory bit if you enjoy working on these kinds of problems firebase is always hiring this is more or less how our service works we've scaled this up to handle thousands of applications at a time a million over a million concurrent connections so with some tuning this can definitely work so we have um just one actor system um you can use multiple depending on your workload but uh we are um on a given server multiplexing uh just over one actor system so we actually uh actually another kind of nice property between uh from all this decoupling is this doesn't actually all have to run on the same server uh your you could be sending messages to actors on remote servers and just have a bunch of edge servers that are running nete pipelines we have them coupled together but we have our applications split over different servers up in the back sorry uh for i'm sorry uh persistent connections oh load balancing um so actually we we looked at using load balancers at one point a lot of them were saying things like they'll handle you know 100 000 concurrent connections and we're like all right but one of our servers kind of already does that so we've done it at our protocol level basically we have a hash ring and when you connect each server can tell the client where the actual server to connect to is uh so that's what we're doing for load balancing i want to thank greg from firebase