SF Scala, Greg Soltis: High Performance Services in Scala

SF Scala, Greg Soltis: High Performance Services in Scala

Recording: SF Scala, Greg Soltis: High Performance Services in Scala

designing highly concurrent multi-protocol multi-tenant services in Scala first uh oh my my name is Greg Soltis a work at firebase and the firebase is a mysterious company we make a back-end as a service but it's a real time back in as a service so what this means is clients connect directly to firebase they use it as a database but when they change values in that database other connected clients get a call back directly in their client side code typically within milliseconds so our version of hello world is building a chat application you can do it in about five to ten lines of JavaScript and we manage all of this on the back end with scala firebase is an example of a highly concurrent multi-protocol multi-tenant service written in scala that's where i came up with the talk title all right so what are we going to talk about um we're going to walk through building a scholar program that understands multiple different protocols it's going to handle a large number of connections and this will be for a single server if you've got a cluster of them you'll be able to handle many more and we're going to run multiple independent applications on that one server in that same process what's so hard about all this mostly comes down to managing complexity I we're going to be in a highly concurrent environment and as we all know that's a difficult programming environment so some of the things we're going to discuss our strategies for managing that complexity and keeping it as something we can reason about also dynamically portioning out resources we have we're building a multi-tenant server here and we need to keep applications from interfering with each other despite the fact that they're using shared resources they're running in the same process they're using the same access to memory and the network okay some quick definitions before we get started network protocol for our purposes not going to tie it to a specific layer of the stack we're just going to say anything that requires different code paths to send the same message to different clients so this could be TCP versus UDP or it could be responding to a rest request versus sending something out over a WebSocket the point is we have some code that we we can't reuse across different connections large number of clients this number I kind of just picked it's arbitrary but the purpose is we want to rule out a single threaded solution so kind of with with some apology to the node.js crowd we are going to handle a large number of clients using threads and coordination between them application this is a little bit of a weird definition for an application and the reason i picked it is because it approximates a worse case most applications are many applications you can decompose a bit more than a single thread likely all of your applications can be parallelized in some manner but if we can do this with our worst case here then your application will be that much more efficient if you can decompose it a little bit and we'll talk about how to do that as well all right so to start with we're going to handle a single application just one protocol and a small number of clients for a small number will consider this something that we can handle with just a single thread to get started so for for an example say we have some static HTML and JavaScript application being served from somewhere corporate intranet portal or something we're going to write a web service that will add a little bit of dynamic content to that so these are some somewhat contrived requirements here given that it's contrived we're going to pick something that's a little bit overpowered for the job we use a library called Nettie it is a well an asynchronous event-driven network application framework it's written in Java but since we're using Scala will have no problem using it and we use that to manage the networking for our application so how does Nettie work Nettie gives you a couple of different abstractions the main one is a pipeline of handlers so each connection they refer to as a channel each channel has this pipeline of handlers handlers are things that respond to events on that channel an event might be something like here's some new data that just arrived on this socket or here's some new data that needs to be written to this socket or this channel just closed and then the pipeline orders these handlers and you chain them together to get interesting functionality and we'll see an example of this in a minute underlying all of this as the Java niño library and nettie manages I event loops one at one event loop per thread and assigns channels to an event loop so what that means is all of the events for a given channel are tied to a given event loop which is on a single thread this is one of the ways we will be able to manage some of the complexity of doing concurrent programming later is that we'll be able to say if we're working with this channel we know it's on this thread and to start with we're just going to use a single event loop to keep things simple alright configuring Neddie some of this is going to look a little bit familiar to those of you that have done socket programming before we're just binding to a host in port the interesting part is the child handler where we're saying any accepted connections use this thing to configure the pipeline for that connection and this is what the pipeline configuration looks like so we've got a couple of groups here on the inbound side where we're saying okay this thing is going to take bites and turn it into an HTTP message or a piece of one then we're going to aggregate them until we get a full HTTP message to deal with on the outbound side our application is going to produce an HTTP response and that handler is going to say okay I have an HTTP response i'm going to produce some bites and those are going to go out on the wire and those ones are all standard handlers that ship with Neddy you don't have to use them but they provide support for a lot of different protocols out of the box and then the last thing in our pipeline here is our our first version of our application which is going to be a bespoke social local big data weather application we're going to use it to let us know if it's foggy outside the firebase office and we need to wear a hoodie or if it's not foggy and we should just bring our hoodie for later and this is a little snippet of code from that application so a couple things to notice I we're doing kind of some low-level HTTP stuff here where we're checking the method where we're calling this get boolean response which is a method that we're going to have to define Nettie is not a full-featured web framework like rails or Django or something like that but you can build one on top of it it provides kind of the the next layer down for abstraction some other things to notice is we're just directly grabbing our state here right so this is going to be a singleton object and it's just wrapping that lien is foggy value and that's fine because right now we're on a single event loop which means we have single threaded access to this which means similar to a nodejs application we're never going to have two requests accessing that at the same time but in for the same reason it's only going to scale up to a point there we go alright so what do we have at the end of round one can accept gets and posts we have the boolean state that we can return an update as I said we have no concurrency yet and our application is directly constructing these HTTP responses but this means that you know our co-workers working on the internet portal can embed a little widget that makes a get request to our service and maybe a button that does a post if you want to update it ok so our application is pretty successful at this point everyone's pretty into it so we're going to take a couple of big steps here in the next round the CEO loved it he wants us to roll it out to the whole company big multinational is going to be a lot of people single thread is not going to cut it anymore and on top of that our server you know let's say 16 core server I really pretty lightly loaded we've got one core busy handling that one thread but the rest of them are just sitting idle so we're going to do this by using multiple event loops in that configuration we saw at the beginning we were handing off new connections to that event loop that had the pipeline constructed for the channel this is going to change now we're going to hand off to one of several event loops chosen by Neddy really round robin but for our purposes it's going to look random which channel goes to which event loop and that's going to give us a whole bunch of different event loops operating concurrently parsing those HTTP messages or rendering the responses out two bites and that's going to solve some of our core utilization issue we're going to be able to spread the load over our server now a quick detour the the marketing department heard about how popular this application was and they want some stats for a blog post they want to know exactly how many times that is foggy value has been read and how many times it's been written remembering our definition of an application is that single threaded state machine this is going to translate to us having to serialize access to that little piece of state which means we're going to need some locks at least to start with here because now we've got multiple event loops all trying to get that state and update a counter or update it and update a counter at the same time and we need to manage that okay the other big step we're going to take in round two here is we're going to add some some live updating refresh is too slow so we can do this with WebSockets will have a the people working on the internet will just open up a web socket to our server and we will be expected to send them the state and additionally if somebody changes the state will broadcast that change out to them so unfortunately the IT department has a collective motto of never upgrade and they still use IE six so we're going to include long polling as well and this is going to give us our three different protocols in addition the IT department mandates that everything has to go over port 443 so we can't even cheat and run these protocols on different ports we're going to have to have another strategy for dealing with that but we're lucky with all three of these protocols that we're supporting which is the basic request response for HTTP the WebSocket and the long polling over HTTP they all start out looking like HTTP requests and that means when we get that initial header we the initial line for the HTTP request as well as the headers we can look at it and figure out which protocol we're dealing with and this is where Nettie really shines is because that pipeline configuration that we saw earlier is not static we can reconfigure it on the fly so we can say okay this message that started out kind of looking like a get request is actually a WebSocket upgrade request so get rid of all that HTTP stuff we had in our application and swap in the WebSocket handler and let that handle the traffic on this channel from now on and again we can do this dynamically independent per channel even if they're on the same event loop oh yeah we also need to deal with the persistent connections right so we have before our application is just request response right what is the state or update the state now we've got persistent connections that come in they want to get that initial value of what the state is but they also need to register themselves for updates right somebody else is going to change that value we need to broadcast it out to all the people that are on the web socket or long polling connections all right so here's a couple of snippets that we're going to add to our application at the top you can see the multiple event loops that we're going to use and this is part of your initial Nettie configuration you can see the change to our pipeline there so rather than that singleton application that we have we're going to swap in this protocol choosers it's just going to look at the request and figure out which handler needs to be swapped in later and you can see an example of that at the bottom snippet from the protocol chooser that just says oh it looks like this thing is a WebSocket and I'm cheating here i'm just using the extension on the URL but you could look at headers you could look at whatever piece of the request you want if you're dealing with something that's not HD HTTP maybe look at the first few bites of the request or you could just run this on multiple different ports and say oh this came in on this port I'm going to use this protocol okay and here's what our application looks like now one of the things to notice is that there is no longer anything HTTP specific in here and that's because we have to deal with all these different protocols but those different protocols don't really matter for the semantics and business logic of our application so instead we have you know our set state gets stayed and get the marketing stats and this new stuff at the bottom that is for the persistent connections to register themselves as needing a broadcast when the state changes and removing themselves as well right so if that WebSocket closes we only want to we don't want to waste time trying to send it or you know keeping that registration around in memory and these are this is the what the persistent connections are responsible for doing is implementing this trade and what this allows the application to do is just say okay there was a state change I'm going to loop through all these registrations I have and broadcast that state change without needing to know what the underlying protocol is right so our application no longer needs to know about web sockets or long polling or anything like that we just need handlers that implement that trait and this is a top piece of the WebSocket handler here right so this is our our definition of different protocols the WebSocket framing is going to look different from the response to a long polling request okay so here we have the end of round two we have capacity for a large number of clients like in this example I'm using 32 event loops you will be able to get that 100,000 number with that many event loops we have a protocol independent application interface right so this is this is some pretty good design when longer have HTTP semantics inside our application code and similarly the protocol independent broadcast interface is also a good design the application doesn't need to know anything about what we're how that broadcast happens it just performs it and we have a mix of protocols we have some that are request response and some that are subscription for broadcast and so I have all the code for this up on github and I'll the link will be posted at the end if you go and look at it we also have a lock around the state anytime you're doing an update or a read you're going to have to grab that lock so we can do the the proper marketing stats and fulfill the definition of application that we talked about in the beginning all right so now we're going to make this multi-tenant the app has gone really well and one of the other engineers noticed that we could take this idea just you know synchronizing boolean values and apply it to a bunch of problems you know the Boston office wants to know if there's a baseball game New York City office wants to know if Eber is currently bend in New York City and you know maybe HR loves it so much they want to apply it to their PDF to word resume converter for some reason but the problem the accounting department doesn't want to pay for another cluster for each of these applications and they're observing that you know the latency is a little spiky on the machine that we have and the cores are still a little bit underutilized it's a very technically savvy accounting department okay so we've got our our generic interface for separating out the protocols already we did that in the last round but now we're going to need a generic application interface we're just synchronizing boolean state we're going to need a place to define the application parameters right so some of these applications might have subtle tweaks or behave a little bit differently so just some way to to manage that complexity and finally a way to manage the concurrency for each of these applications so to do the define the application parameters you don't have to do it this way and in my example code I just kind of hard coded them in but the way to do this is to have some sort of external queryable data store and have a mapping for some indicator in your protocol to which application to run and especially if you're your multi-tenant applications like in this example exhibit more or less the same business logic like for this one we're just synchronizing bullion's for all of them they just have their their separate booleans they mean different things and for for us since we're using HTTP as kind of like the initial bit we can do a URL to application mapping so you could keep that and you know some key value store or file system or something like that but in general it's good to keep that outside the process so you can update it without having to restart the process which will kill all those persistent connections so this will let you you know deploy the is uber banned in New York City without disrupting the is there you know is it foggy and Soma ok managing the concurrency is going to be a little bit tougher for this right so previously we had just that one lock around the state and now that was kind of ok you know we had safety we would occasionally get things may be backed up waiting on that lock but that's not really going to work for this multi-tenant application blocking on that lock blocks the underlying net e event loop which previously was kind of ok because all of the traffic was going to the same application if that application was slow kind of expect all the traffic for that application to be slow but now we have multiple applications we said part of multi-tenant service is to keep applications from interfering with each other and so these two things kind of conspire to give us a problem which is that I blocking one of these event loops is blocking a kind of a random subsection of traffic for other applications and if you have one pathological application that's really slow it's pretty easy to quickly block all of your event loops so this simple locking strategy is not really going to work for us anymore instead we're going to use acha acha is an active library actors are very lightweight concurrent entities they process messages asynchronously using an event-driven received loop sounds kind of similar to the neti description but we'll get into some of the differences in a second quickly for discuss actor systems an actor is a behavior function and a mailbox that is typically a queue and that queue contains the messages to process for that actor the actor system is responsible for successively calling that behavior function with each individual message from the mailbox and typically you can multiplex many more actors than threads over over a single thread pool so you know we might have a thousand of these applications running on like 32 threads or something like that and one of the guarantees that we get from the actor system is that a given actor is only processing a single message at a time so they behave in a single threaded fashion even though the actual thread that they're running on might change from message to message but that's fine for us all we need to know is that we're serializing access to our application state we don't actually care which thread it's on so the quick comparison here between akka and nettie first of all they both could be used by themselves to solve this problem however they have kind of their own separate strengths can make it a pretty good idea to combine them so Nettie is extremely tuned for network performance they've really put a lot of time into minimizing allocations and tuned properly and you can minimize copies in your pipeline and you have that that nice kind of single threaded access to your channel it's always on the same thread so you can use thread locals if you have some state associated with it whereas acha is a bit more generic you're never guarantee which thread your code will be running on only that it's only going to be running on one thread at a time but it's also a lot more dynamic you can decide like okay I've got this stream of events coming into one actor and I want to filter out some of them and send it to this other actor I and you you get to manage your concurrency and kind of more interesting ways that way with Nettie you can do that it's a little bit more difficult though to move event streams from like 11 event loop to another one and to reorganize them according to properties of the messages rather than which channel they came from ok so let's talk about actually using these actors in our application so the goal here is to not block Nettie's event loops right because we don't want to block that random subsection of our traffic so what we're going to do is when we decode some message off the wire maybe it's a get right we want to know what the current state is we're going to send a message into the actor we're going to encourage in the mailbox we're going to use what's called the ask pattern that is part of acha and what that really just means is we're going to include this message it's going to return a future and when the actor responds to that message that future will be completed and then we can continue on from there and it's the same bit for actually changing the state we're going to send in a message with the new updated state that we want to set and we're going to wait for that future that is going to say okay the state was changed you can respond to the client now one other interesting piece is that remember we have to register those persistent connections if they want to get broadcast events whenever that state changes we need some way for our actor to do that so those registration events are also messages that we send into the actor there's a couple of ways to handle this you can send in some object that knows how to get back into the neti event loop you can perhaps wrap your clients in other actors and then the the actor that you're using for your application can just send a message to them they're sort of trade-offs to do both of those one of them is managing context switching actually depending on since you're not guaranteed what thread the actor is going to run on you may want to guarantee that it's going to go right to the neti threads all right so this is um what the message is that we're going to send to our application look like Scala really helps us out here with guaranteeing that these are going to be immutable as well as we're using a sealed trait so that means our actor can say given that we've got one of these boolean sync app messages we can write a match statement that will have is guaranteed to handle any one of those messages that we could get there's a little bit of weirdness with type safety and akka but once we have one of these we can do an exhaustive match and this is what the internal bit of the actor looks like so here we're saying okay we've got one of these boolean sync app messages which one is it all right it's a get we're going to handle that and that last line there is sending the state back and that's going to complete the future and that's going to let the the code that sent that message in its going to get the on complete for the future and it can go right back into the net e pipeline and send that response one thing that we don't have in here is locks around the state so we're guaranteed by the actor system this thing is only processing one message at a time so as long as it encapsulates our state it's giving us that serializability that we wanted for our application all right so this is what our server looks like at the end here we've got multiple protocols the request response or persistent broadcast we have a whole bunch of Nettie event loops so that we can handle all of that traffic and then we have the actor system so that we're not blocking the event loop when we are interacting with our application if we have that sort of pathological slow application that mailbox is going to grow but other applications will continue to be able to handle traffic and a couple of extra results that were not initial goals but turn out to be pretty nice we have that interface for new protocols right they have to send those messages and just implement that trait if they want broadcasts so that's pretty easy if you have some new you know you want to add just a raw TCP socket to this program you can do that we have the protocol agnostic interface for each application right our application just deals in those messages that we saw you're sending them and waiting for the futures which makes things easier to test testing your application logic no longer requires anything involved with any of the networking you don't need any of the WebSockets you can do it all synchronously and local which is great for business logic tests and finally because that actor is guaranteed to only be processing one message at a time that log of all the messages that we process is an effective audit log for our application if something goes wrong there's some bug in our logic and we have that log recorded somewhere you can go back and play it over and step through all the different state changes and see where things went wrong and as long as you have that log you can do it over and over and it's guaranteed to be the state changes that your application went through and in addition that that actor thing I forgot to mention with it the actor system is this is a good place that if your application is like decomposes a bit more than just the single threaded state machine this is a good place to do it you could have a bunch of like child worker actors that you could fan work out to or whatever hierarchy you want that's kind of the nice thing about being able to filter out some events and send maybe some of them to one actor and some of them to another actor you really get to choose based on the semantics of your application rather than what socket that event came from and that's it so this is the github link there has the code I for the for the application that I wrote that I stole the snippets from you can find me on Twitter and if you're interested in this kind of stuff firebase is always hiring the time for questions yeah oh sure yeah actors across the network as a firm machines different processor and if you do what do you do with the delivery alright so the question is if we run our actors across different machines and how do we deal with most once guarantee so we we actually don't although you can you're going to have like anytime you have you're going to like queue up some like essentially a job queue at that point you're going to be dealing with that problem the the best I can offer is try to make your operations item potent so if they get if they're received multiple times it's okay but fundamentally you're going to have to choose between it most once or at least once see yeah so how do we handle that akka does not guarantee delivery of the message so we have our akka is contained locally so it's not technically guaranteed but the set of things that can go wrong in that case tend to be like catastrophic things like the JVM crashed rather than more typical failures like you know the network network connection flapped or something like that also we have we've done a bit of logging with the the dead-letter mailbox so if a if you send a message to an actor and that actor no longer exists for whatever reason it gets sent to this special mailbox called the dead letter mail box and we log those and look for them can you speak to ye um let's see so the question was why we chose Neddy versus spray ah so can't speak too much to it because it was before I got here but it's I the neti folks have been great in terms of what they're working on is tended to align very well with what we're working on they're really like just focused on performance they've they've done some kind of interesting tricks especially for something on the JVM like they have reference counting for their buffers which is weird in a garbage collected language but they're actually doing things like that for good reason so we've been pretty happy with it Neddy's never really been a bottleneck for us I don't really have much experience with spray so yeah so if you know why do you suppose so oh sorry in real life do we expose the Java process directly to the web or do we proxy it so we expose it directly to the web although we have a bunch of other stuff in there to kind of manage some of the common problems you can have with doing that in particular ssl nettie recently I released an update where they forked a small piece of Tomcats native module and you can use openssl from the JVM which is pretty nice because it's much much faster than the standard JVM ssl alright