Reactive.community: Ben Christensen, Reactive Extensions (Rx) at Netflix

Recording: Reactive.community: Ben Christensen, Reactive Extensions (Rx) at Netflix

can you hear me okay in the back without a mic yeah okay all right so today my intent is to give a little bit of idea as to how Netflix is applied reactive programming over the last couple years in the edge systems where I work and then give some kind of insight into how we've been thinking about what we would do if we did Greenfield like starting from scratch again and what it would look like for us to go fully reactive and then tell you a little bit as to where we're going in the future in the edge systems engineering and where what that means for our X Java rxjs and some of the other libraries that we're using and if you're wondering what on earth edge engineering is all about edge engineering is the the it's the group of teams at Netflix that manages all the incoming traffic from devices and you eyes so it's the it's the service gateway it's the public API is it's the the routing tier it's all those different teams that the build and operate all the traffic the powers all the devices out there but it it does not represent any of the actual business functionality it's all in the middle tier those are all the the things that we end up depending upon so it'd be great if we could always just start from scratch and do a greenfield development but it's incredibly rare more likely we think that our applications look something like this generally this is how engineers think of software that we feel is decrepit and old and not-so-great more likely though it's like New York City which is actually very sophisticated very mature is getting the job done but you've got the equivalent of decades worth of different like times of building des fitting things differently together you've got some new roads some old roads you've got some nice new plumbing some ancient plumbing and this is generally the reality of any successful system because it's incredibly rare that a system got successful right after it was Greenfield generally you've been through a lot of iterations we all want something looks a little bit more like this I have actually never seen one of these so if you see if you if you're aware of a software system out there that is representative of this image as opposed to the previous picture I'd love to see it but our goal at Netflix a few years ago is to try and take a few of these things and try and bring it and bring it back into the New York City picture get it a little bit closer and it actually was a very simple problem we were trying to solve we were trying to take single incoming calls and do a lot of things concurrently and the reason why we were trying for that is several years ago we noticed that the device traffic coming in that it was based upon the the typical restful model was very fine-grained and incredibly chatty anyone a request coming in would hit one thing on the back end typically it was actually like two or three would you like off get some user data and then do the one thing I did so for every one thing you did you actually had a tax of two other things and that was how all traffic in the front door behaved and we wanted to do something a little bit more like this where I could come in a little bit more coarse-grained and go and do eight or nine or ten things and better utilize the latency round-trip over the public Internet and we realized along the way that we needed to embrace asynchrony and probably something along the lines of callbacks and the three things that that meant for us as we were figuring out how to solve that is we needed to do service composition that was actually what we were looking for we weren't looking to be async just be async we actually needed to compose the the responses and the behavior of multiple different services microservices if you want to use that term but services and while doing so we had to take into account error handling I couldn't just have things blowing up all over the place and in a synchrony oftentimes error handling can be challenging and then developer productivity had to be taken into account we couldn't just ignore that aspect and say we're gonna solve this but in some crazy hard way that all of you are gonna have to go through some like arcane training and you're gonna lose you know everything gonna take twice as long to develop that that we couldn't have that be the answer and it did take long for us to realize that callbacks was not at all what we wanted to do all you have to do is go google anything about callbacks and like it almost always ends up at callback hell like there's literally web sites called callback hell and we didn't want that so we wanted something more along the lines of futures and futures as long as you're not using Java futures pre Java eight they actually are pretty great Java futures which is where we were at circa four years ago they suck they're actually they don't even deserve to be classified in the same space of what we're talking about here I'm not going to go into all the different types of features out there Scala's got great ones Twitter's got great ones guavas got a decent one job eight finally got involved in in the gut in the game with completable future it's good it's a little odd in some of its design but it functions the the thing that was unique in what we were I don't know if it's unique or not but it was clear for us is we actually have a lot of cases were it was an asynchronous request for a list of something and that became quite common in our patterns and so for us an incredibly common pattern is you get a list of list of movies and then for each list and then within that list for each movie you go them fetch a bunch of stuff and so we wanted to have an a model that could not just model the request/response semantics but also model the the iteration of the data that we were receiving and the this last one it's generally not a good way to do it with futures but it's a future of list of futures is actually semantically kind of what we were going for we want it to be able to asynchronously fetch something that would then asynchronously send the data back to us and so all three of those actually get represented by the stream type that concretely in our in reactive extensions is called the observable and effectively this led us to a point where we treated everything as a stream everything that we modeled as far as our interactions with services became a stream in how we thought about it and how we we modeled it and coded it wrote the code so instead of me using Netflix things like I often do am using orders and products and shipping status instead because virtually everybody has at some point messed around with a system like that and so in a synchronous world these three things would have ap is where you invoke them and you sit there and block and you get back lists of items or a single item and this is the the typical blocking API that most of us have written for a very long time and most of our code bases I imagine for most of you are still they still look like this and it's a lot easier that way actually it because it's how we think you request something and it comes back to you however the behavior of this is that you typically step through each item one at a time and when you're dealing with IO over a network boundary that typically is time that is wasted and it starts to pile up and affect two things your user experience latency so that they're perceived the perceived time for them to get a response back and also typically the the way that most of our systems work means you're really using your your infrastructure inefficiently so the reality for us though was that almost everything we talked to was synchronous i/o like this so we were doing inter-process communication RPC over apache HTTP we were hitting memcache DS systems and ironically even though memcache is phenomenally good at providing you async approaches to talking to them all of our clients were synchronous just because that's how they had all been modeled Cassandra was all synchronous blocking i/o to them and we were using Tomcat and servlets everywhere like that was just the way that everything was done so every single one of those is blocking and so the question for us is what on earth are we going to do within that worldview to achieve some of these things we were trying to do with a sink and so the reality for us is that we actually couldn't change a lot of of the ecosystem we had like there was no way on earth that we could say we would like to improve a little bit over here so we need to rewrite Netflix like that just wasn't an option not only was it would it have just been laughed at and we would have been like people would have questioned or sanity it just it would have taken multiple years and probably would have destroyed the company in the meantime and so we needed to look at what we could change what we could change was the service layer the the abstraction on top of all of those things was within our control and so we chose to leverage the fact that that was fully in our control and start to decouple ourselves from the the choices and architectural things that we couldn't touch now that's not in the most efficient way to solve all these problems but it was far more pragmatic and this is what I'm gonna start talking about now is if effectively how the Netflix hedge systems have been running for the last three ish years in this like mixed mode of part of it a sync and reactive and the other half of it all those synchronous things that still exist across most of the company and how we kind of pulled those two together to to achieve what we wanted to do without doing the impossible or just giving up so we created an async facade on top of a synchronous infrastructure and the two blue blocks here are the pieces that are all synchronous and they all stay the same so the one on top represents that we were still living within Tomcat and we were still living within servlets which is all blocking i/o and thread per request and at the bottom of it the big blue one at the bottom represented all the dependencies that we needed to talk to and for a variety of reasons the client libraries were not something we could go and change anytime soon and so we just had to deal with the fact that they were all going to be blocking synchronous i/o for a good long while so we went about changing that and the question is is how do you how do you get from the top piece to the bottom and so we wanted to go from everything being synchronous API is that get orders to having like a get orders async we didn't call them that but how do you that's what we wanted to do and so we went through our entire service layer and we rewrote it from the ground up with everything as observable api's we didn't leave a single method in there that was synchronous and blocking every single one became an observable API and it that meant that we started to write code that looked more like this we would have an async of an async method that we would invoke that would then get composed with in another async method call that all gets composed into an async stream my goal right now is not to walk you through what flatmap and those types of things mean due to the variety of things I'm trying to communicate today either you understand flat map or just park that on the side come up and harass me afterwards and I can point you to some places that that explain what flat map is because the first time I experienced flat map I had no bloody clue what was going on so don't worry if that's you right now but just think of these these things all being composed together and it kicks it all off asynchronously it all happens in unicorns and Happiness the the difference here is that it all just happens in so if I do that again that same example from before this time it kicks off and then it all snaps into place because it fanned it out and it allowed everything that happened asynchronously it fetched all that data and instead of it stepping through each one one at a time it allowed them all to happen concurrently composes them all back together and it hasn't changed actually the amount of work I've done at all but it has changed the perceived late latency for the user because I leveraged parallelism to go out and pull these all back whether it's happening on multiple threads or an event loop or whatever's orthogonal to this it's all about the perception to the user so how do we get from a blocking API to a non-blocking one and that's what I'm going to spend the next a little bit showing some of the very practical ways that you can take something blocking and turn it into non-blocking particularly when you can't change the underlying infrastructure because that's one of the biggest issues that I hear people talk often talk to me about which is hey all this is great and fancy and wonderful but the reality is I got JDBC in my system and that database isn't going away or whatever it is so I'm going to talk through some of the ways that that you can bridge those so one of the most simple ways is you literally just wrap the API and so in this case I'm gonna be using the RX Java syntax here for this and defer is the mechanism literally for just saying when this is lazily executed later you're gonna run the code that's inside here and inside there I'm literally going to go and execute that blocking call and just return whatever it gives me and then that would all be quite useless if I don't actually make it execute asynchronously so I inject concurrency by running it off on an i/o scheduler and so in rx Java everything any time concurrency is going to be added it's parameterised which means that it can always be injected through methods and overloads and a scheduler you can think of it in Java land is like an executor service except it's raised up a level so that we take control of time and we can control things like pit like thread affinity so that all the events coming through wall stay on one event loop things that executors do a miserable job at we use those under the covers in and make sure that they maintain the contracts enough streams so that everything one stream stays sequential for example but what it means is that you can control you can choose which threads you want to put anything on and so the i/o scheduler literally is just unbounded thread pool that will allow you to block something on there because you're putting the blocking i/o over there we also have computation schedulers which is a pool of event loops if you're doing anything non blocking you can put a new thread on you can run it whatever you want question schedulers no schedules are completely outside of backpressure they'll literally just queue up whatever you put on them back pressure is the is the responsibility of the stream itself and the and the operators so this get shipping status you'll notice in this example we didn't have to change that so this is an example of I've been given a third-party client library that I have to use and I don't yet have the the ability to influence them to change I just got to deal with it and this is a way of doing it and this is how we did most of it for years because trying to convince everybody to change just for us wasn't going to happen now any of you who have dealt with system efficiency or just like kind of squirming right now going like really you're just like wrapping everything in threads it's like yeah actually that's what we did and what we found I'll get to it later it definitely has a cost and overhead but because of what we were what we were dealing with here is the per the perceived user latency on things measuring tens or hundreds of milliseconds and so the overhead about adding extra thread scheduling events and those things was was worth the cost for what we were aiming for yeah it would be very similar to this there's a so the question if any of you didn't hear is if we were going to do it again would we change anything about how we did it there's one area that I would dramatically do differently it was a really dumb mistake we've been paying for ever since and it was actually I'll get to it in a little bit but it was the layer at how we hooked into the tomcat servlets the actual wrapping a background stuff there's only a few ways you can do it it always ends up being wrapping it with the thread somehow and so that has never been the problem for us we ended up with about three different ways that we do it just in different places that were a little bit more appropriate and we've been pretty happy with all of them in reality though we actually don't do what I just showed you because we do something a little bit more complicated for fall we use history for all that stuff actually so hystrix is our fault tolerance library and bulk heading and along for the ride comes concurrency and metrics and insights and bulk editing and circuit breakers and so we actually leverage that across all of our systems to both make them async and provide fault tolerance so what does it look like if we have a method that actually returns multiple things which is quite common for us in this case it's also modeled as an observable and the only difference it all looks the same except that instead of saying just one item in just do from now the observable representation here is not nearly as valuable as it could be if you actually had a true stream because all what's actually doing under the covers is it's gonna still get back one thing in one response and then just turn it into a stream and iterate it out and so from an efficiency perspective we're actually not buying ourselves much but what it does buy us is the programming model and so that over time as we change things the abstraction and how we program against it is it's representing how the data actually is correctly and then as under the covers we can change out how whether it's streaming or batch or or however we do it and it also means that when you when you start to compose things together it's honest about the data type you're dealing with I can compose a stream and then for each item go and get I'll compose against all the other things but for any of you who are wondering like is this really benefiting from how you stream data no it's not it we're still fetching it all is one batch request response and then just turning it into a stream in memory so this is the simplest possible way of bridging from blocking to non blocking as we literally just wrap them with threads so what about JDBC so JBC you have this get orders method and you got all this code now one way we could do it is before you just let it all happen and then you you you do the from on it however if you can get inside and re-implement that get orders method all of a sudden you can start to take advantage of true streaming responses and so this I'm now assuming that you can edit or refactor or add a new method to that previous method so that you've convinced that team who gave you that third-party library or it's your own and you're using JDBC so right here this is the way that you're returning a list of orders that you've accumulated all right that's not the most efficient way to do it if instead I returned and observable instead of a list now I created an observable and I do all the same blocking code inside except now inside my while loop I'm on next thing out the stream as I'm getting the data from the database rather than accumulating into a data structure and then returning that data structure at the end so now I'm one step further towards representing the data as it truly exists and I'm also not a allocating memory in these bursts and so one of the things that you see in systems a lot when you profile them is it we literally like in a function you allocate this huge amount of data you pass it over and then you like turn it into something else you throw that away and you just keep allocating and and reclaiming along the way the observable approach you never allocate any of that memory because there's a pipeline that just shoves it down and you just process each item as it comes through and so it's much better for like GC pressure and those types of things the error handling and completion error handling is important so this is one of the places the first places where I've demonstrated how the error handling is done so the terminal event on any stream is either a non complete or a non error so on air you can think of it anytime you're going to throw an imperative code on error is the way and errors are just events they're just an event on a stream but the there's special event they will terminate the stream you can't have anything happen after an error and you'll also see here this is unsubscribed and so any stream is capable of canceling early and so this is the part where because we're in an imperative block of code here where it's checking effectively the cancellation token as you're going around it if you didn't have that and you canceled you'd walk away but it would still be off running in the background and so this is the way to be efficient now this is where this is going to be the most complicated piece of code I show today I'm going to I want to show at least how you can get the backpressure stuff in there if you want because that previous example I just gave absolutely no back pressure whatsoever that will firehose the data at you and you have to be ready for it or not and if you have queues in the way either they're going to be unbounded or they will blow up if they're bounded so if you want to get the backpressure approach I'm going to use a generator function to make it a little bit simpler because if you do it but from scratch it's painful and so this abstract on subscribe at the top is going to give us a kind of a state machine to help us out so I create this thing and the first thing that happens is I want to have a state object of the result set and so when this when this thing is involved wouldn't subscribe to the first thing it does is it executes this lambda which initializes the state and that state is I'm going to get the database connection I'm going to prepare this to create the statement and then this is a this is for my sequel code here that's the mechanism you set the fetch size and these arguments for type forward only and everything what that does is it's actually telling the my sequel driver to stream results because off I think it still is the default that it actually batches everything up which isn't what we want to do so this is telling it to stream the results down the connection rather than batching them and then I actually execute the query and so I'm doing a really stupid query here the select star from orders and so if there's like 10 million of these things the thing is is what this actually could work because you're a streaming model and will support cancellation and so it will not blow everything in the memory at any point in time it will just stream them through the next lambda here is this will be invoked at the rate of which the consumer can drain it and so this is the point where rx Java and the reactor stream's model they kind of support this push-pull back and forth so as long as the consumer is fast enough this will execute as fast as it can possibly go and just keep on pushing if at any point in time the consumer is slower than the producer this will stop executing and it will effectively turn itself into something like in like a batch iterable or an async enumerable and then it will only run whenever the consumer requests more and so in this case what it does is it grabs the state which contains the results set in it and then it does the the next that moves the moves it forward one grabs a result on next sit down or completes as it needs to and because this state machine takes care of this for us if any one cancels we close the connection and that will get invoked either on cancelation or on termination of the stream and then I just ignored for brevity on the slide some of the try-catch stuff and then again because JDBC is still blocking I need to move all this off onto an i/o scheduler because I need it running in the background cuz it will block so we end up using basically these mechanisms from observable to allow us to bridge from blocking code to non blocking and it goes from very simple just wrapping up a method up to the much more advanced example I just showed you and everything in between so that's about how we wrapped at the bottom part here so the async facade down to the blocking all the blocking calls underneath but what about the top layer if I have to still live within a servlet world or a blocking i/o thread per requests world you end up generally with code that looks like this so this do get for anyone who's done servlets you'll you'll be familiar with this you get the servlet request in response and it's a void method and it expects everything to be synchronous and blocking if you just asynchronously do stuff and like walk away from this it doesn't work the the response will never get back to the customer it will close the connection just bad things happen so how do mix the two well I have to deal with these blocking calls but I still want to perform on my composition asynchronously so I'm not getting any efficiency gains like event loops or anything like that I'm still a thread per request but when that request comes in I'm gonna fan out and do all of my work on multiple threads so in this case it's that code I showed earlier I go off and get shipping status it returns unobservable it's getting the orders asynchronously for each order I get all the products and for each product they get the shipping status so this is potentially a very large fan out and it's going to happen all asynchronously when it comes back on to the servlet thread though it kicks all that off and then I'm I say to blocking which we explicitly named that way in our X so that you know that you are blocking now we didn't want to have like anything like that it's just for reach or something like that we very explicitly like you see that in your code you are blocking the thread like behind the scenes that's a countdown latch we are literally just blocked parking the thread right there and then we turn it into the equivalent of a Java eight stream which is not a sink or push despite its name or an an iterable with higher-order functions and we iterate over the responses the output and we write them out synchronously on the blocking i/o response for the servlet okay and then because we've moved back into blocking it will throw exceptions and so you're back into imperative synchronous world and so what we've done is we've wrapped all the imperative stuff around this block of async and we bridge back and forth between them and so what we've done right there is we've mixed the two worlds we're within a blocking system but we're fanning out all the the code all the i/o requests to do the work concurrently now previously I'd mentioned that there was one thing that I would have changed dramatically if we if I could go back in time so we our system is way more complicated than this we actually have a layer of groovy on top that all the different user interface teams can write their own code dynamically and deploy into our system they write their own web service endpoints for each device and I made a really stupid mistake I gave them the actual request and response references and gave them full control of the subscription lifecycle of the observable and that completely had has completely handcuffed us ever since to fix all kinds of issues if I could go back in time it would have saved us so much grief I would have instead just had them do a function that declared and observable and gave it back to us and that we control the subscription of that observable and hooking it into the i/o that has caused so much grief that that's the one thing in the last three years that I wish I could change otherwise all the other integration points have been quite quite good that's the one thing if you're doing observables like this you really users of the system should never call subscribe it should be like it all gets composed together and then the system at some point goes says go and it subscribed so the whole thing is just all composed together because whoever controls that final subscribe controls the full lifecycle of everything that's going on so to blocking this is the bridge between a sink and sink and its most common use is actually in unit tests and demo code but when you're like throwing something up on a gist just so you can show someone how to do something that's its most common use because you just need to block the main thread otherwise like Java just shuts itself down but it can be used in production systems if you know what you're doing just don't use it on an event loop okay just don't do that that would defeat the whole point so how did it go we added this to an existing code base and how did it go this is a really important question so it took us three tries the first attempt we were actually building the appletv endpoints we the first two were definite throw aways they sucked the third one still sucked but it worked and we did that there's three or four of us involved in the in those first like two months and this is kind of how we felt the whole time like what the hell are we doing like why are we putting ourselves through this because if like I take myself back in time all of us we had just been doing imperative programming for like our entire careers like this whole like functional thing off on the side a little bit like a sink experience is like yell like throw something an executor service and get a future back and block on it right so what we found is that the tech worked so the Rx and that's not because of us that's because Eric Meyer who invented this stuff over Microsoft his tech worked and our implementation of it worked we were still figuring out what it all meant so we learned some things not quite at that point but over over time as we matured so first of all we had to relearn a lot of our idiomatic approaches to how to solve things a lot of the things that we just took for granted over the 10 15 years most of us have been coding we had to like unlearn those a relearn new thing so an example is if you have some iterables or a raise or whatever and you iterate over it you stop doing that there's no such thing when you're doing a sync instead you have you map over something or you like you apply functions to this stream that just kind of happens underneath you and so you're like inverting the model and this took some time to get used to another one is try catch some really interesting code over time as people were in between phases and they were learning they'd write all this async code and they put try-catch blocks all around it and be like so it's not working like what's going on like because the because that method like was done like five seconds ago you're over here now you're like you're never gonna throw an exception over there it's all async right and so to learn that you have to like start to do your error handling as it's an event through the stream try catch totally useless on the outside of a stream within a user function that's fine so this is another one of those those things that what because we're working within a language and a runtime that was never meant for this we had to like you have to pay attention to what the hell you're doing with a try-catch the other thing is we had to invest in documentation and documentation was hugely important for what we were doing because it was all new to all of us and so we not only did we spend the time on the Java Docs but on we did wiki's and we had tutorials we actually paid for a tech writer and we've we've had him for the last three years on the project and so we actually invested money in it not just developer time but money to get a proper tech writer which made a huge difference in the quality of the Docs and even as we moved it into open source she stuck with it and it's pretty awesome if you're gonna do open source do it this way the develop you commit code and then he sweeps in after the fact and commits all the Java Docs after awesome so and like the diagrams there was an engineer on one of the teams I work with who one day is like you know we should do these so like I literally was walking through the office one day and like I saw these like printouts like like cheat sheets like you know like you when you study with like flashcards they had like the ten or twelve first operators like flat map like how does this work like who did those soon as this guy George and so I'm like whatever you're doing we need more of that but like did you put it up on this on the site and then that became a thing and now like every single operator has these things done as diagrams that visualize what's going on and eventually over time it matured into a website and lots and lots of documentation then the community got involved documentation has been key and we also found we just had to allow for time of mistakes for people my experience is that it takes an engineer who and by engineer I mean someone who's who's been around like ten ten plus years of development so they already know code and systems and operations and stuff and they're moving into this new model two to three months about two to three months for them to truly get it not just so they can like hack something together but so that they like have that aha moment and they've changed their thinking and they're like legitimately able to code it and debug it like debug it if you can't debug the code that you don't get it yet unit testing and debugging have been these are these are important areas to understand to so first unit testing async unit tests not nearly as hard as it would seem so here's an example of an observable and this one is asynchronous the way I can tell it's async is that it's doing this interval thing with time so the anytime you have one of those in rx it actually has an overload that I can pass in a scheduler and I'll show that to you in a second this one implicitly under covers is using the computation scheduler so if I subscribe to that my unit tests will just complete okay it'll just complete and it won't even finish at the bottom at the top sorry this test subscriber comes right with rx as a first-class citizen for hoping you do testing and you subscribe to the test subscriber and then the test subscriber gives you helper methods like awaiting a terminal event it does the the countdown latches and all that stuff for you so it blocks the j-unit thread and then it allows you to do the assertions of what you've received and so that you can have a sync code that runs and then do all the the synchronous blocking stuff gets far more interesting than that though when you when you're testing things with a synchrony and time this test subscriber scene that you subscribe with that but this test scheduler this is where it gets really interesting what you can do something is really challenging when you're dealing with time and unit tests is it's completely non-deterministic so I've spent I actually spent three hours this morning trying to deal with some stupid race condition and code that I haven't figured out yet it's because on my laptop it works beautifully my codes awesome on my laptop when it hits Travis CI all hell breaks loose like like the tests are just failing all over the place so I've got some race conditions somewhere this helps with that stuff because this test scheduler actually does no concurrency whatsoever so I pass it in to the interval which needs time and now I actually get to deterministically and artificially advance time as I want to so I advanced time by 200 milliseconds I know guaranteed whether that took one millisecond or two seconds on a given machine that I will have emitted only one tick from the interval and I keep advancing the time and now I can see that I've received the five values out after one second totally deterministic this allows you to take something that has time involved and generally concurrency and make it all single threaded and deterministic you can use your debugger to walk through it like everything and all of a sudden become synchronous and then you flip it back into production with real concurrency and then all just behaves like you would expect with all the normal race conditions and all that stuff so that's incredibly useful a think debugging that I was still hard and it sucks running on platforms that were never built for this like the JVM they're stacked races have no idea what to do with this kind of code so lambdas sorry back back Oh No I don't know if any of you have done Java eight stuff yet I love lambdas but they make stacktrace is just ridiculously bad so that doesn't help and then you end up with like whenever you schedule something asynchronously like it completely loses the stack trace we have spent in order to denounce of time figuring out how to stitch call call stacks together to do it in any same way it destroys your performance by like orders of magnitude and all the insane ways to do it they even come close to performant are just like you're dealing with like C and assembly inside the JVM guts and like thus far we have not yet found one that is worth us doing in production this is something like I know that types a it's been doing some stuff on in this space I would love for Oracle just to solve this one for us it's this sucks so be ready for that and also the stack pollution just a really no good good way for this to keep e is a start-up that's working in this space trying to solve some these things we've spent some time talking to them there's a lot of work to do in this space of making stacks better and stack traces in async useful oryx java does try by putting in certain error messages are useful but it can only do so much with the with what's got debugging stepping through an async i mean it's really it's no different than if you're using futures and thread pools just when you're using rx you've got lambdas and like these sequences and stuff step-by-step debugging pretty useless you never actually step through what you want to step through because it it has no idea what you're doing cuz it's not imperative so you set these breakpoints you'll never actually end up where you wanted to know so this is a problem async code you just don't use debuggers very often you end up doing that so it's not happy and what like the the race condition I was debugging this morning I got lots of print lines for my code base right now so this is a reality of going async and an even greater reality of async plus functional composition and if you're thinking otherwise like don't be scarred well when you when you get there and realize that this is what it's like and yes that is the state of the art so async debugging is hard and it's a work in progress I'm hopeful that as companies like to keep Ian type safe and others focus on this and invest money and tooling it will get better like I know that type safe with scale has been doing some stuff recently on on the tooling and IDs in that space I hope that that becomes much more widespread as time passes so how about once it's all done so these observable api's worked well and we were able to go from call patterns like at the top where he had lots and lots and lots of very fine-grained calls to one like at the bottom or you'd have one coarse grain that did everything concurrently in the background on the backend much more efficiently we removed a lot of the redundant stuff that we didn't have to do every time and that all worked really well we replaced the blocking api's with observable api's and the layering of our system looks something like this along the top this is all of our clients and devices out in the field so this is like your PlayStation or your TV or whatever and the majority of those run JavaScript for the UI layer and C for the so it's got C in the bottom and then JavaScript on top and the most of them use rxjs to do all the function composition and asynchrony then we we've even now we're still using synchronous HTTP we're all a little bit I'll tell you where we're headed with that but it's still synchronous HTTP and then the groovy layer we use our X group E and then the observable API is in our X Java and then all those porcupines along on the bottom that's the the hystrix layer and we use that to bulkhead and isolate all of our backends that we talk to and that's also what gives us our concurrency and asynchrony is they're all pushed off into their own independent thread pools for concurrency reasons but also primarily for fault tolerance it is why we had those thread pools before actually we wanted concurrency we did that for both getting reasons so definitely challenges mixing approaches when you've got these two different ways of doing things in the same system it's really easy to like all of a sudden have something that was nice and concurrent all of a sudden all becomes so again it's really easy to get confused as to what's going on like honestly we have like over a thousand threads in that particular system it's kind of insane so too many threads it's hard to tune in shedload when you have a system that complicated and that like mixing patterns like that so that's a reality we've had to deal with it's really easy to accidentally block async code for two reasons one because you've got like a huge chunk your code base is still blocking that you can accidentally invoke but also because we're working in a runtime and a JDK it's api's where it's all intended to be blocking and so you gotta like be careful what we're doing here's just one example this was one that the top is concurrent this one is it is sequential and the only difference between them is really it's four characters in the method you're invoking so the map and flatmap if you don't actually get and grok what's going on here it's really easy for someone to say I'm just gonna map over it at the bottom and call the blocking get shipping status but you just made the entire thing sequential you just defeated the entire point of doing this and you shouldn't even be using this style of coding so what we did to help solve that is we we just made it so that we don't have any blocking calls in our API so in the inner layer of the API that you should be working with we just got rid of them all now because there's no modular system in Java 8 and earlier there was no way for us to prevent someone from going off to something else and touching it but at least we made it so that within the API space that they should be in we didn't have blocking and non blocking call sitting next to each other so that you couldn't just fat-finger it autocomplete into screwing yourself over and so everything was observable and so we try to avoid that case so this is a this has led us over time to the question is this should we go fully reactive as we explored that there's all dragons out there's like myths and legends and things like well yeah it's cuz it's better totally it's better the best way I got this I'm not going to name the company but there was a company who had just went fully to nettie and they went fully non-blocking and so I went over there one day I started asking him like walk me through it and at the end I'm like so why did you do this like that's a lot of work why'd you do it cuz it's better like but like what performance gains did you get like was it more like we're not really sure like the hell like that's not an answer that's not engineering so cuz it's better is not a legit answer and then there's also this like if you google any other stuff like this you'll get it's worse equally as often as it's better and so that's not helping the cause this is actually almost the answer it really doesn't matter what you do and you start digging more and more and you get like all these things and my conclusion is that theoretically doesn't matter what you do they're actually all equivalent depending upon the implementation because at the end of the day it actually is all just scheduling things and assembly somewhere on to a silicon and my CPU and if I could have a threading model that works better like a blocking threading model it works better than a reactive model with like event loops and stuff if I implemented them correctly and if you start to look at like the the c10 KC 100k and even now see 1 million which is about like the the like how many connections you can have on a box this is the only place I was able to find anybody who actually agreed that you had to go like non-blocking to achieve this otherwise they're like unless you're doing that it doesn't matter so this is a really hard thing for us to say like well what are we going to bet the future on and where are we building towards and we didn't want to be myths and legends and like different opinions and stuff like that so you know we did the worst thing you could do we wouldn't tried to do a benchmark which is disaster and so we wrote up a test that basically simulated the type of stuff we do you kick off two things in parallel and then that one does some more work and then some other work and it all comes back and it takes about 154 milliseconds for all this to happen like fear like that's the fast as it could possibly be now starting down a path of this kind of thing you get it wrong guaranteed it took us over six months to actually have data that I actually trusted and the only reason I feel like our numbers are even remotely meaningful is because we had the chance to work with Brendan Greg who has written a book this thick on system performance and the guy has forgotten more about computers and I will ever know in my life and so I had the the I was lucky enough that I got to work with him on this and it was fascinating to watch him do his thing like he would like warm up the machines and let them run for a few days to like get the jitter out of the systems before he was like actually trust that the data on them was legit he got a patch into Java 8 update 60 so you could attach profilers to it so like this guy is legit so as we went through it we certainly tested on this we were testing Nettie and Tom cats that's we were looking at so these numbers are only representative of the world view we were trying to do but they helped us a lot and they actually do teach us some things so this is telling us that our low test is actually saturating both of them that we that we were pushing both of them to near 100% CPU usage a low test that does not do that is it's wholly useless thing because then you're not comparing the same things this is where it starts to show what's going on so Nettie has lower CPU consumption the read per request it was taking less time than Tomcat per request now that actually doesn't mean anything about architecture that could just all be that it's code is better which actually is true okay now this one though the throughput Nettie starts to trounce Tomcat as you as you push it okay so the the RPS is able to get way better now this is just again because the CPU consumption per request on Nettie is so much lower than Tomcat so again we actually haven't proved in anything about the architecture yet just that Norman and those guys write really good code so the theoretical best possible time over there on the left is 154 milliseconds and you can actually see at low concurrency the two of them are pretty close to each other the Tomcats a little bit worse again just because it's not as good the code is not as good as Nettie but you start to see that as you you hit the right side it's there definitely is a difference and it's really severe when you look at the max latency so this is see average latency is totally a useless useless thing and I put it up here actually on purpose because if you're just looking at average you don't know anything about your system okay so because I'm sure not trying to take the entire night I'm not showing you like every immediately like every single percent all along the way so I'm just gonna go to the max latency and the max latency you can see that it just goes all over the map like you go from 900 ish milliseconds up to 2200 milliseconds so what's going on so the CPU instructions per cycle this is pay attention here that Nettie is actually doing more work at per cycle as you get higher under load the first time I saw that I didn't understand at all what was going on like it's actually getting more efficient the hotter it gets and we start to look at what's going on with threads you really start to see what's going on so remember that Tomcat is a thread per request and Nettie is running on event loops and so you can see that when the system is not under heavy load it the difference between them doesn't matter that much and it's really just that the the OS is context switching back and forth because any one thread is not hot enough for it to basically stay pinned but once you start to get hot Nettie really starts to get take advantage of its architecture and you can see that the Tomcat model hits the the peak and plateaus and what it can do so under light load similar this is where you start to cause less migrations you improve the instructions per cycle and the CPU uses per request with the CPU event loop architecture so why wasn't any faster for us in this case and Tomcat so first of all very clear Nettie just is better code in how much time it is spending on anything less object allocations method invocation all that stuff is better the event loop architecture does reduce thread migrations which affects cache warmth memory locality and instructions per cycle which gets back to an original metric of how much CPU per cycle which is actually what's affecting the throughput and so an we're and then on the thread pool architecture when we actually did flame graphs like where you actually look at where all the time is going we could see contention on the locks and the time that thread migrations were taking so lots more details that are not going into tonight but what we determined is that with current JVM and Linux kernels the event loops do have efficiency benefits now take a different JVM take a different Linux implementation different threading implementation nothing I just said would matter but with the current generation of Linux and in the JVM these things actually do matter and we can take advantage of that an architect accordingly so what we also found is it was actually a lot simpler to get Neddie to perform well the event loop model versus the the the tomcat one but it's certainly not easier - it's a it's a harder programming model to do but once you get it working way simpler to scale the thing I actually kind of think of it kind of like this Tomcat and Eddie like Tomcats really easy to get started with and then it becomes like that when you try and scale it in lots of different ways like the threading models are hell as you try and actually tune them as you put your systems under load what I found with the linetti like we spent two weeks tuning Tomcat we spent about 15 minutes tuning nettie like it was like how many threads do you want and like the event loops and we couldn't ever tuned it any more than that and so that was actually a big factor for me because operational complexity is a big deal in big systems and how your system fails is a huge thing so for us we spend in this enormous amount of time dealing with what the hell does our system do when you're passing the 80% CPU mark well we're scrambling to like scale up or something nettie literally just like yeah whatever I'll like speed up for you it gets faster as you push into like 80 90 percent whereas Tomcat like Falls to hell and like your whole system breaks that was a big deal for us so it does have benefits so where are we going with this and then I'm starting to finish off now if you're getting concerned that I'm going to talk all night this is the tail end now so I'm going to take you into some of the this is like if I worked at Oracle this would be where I put up that like that legal slide of all these forward-looking statements mean nothing and you can't count on anything so this is what we're trying to go with it so we are actively adopting Metis at in the edge systems not the rest of Netflix so if anyone ever asks to you they're not doing the rest of Netflix is not it's at the edge systems where we face the internet and we face the deluge of traffic coming in where that's a much more important thing for us to deal with we are actively adopting nettie right now and this means the event loop architecture non-blocking i/o and right now it's still just HP we're getting HTTP one working first we will then be leveraging WebSockets TCP maybe internally and HTTP 2 so all those things are coming for us we're actually experimenting with Aaron right now which is a protocol and implementation 2 by Martyn Thompson and Todd Montgomery it's actually built for like Chicago Mercantile Exchange and trading companies this thing is bloody amazing we can't even get close to this performance of this thing with Nettie and it's a protocol over UDP so it's not tea spear anything like that it's a it's an own custom protocol over UDP event loops no I'm blocking Iona stuff we're using it very surgically on one or two systems where throughput and latency is very very important to us like we're talking like median latencies of 300 microseconds for one kilobyte responses p99 nine of like 800 microseconds very consistent and that's an AWS networks so we're not even controlling the networks we can't get anywhere close to that with our other stuff so we're pretty impressed with this so far and are pursuing it reactive socket IO this is so new that I almost shouldn't be talking about yet but we are developing it completely in the open so if you were to go there right now you can see us doing all of our development right from the very first commit and like issue we've been doing it in the open without ever announcing it and a few people like quietly found it and got involved with us so lucky enough to be working with tog Montgomery on this talk Montgomery is he's done like protocols like 20 years so weird is that we're actually building our own network protocol for use over WebSockets in TCP and air on and it's multiplex duplex binary it supports request response streams with the reactive stream semantics channels bi-directional and it uses the reactive stream flow control so you can literally have the reactor stream publishers on either end and it all just composes together over network boundaries so it kind of looks like this reactive stream publishers on the left those are the the semantics for like these are the what a client might look like and our intent is to basically allow us to now finally have our network work with us instead of against us because networks are actually pretty good if you use them well just we never actually use them all well if I turn this into our ecstatic that so a single which is like a future except we didn't call it a future because this is lazy and futures are eager so it's but it's like an observable with only one item in it and we can optimize it differently the observables and this is how I would summarize what reactive socket is for us it's a formalization of the interaction models and the flow control needed for our applications built in such a way that we can put it over multiple transport protocols and so that and we're formalizing it as a natural spec and we will end up with a TCK so that we can have implementations in all the different languages right now we're working on Java and JavaScript someone has considered doing it and go and it's all just being done up on github whether it ends up being successful or not can't tell you yet but our early signs of this they're pretty positive just if you do go use the code don't use it anywhere yet that's the code that has all the race conditions in it I was telling about this morning but the protocol spec itself I'd say is like almost done the spec itself really will only iterate at this point if we find things as we're building and using it but we would we'd love to have your input if you want to actually come and get involved and we did it because honestly we couldn't find any solutions over things like WebSockets that met all of our needs and WebSockets if you want to use it it's got frames and you have to invent something so instead of us just hacking at it with clever engineering we decided to actually like legitimately define a protocol and so we brought in an expert got Todd Montgomery who's got 20 years of protocol design experience and he like the first day of working with him yeah here's the draft protocol like in his first day I'm like yeah you earned your money so rx Java 2 is under development now and really what it was just an idea up until about three weeks ago then another committer on the project he's over in Hungary David Carr knocked he literally like ported all of Arc's Java 1 to 2 with reactive stream semantics in like two and a half weeks the guy's a machine so we're in the phase now where we're basically going to spend many months just reviewing and iterating on API and the reason why we did this is first of all there's some design changes that if I could go back in time two years with the understanding of using it in production and stuff like that there's a few little tweaks that we would make that have prohibited like a few performance optimizations we could do like there was like some extra object allocations of things that just we couldn't get rid of them without breaking the public API this gives us a chance to do it Java 8 9 instead of Java 6 & 7 lots of goodness there if for no other reason than our sanity so we can write the code of lambdas and not want to pull rip our eyes out we're adopting the reactive streams interfaces natively and so observable actually implements the publisher which means that like this code here this handle request response the returns of publisher I can just return an observable from it and so basically the observable is just a concrete implementation of those types so this was always our goal the rx Java team was directly involved in in creating the the reactive stream spec and but arcs Java one was already baked and so we we supported the semantics but you needed a bridge for the the types so now we're able to natively support the types because we couldn't do that and be one without a breaking change and we're currently still just debating a few decisions we might take the observable type and split it into two observable in flowable we might let observable go back to being non backpressure and be just purely push because there are certain cases where that actually does model the world more accurately and where that's what you want and you can optimize it differently because you don't have the overhead of the request and semantics and negotiation and if we do that we would call the back pressure type flowable basically we're just it took me a while to adjust to that name but in Java 9 and JDK 9 the reactive streams publisher type is going to be if it gets through the process it'll be Java util concurrent flow dot publisher and so they're calling it flow so we'll be flowable if that's what happens single and completable single is just the stream of one completable as a stream of zero yeah we funny enough actually we find that we would really really really like to have that because we use observable void all the time to represent something that will emit no data but can complete or have an error and we just want something that can compose and so if we did that we would actually support all possible those four types of support for us everything we need to do rxjs JavaScript side is being rewritten from the ground up as well right now Netflix and Google are working together on this it's all about performance and debug ability rxjs three powers all of our stuff over devices but it just has some design decisions again made years ago that just kind of suck in retrospect and yeah pretty ain't this is like significant performance gains the RX Java one is only in like the most like there's certain cases you can achieve it here it's everything because of some fundamental things about how it works and then they debug ability so here's just a capture of the flame graphs in chrome of the new and the old and it dramatically simplifies the frames and what's going on and I'm not going to walk through this right now but it a big part of it is it gets rid of the trampolining when it's not needed and things like that it makes it a lot easier to actually debug what's going on so in closing reactive programming can be incrementally applied you can add it to your new york you can make it work and get benefit from it mixed codebase is our challenge though and you have to just be aware of that coming into it concurrency via a sync service composition does work and after using the more imperative style of concurrency in the function composition way I will always now go to the composition way and after three years of using it in production systems I can confidently recommend it as a good solution the Ark's Java schedules and threads can be leveraged to help you bridge them or like what we do we actually don't do that we just use hystrix for that because it kills two birds with one stone it takes care of our bulk heading and resilience and monitoring and metrics and all that stuff while also giving us the concurrency around those blocking things the truth though is that concurrency is still non-trivial and arcs Java does not trivialize that it is a is a tool to help you provide abstractions that compose it but you're still dealing with asynchrony and concurrency and all that entails so don't it's not a silver ball we never pretend it is and I I do see people over utilize it and I do see a lot of like well it's now like my hammer I'm gonna use everything with it just like any tool it's just the tool so hopefully this can help you pick a few pieces from this landscape and pull it back into your New York here's a lot of different resources reactive xio if you just remember one thing you can go there and find all that stuff and I've talked a long time but if you want to ask questions I'll hang out as long as you want to talk at me so thank you yeah yeah so none of those longer-lived things are really Americas we're still talking about transaction times that are within what you want a user to be waiting for so it's not like we have these things open for 20 seconds or something like that the the key thing is that instead of you allocating a chunk of memory let's say for a hundred items or a thousand items or whatever and then passing it along you never actually allocate that you you literally just pass each item down the pipeline as it comes in so all the resources for the database connection itself all are all are all the same it's what you're it's whether you are accumulating at all as one or not it's very similar to the difference in document or streaming style encoding and decoding so if I receive a big like JSON or XML document or whatever and I can do the document style where I just slow the whole thing in I do this huge decoder encode this huge glob in the memory and I'm allocating and releasing these huge chunks and you can see this affecting your system versus the streaming parser where it just flows through it and you just do one element at a time it's similar to that but yeah that all those concerns that you just brought up yes they would exist if if you've got transactional issues you know holding open connections along time all of those would apply that answer a question the question was do we consider akka or Scala so I'll answer a Scala one first three years ago it wasn't even an option for us we recently considered it as we were going down the green field rewrite and we decided against it we decided to stick with Java eight for two reasons one the primary goal of this rewrite is to apply lessons learned and have a system that we can operate easily with the existing talent and we basically all agreed that if we were to adopt a new language that we would get 12 months into it and need to rewrite again and and so and also we preferred the the the simplicity of just using the base platform and we felt that Java eight lambdas the were just the addition of lambdas is really all we felt we needed to use this programming model so that's on the Scala question on the akka one we have looked at it we're aware of it we have never chosen we have never been interested in pursuing it primarily because in our environment the primary gains that off getting for us is effectively the micro service type distribution we don't need or want that we already have a very sophisticated micro service architecture with discovery and resilience and scaling all that stuff and secondly we have a real allergic reaction to large with frameworks honestly that dictate how we write our systems and so we have chosen to stick with libraries that I don't have to use them in different parts of the code just because so those are the those are the reasons why we've chosen what we've done Roland and I have debated many hours about akka versus other options so I get we could talk about that offline if you're interested you had a question right here I think okay fibers I think fibers are fascinating but only if the runtime natively supports them we've explored quasar pulsar or whatever it's called we actually had its lead engineer names escaping me right now come to Netflix and present on it I think it's a fascinating engineering effort that he was able to get anything like that to work on the JVM but we're not gonna go anywhere near that type of thing just because the it's not actually solving the problem in fact the evidence of that is that you can't use the fibers on the JVM unless it's using non-blocking i/o anyways because they're not solving the blocking problem so if that's the cases like if I already have to migrate everything over non-blocking anyways Amazo just use RX cuz that it also doesn't solve the composition problem so I think fibers are great as a better thread but it doesn't solve composition I still need a layer on top like Rx or akka strains or something like that for composition I'm it's interesting to me that in Java 10 and on I finally started to see fibers show up is something that they're considering but honestly like I'm not counting on it ever I am interested in watching like what go and then we're doing so I think fibers play a role if it can be done well but I've never actually operated a system in production with it seems to be a lot less documentation around like you've got a socket see we try to figure out I didn't quite drop the question so you're saying like how do I create an observable from scratch to with some data source and what's the what's the premise of that question is it like the the complexity of doing a right so using our extending are using some other library because so basically the way you create observables like the is literally you observable that create and it just gives you the observer and you just on next into it and if you're if you're ok with no back pressure it's stupid easy so if you if you don't care about the back pressure issue you literally just on next until you're done and you complete and pre adding back pressure that answer was sufficient and it was done post back pressure it actually is not an okay answer I have found and it's one of the things that we're currently working on in in 1x and in 2x which is adding the equivalent of generator functions that give the state machines for how to do something that supports back pressure because it is honestly nearly humanly impossible to write back pressure like raw back pressured observables without having a bug because of the concurrency and everything about it and so if you're trying to do back pressured ones come on and help us figure out the right like we think we have it but we're still in their spare mental mode so that you have generators to do the state machine for you because it's not easy and this is actually the strongest argument for us splitting the observable and flowable type into so that you they represent what they are and if you don't need back pressure just go use the easier type I don't know if that answer the question ok so when you when you adopt the reactive streams model where it's got the the flow control built-in as soon as you've got flow control that means you've got bi-directional communication happening on the stream and you're only allowed to emit data when there's been capacity expressed to you well what that means now is I've got state happening on both sides and even worse it can happen concurrently so while I'm emitting I can receive further interest telling me that I want more and knowing how to actually like wiring that up correctly so you don't have bugs and you're not emitting when you shouldn't be or concurrently emitting or whatever is actually really hard to do well yeah so that code I showed you with the JDBC one that's the easy version of the the state machine because it's the synchronous model where it it just loops basically like on an iterable and that so if you're doing a synchronous iterable style reactive stream it's pretty straightforward and then the state machine is pretty easy because it's more like ask an operator you could think of it like that an async source is actually stupid hard to do manually and think of an async backpressure source as paginating over like straight a database results ten at a time and you say I'm interested in 15 well I'll stick with tens you you request ten five you're five of the way through that ten and the next request for ten comes in and you're doing that so that you're going and fetching them while you're still consuming so that you're efficiently like flowing and how do I kick these things off asynchronously especially because now they can come back interleaved and then make sure that they sequentially get concatenated back in and then flow with the with the correct flow control it's not easy to implement from scratch and so we think that we have an abstraction now that can become a proper state machine for that I'm hoping that that will get into the API within the next month and we'll leave it experimental for a month or two we annotate new API is just that we have the freedom to change them and harden that I honestly think that that's the missing piece right now that is still made observable creation way too hard because flow control is just hard so that hopefully that context helps yeah so the JBM ones all kind of originated out of our work in our ex Java and and they're they're basically adaptors on top of our Java other folks they'll have started to maintain them like Kotlin is maintained by cutland engineers there's nothing to do with Netflix for example and they have all the commits and release privileges on that project and Netflix has nothing to do with it outside of the JVM you've got like Swift rx Swift for iOS is now a thing and that's evolving you've got Rx net obviously the original one that's all Microsoft that has nothing to do with the github repos that we have rx they even have one for C++ there Ruby has never really taken off probably because Ruby's dying but it's never really gone anywhere and there's a few others in various stages of maturity but you can basically if you go to github Sacha reactive X you can see the Netflix is only involved in rx Java the new rxjs the previous one was all Microsoft so the new rxjs and we're working with the the Microsoft contributors on that and then rx groovy and we we kick-started scallop it ultimately was taken over by other Scala engineers in the community because we just don't use it enough to give a proper stewardship so thanks everyone