Scale By The Bay 2020: Dean Wampler, Ray: A System for High-performance, Distributed ML Applications
[Music] thank you very much it's uh my pleasure to be here back with all friends at uh scale by the bay um if you have never seen one of my talks before i tend to put in pictures in the backgrounds of hiking trips um this was a mistake uh it was you know it's actually a blurred image but i love the way it came out so i kept it uh these pictures are from uh the gila wilderness area in new mexico uh so yeah i work at domino data lab i previously worked at any scale which was the company that was spun out of berkeley to develop ray and that's how i got interested in it i think you'll find it a really interesting toolkit for doing distributed computing what i particularly like about it is from the end user point of view it's extremely easy to use whereas maybe it doesn't give you quite the you know bells and whistles and flexibility of a toolkit that you may know like aca or something like that but you know it has a nice compromise for you let's say the 90 case so anyway uh briefly domino data is um provides a enterprise platform for data science it's it's designed to integrate a lot of the open source tools you know and love like spark spark t-shirt um uh you know ray python r matlab you know deep learning frameworks and so forth to give you an environment to help you manage all your resources your experiments your deployments and all that stuff so anyway if you're interested in more check out dominodatalab.com we are hiring scholar developers if you're out there looking to love to hear from you and also i know that any scale is probably hiring for people with experience with ray or interested in ray for python and they also do have a java interface up now besides python if i forget to mention that later okay so anyway let's uh make the case for ray you know why was it created uh what is it designed to do and you also see the api it's actually a very concise small api but it does a lot of things for you then i'll talk about the uh at least one of the machine learning libraries that's built on top of ray how it leverages right under the hood while mostly hiding ray uh behind you know a domain specific abstraction but i'll also talk about ray from microservices it's a very general framework and i think it has some interesting implications for how we build whether we call them microservices or services i think you'll find that an interesting bit and then i'll finish with just a bit about how you could uh learn more about ray okay so first you know why ray what what problem is it solving uh you know why do we have it well it kind of uh emerged out of a couple of uh problems that we faced in the industry this chart on the left is a log chart of basically the compute required to train neural networks going back you know about eight years or so and in fact the um the alphago zero is actually a toy compared to some of the natural language processing uh neural networks out there that are on the order of 200 billion parameters and require millions of dollars of compute resources to train so it's just been this huge exponential growth basically 35 times each year while moore's law has only been growing by about you know 2x per year of two years or so so we have to go distributed that's the only way we're going to solve this problem even gpus are not going to just get us out of this conundrum the other big trend is that data science has in large measure migrated to python you know there are other languages i just mentioned a few of them in fact but a lot of the stuff being done today is done in python and so you know bringing these two together people really need compelling powerful tools for distributing not just on a single machine like multi-threading but going over a whole cluster and there aren't too many ways to do it that are easy to use and i think ray is actually the easiest to use i've ever seen yet it gives you really good performance as well okay so it really was cra this is the key thing it was created by the people who needed it these researchers at berkeley and what they wanted is something that did a lot of this heavy-duty lifting of you know for distributed systems but they didn't want to have to think about it too much because they were trying to do research in artificial intelligence and so forth so they just wanted something that worked that we kind of stayed out of the way most of the time and i think you'll be kind of amazed at how well it does this but just a few more details to uh to set the stage um this is kind of one way you could think about this the sort of processes you have to do in a typical environment that's going from you know modeling your problem to deployment i won't go into all the details here but all of this stuff typically requires distributed computing to scale there's a lot of tools that are available for various facets of this problem uh and it's kind of an engineering challenge to glue all this together one of the goals with ray is maybe if we build the right distributed framework under the hood then all of the other tools that we need that are specific to these different problems can be written on top of ray with relatively you know little uh effort and you have kind of a unified framework underneath uh and so that's kind of the promise of ray um there's actually um some more libraries now that you know like third-party libraries are starting to use ray like uh hugging face horvath spacey those are three examples of using ray in different ways under the hood to get distributed scaling all right so let's actually see how it works um and you'll be kind of amazed at how little of an api there actually is but how concise and powerful it is so one of the things they did right i think is start with concepts we already know from python and really any language that hopefully you'll make this will make sense whether you're using scala python java whatever but how can we extend those concepts to give us distributed computing uh without a lot of mental overload so i've got a couple of python functions on the left one of which is going to return a numpy array numpy is a popular library for manipulating arrays of data in the python ecosystem you know i don't really care how it's done i just want a method that returns an array and then i'm going to have another method that adds two of them together and you'll see why i'm doing this in a second very familiar kind of python code if i want to turn this into a distributed task which is the ray terminology then i just have to add this decorator at ray.remote and now i'll be able to run these things as distributed tasks and what that actually looks like well i guess i should say if you for completeness you're going to have to do a few imports and you're going to have to initialize ray now if you're just on a laptop or you know server it will you know spin up array locally and let you use all of the cores and your environment that's actually a non-trivial thing for python but if you're actually passing the right arguments to this method it can connect to a big cluster and the largest i've heard of or there's a company in china that's running like 10 000 nodes of ray to do various things all right so uh the other difference is that you don't call this method the way you normally do you you call it with this dot remote invocation and the main reason for doing this really being python's a dynamic language it'd be easy enough to intercept the call to make array and have it do this magic but having the stop remote is great documentation you can read this code and very quickly see what's invoking ray versus what is just regular synchronous python code this thing is going to start this remote task somewhere in a cluster here i'm saying it's going to run on node 1 somewhere and it's going to immediately return a reference this will be our handle you know like a future where i'll be able to get the value that's computed at some point in the future i'm going to do this again this time maybe it schedules on another cluster node and then finally i'll take the two arrays that i get back and i'll uh add them together with this add array invoked in the same way as before so if you look at this code just kind of glance at it it doesn't look that much different than what you would write if it was just a single threaded uh you know program using numpy but there's just a little hooks that we've added here that give us this distributed computing there's a couple of other things that are going on automatically for us that make it a whole lot easier to use uh finally i should say how am i going to get stuff out of this reference i call ray.get and that will block until this ref 3 is available but here's the important thing first of all um i can just go ahead and call this add arrays and ray will internally schedule it when it actually has the data that it needs available from the first two tasks that were invoked so it handles that sequencing for me i don't have to put in logic to wait for these things to be ready uh and then just you know call them call add arrays when it's ready now the other thing it did for me is add arrays takes two uh numpy arrays but i actually passed it two references so ray will automatically unpack those and extract the values and do that for me so just you know minor tweaks to how i would have written this in this uh synchronous um you know single threaded way i've got something that can now run at scale over a cluster and it does all this sort of magic behind the scenes but they have to do with typical and distributed computing so that's pretty nice and that's what attracted me to right beginning in the beginning now one thing i haven't solved at all with this example is like managing distributed state in any way um and here again they took something we already know how to do which is classes which is the you know the typical way in an object-oriented language that we encapsulate bits of state uh you know manage updates and accesses and so forth so i've got this very simple class that is going to counter every time i call increment it's going to increment this value internally something's very easy to understand if you know pretty much any object-oriented language and to make this an actor and they actually use that term which you may know from akka and some other frameworks airlang for example all you have to do is add this annotation again add ray.remote or decreator i think's the correct term and now it's going to be distributed over a cluster and the state is going to be managed wherever this thing is running now there's one other thing i have to do unlike python i can't just reach in and read the field values of an actor i have to have an accessor method to do that read or write so i've added this getcount method if i want to retrieve the count later it's also returned by the increment method as you may have noticed and then we call it the same way we construct actors with dot remote we call methods with dot remote and we can also call ray.get with an array of references so it will wait till they're both finished and then return you know the first one is going to have the value one the second one will have the value two because that's you know i called the first one it incremented to one return that state the second one's going to have this return the state of two so that is really it i mean that is like 80 90 of the ray api there's other methods for doing a little bit of configuration you can manage those references a little bit more carefully if you want and so forth but it really isn't much more than that so it's a very concise api that has a lot of power behind the scenes which is great all right let me talk a little bit about machine learning libraries that are based on rey and leverage ray and i have this chart earlier that showed you several of them the one i'm going to talk about is my personal favorite which is the reinforcement library called rlib it's probably the biggest and most widely used of the ray libraries that come with ray and you can get to documentation about this if you go to rl lib.io i kind of glanced uh glossed over earlier but the ray dot io is the website to find out more about ray all right if you've never heard of reinforcement learning before you don't know what is concisely this is what it's all about there's some environment some world that an agent is trying to negotiate in some way the agent is going to observe the state of the world and make a decision about what action to take next then observe the new state of the world and whatever reward was received by that action and the goal is for the agent to maximize the cumulative reward you know we call these things episodes or we go through these steps and by the end of the episode we want to maximize the reward now how's the agent going to know how to do that well you what you end up doing is running these sequence these simulations or whatever over and over and over again and you train a policy under the hood that tries to get really smart about making the right actions given the circumstances and one of the ways that you can train those policies is with a neural network but that's not actually required for this this is actually not a new idea there's variations of reinforcement learning that have been around for a while but it really got popular a few years ago primarily because it was used to beat the world's best go players it was used to achieve expert gameplay in a number of games like atari games and it just really uh proved the benefit of this powerful technique for a certain kind of sequential problems when you're playing a game or you know walking through a an obstacle course these are sequential processes and you have to learn how to navigate them so quickly some of the other applications of reinforcement learning it's being used for things like robotics and autonomous vehicles training them how to operate training like this robot a hand to place books it's starting to be used to optimize and simulate industrial processes this i think is one of the interesting cases where it's breaking out of these examples that are maybe kind of limited for most of our interests like robotics and games are a lot of fun but you know day to day the kind of stuff all of us are doing in enterprises maybe not quite as relevant but when you start modeling the the workflows in the enterprise the assembly lines and i picked a state-of-the-art picture of an assembly line here this is the kind of stuff that's also being used there it's been used for system optimization like optimizing power usage and hvac systems you know network routing hopefully a little better than this a picture of a crappy uh wiring job that somebody did one of the interesting recent examples has been actually using it for ad serving and recommendations recommendations is a problem we've known how to solve for a while we've been there's these things called collaborative filters and so forth many of you may have actually used recommendation engines in your day jobs well now it's starting to be applied to uh a reinforcement learning starting to be applied to this problem and there's two reasons for that one is that for very very large environments like think about netflix and its user base and its catalog uh the the computation required for some of the classic methods is pretty intensive um and also it doesn't really account well for the evolution of your preferences as uh you know you change and as the catalog changes and so forth this is a system that's actually proving very interesting for modeling these problems and the recommendation or reinforcement learning that is and and doing a better job at these these tasks and of course financing you know they use whatever the latest hot thing is to try to optimize trading on something that is also time oriented which is you know the stock market and so forth so let's talk a little bit more about go as a reinforcement learning problem just to put a little bit more detail on what these terms are in the upper right so in this case the observations are the board state you know where the stones are on the board the actions are where to place the next stones and interestingly enough there's no immediate rewards as you play the way they modeled it is you either win or you don't so there's only a reward at the very end whether you win or lose but they used a neural network to model this and found when they analyzed it that it tend various layers um looked at various levels of granularity of the state of the board okay back to reinforcement learning with ray with rleb it's designed to be not only very performant by using ray at the bottom but also to give you lots of abstractions for building new algorithms this is an area of hot research there's a lot of different algorithms they're structured in very different ways and they also interact with simulators and environments in different ways that's what the top four boxes are i won't really go into those now for time's sake but it's designed to be really flexible for building these kinds of systems this is a night chart that lists some of the algorithms these are like state-of-the-art research algorithms that are available in ray um i won't really talk about them in more detail for for now but actually all of these links and i'll make the slides available we'll take information about them and you can also use this stuff in sagemaker and it's also available in azure if you want to play with these things and you're already using those environments okay back to uh the motivation for ray a little bit you know i mentioned that these researchers are running into problems uh they want they needed to scale to clusters they you know had just huge compute requirements to train new reinforcement learning algorithms and other uh problems in machine learning um and so what are some of the forces that drove the evolution of ray well one is that you know you you're gonna have to run these simulators or game engines or whatever fairly efficiently in at scale and over and over and over again so you need something that's pretty flexible and not just kind of pitch and hold if i can put it that way into one kind of compute like you know a sql query kind of compute problem or a stream problem or you know i'm going to read a lot of records of the same format and do mapping and filtering and you know transformations uh very different kind of uh memory access and cpu access patterns in simulators and the agent that we're training you know whatever we're using for the policy could also be something that isn't a neural network but in this case we actually need to do all the neural network stuff efficiently actually the way uh ray aura lib and other array-based libraries tend to work is they make it a lot easier to to work with pi torch tensorflow and so forth rather than try to write those libraries themselves so in a lot of ways it's an integration system as well as writing custom algorithms all of this has to be done over and over again as efficiently as possible to leverage your resources you know with very heterogeneous compute and all of this kind of drove the kind of um design and intent of ray but also to make it very generic for not only these kind of problems but others that we'll look at in just a second and then finally it is very performant these are some benchmarks of you know hand-rolled implementations of some of these reinforcement learning algorithms versus the ones that were implemented in rey which has to be a lot more flexible and and the performance has been very comparable for them right let's talk about microservices for those of you that aren't actually doing reinforcement learning how is this going to be useful for you well there's a lot of and there was a great panel earlier today about microservices there's a lot of reasons why you build them all i'm really going to focus on is the last bullet point um that sometimes you need to manage things separately you might you know the typical devops thing is i have a team that owns microservice one not only do they implement it but they run it and same for you know team two and so forth well there's an interesting problem we have which is that um we always end up running multiple instances of these microservices both because no one machine has enough resources to to meet the capacity we may have and we're already running other stuff on that machine so we have to go distribute for that reason and also we need them for resiliency and failover and so forth so that if i'm running one instance of the microservice on one machine and it crashes you know that could be a lot worse than if i have uh failover capabilities but it means that we and this has been one of the challenges with microservices is that we've got all the stuff to keep track of now in production even though we might only have three microservices in this trivial example or three different applications let's say i've got you know a lot more instances than that to take care of well one of the cool ideas about rey and whether it's rey or something like it i don't want to oversell this as like a magic bullet but i think this is a really compelling idea that because rey is is handling in a very reusable generic way the distribution of work over a cluster what we could actually do is go back to a model where we really just implement one application instance but behind the scenes ray is scaling the work over a cluster to give us the resiliency and the scalability we need so that we're not so much explicitly managing instances anymore we're back to a more logical view of how we've got three things going on and those three things are in some sense transparently scaling for us over the available resources so i think it's a really compelling idea that we can do this it dovetails nicely with kubernetes because ray's very fine grain recall those tasks and actors i had at the beginning you know very small things they nicely fit into into pods virtual machines or physical machines however you want to do it all right finally some uh just some notes if you want to if you're interested in learning about ray and trying it out um if you're already using some multi-processing multi-threading libraries like job lib or multi-processing.pool in python there's some drop-in replacements written in ray where you just change the import statement and now you're actually scaling up to a cluster so it's a pretty nice way to get some immediate benefit while still using the libraries that you already know how to use and if you know what async i o is it also integrates nicely with array that's another way that you can actually drive the array apis with asynchio once again go to ray.io to find out you know everything there is to know about ray blog posts and all kinds of stuff i actually wrote most of the tutorials at anyscale.com academy and these are actually freely available on github so you can try it out and actually learn about rlib if you want um and then there's array slack this link actually goes to a google form to sign up for the race slack that's the best place to get uh help in the from the community and there's but there is also a google group i don't have it on here but uh check out any scale as well they're actually starting to are beginning to offer rey as a service if you're interested in just having them worry about how to run it so to conclude ray is really a very interesting take on the need for a easy to use but powerful and capable distributed computing environment that really minimizes the amount of work you have to do in python and also in java now to scale stuff to a cluster so i encourage you to check it out and if you're interested in you know deep learning or you know reinforcement learning check out some of these other libraries as well okay that's it for me um uh here's you can reach out to me at this address or spam me on twitter and please check out dominodata as well and uh i'll the slides are actually already at mypolyglotprogramming.com website if you're interested it's a longer version of them but you'll find them there and i'm happy to take any questions for the next five minutes and then we'll flip over to the spatial chat you