Taming Large Scale Deep Learning

SF Scala: Evan R Sparks, Taming Large Scale Deep Learning

Recording: SF Scala: Evan R Sparks, Taming Large Scale Deep Learning

[Music] so tonight you know even though I am i run a startup this is not at all a sales pitch I wanted to keep the talk technical you know developer friendly something that that I thought would be of general interest to the audience so I'm going to talk about you know a couple of lines of research work that both my my co-founder me Tom Walker and a bunch of our collaborators have been working on as well as some stuff that I mean I did around sort of scaling up deep learning and so hopefully you guys will enjoy it it's uh it should be a fun one so the title of the talk taming a large-scale deep learning I'm sure you all saw that and so you know why why are we here what do we care why do we think that deep learning is important well you know we've all read sort of the the front page of a bunch of these newspapers and we're seeing these incredible applications that we thought of as science fiction like five years ago coming to the forefront so you know autonomous vehicles or driving around the streets down here I'm a little scared of that but okay we've got these these speech driven interfaces that that have come into our daily lives from Syria to Alexa and so on massive improvements on on machine translation and a lot of this is fueled by these new technologies and that we all know now we we think that there's really this untapped potential for a lot of further advances we really think that every industry can be impacted and influenced by embracing these technologies and using them but you know it's it's really hard it's really hard to build deep learning powered applications and I think we all know this so we've got these complex workflows and and everybody's workflow is a little different but I'm gonna try and distill what we think of as a as a canonical deep learning application development workflow so before you do anything of course you've got to think about your problem as an m/l problem as an AI problem you've got to set up an optimization problem that you want to go solve with a mathematical objective so that's sort of table stakes but once you've done that you've got the the math kind of set up and you kind of know what you want to go optimize now we collect our data so we this might be data that we go collect as a byproduct of our business this guy poconos Kujawski used to call this data exile and and it might be search logs or something that I'm getting by running my search engine but it might be something that I have to go pay for I've got to go pay some tato provider or I've got to pay some humans to go label it that sort of thing then I'm gonna choose my favorite architecture for the task that I'm solving so this might be you know a CNN for some computer vision problem it might be some LS TM if I'm doing text modeling whatever so pick one that's appropriate for the problem you're trying to go solve and then we get to training and training takes a long time and you know we've got great tools like tensorflow Karras Cafe etc that help us declare our models and train them and but within this we've got a couple of subtasks it's not this model if we have training the models and actually getting them to converge and this might take days on our GPUs and often times we're dealing with things like down sampling that that might help us tackle the complexity of training we might say take 10% of our data and train a model just to get a sense of whether I've got signal or not before of course scaling up to the whole thing and then never is it the case that the first model I tried is the thing that works I've got this process of hyper parameter tuning I've got to fiddle with the knobs associated with my model you know things that might affect the model architecture itself the number of filters and my convolutional layers or the number the depth of the model or so on or things that might affect the the actual learning objective the regularization and so on and so I've got to fiddle with the knobs for a while and and then finally I'm done well no it's going to take me weeks to do this and it's a manual and operating process and then I've got to refine and I've got to iterate so it's never the case that that first model is is you know statistically sufficient if it is I probably would have been able to self solve my problem with a lot simpler methods and so I might go back and decide I need to collect new data or different data I might have to choose a different architecture and I've got to put on sort of my ml model or hat for this piece of the the workflow underneath all of this we have a bunch of complexity to deal with right so we have data management issues we have cluster management issues on the data management side big data sets or the fuel for these models right so we've got multi terabyte data sets often sometimes even bigger we also have this data management problem around what models have I even tried in the past what versions of what software and and things of this nature where we think of this as sort of experimental metadata that we're often not capturing or not capturing through very explicit processes again on the other side of things you know we we might be scaling up to beyond sort of the GPUs on my laptop or a rack somewhere too many GPUs across a cluster and I've got to manage these resources they're expensive increasingly you know nvidia will settle you one box filled with GPUs that's that cost $150,000 that's getting to the cost of one developer for a year and once you factor in power and maintenance it starts to be really expensive and so if these resources are sitting idle we start to feel really bad about ourselves as developers and making wanting to make sure that they're doing useful work further it's it's hard to get these GPUs to train together in parallel and make progress faster than I could with with sort of a single device and I'll talk about that more in sort of the latter half of the talk so I get through training and I've done this iterative process and I've developed my ml application and now I've got to worry about deploying it in production and I was just talking to you know here in the front front row and and we were kind of talking about this problem and you know it's often the case that the people developing developing our models our ml researchers and engineers aren't really thinking about deployment time constraints when they're thinking about model development they're thinking more about how do I get the best possible accuracy on this computer vision task and and not realizing that oh I've got to build a model that that runs in fronts on an iPhone 7 in under 100 milliseconds or something like that and so oftentimes we get this this gap between finishing thinking we're finished training our model and getting it ready to for deployment where we've got the job of some poor deployment engineer who has to go in and drop a bunch of weights or compress the model in some way before we're actually ready to service it as part of our application and so you know we think that that it's pretty important to start thinking about these things at design time further we've got very ad-hoc processes around how do we make sure our models are actually working in practice you know how do we monitor them how do we decide when it's time to retrain them redeploy them how do we version them and so on in production so hopefully I've convinced you that there's a ton of problems you know start to finish in the in this entire model development lifecycle and tonight I'm gonna spend a lot of time just talking about training and I want to you know make the point that while I'm talking about training there's there's a bunch of interesting problems to to worry about you know before and after training as well so first I'm gonna I'm gonna spend some time on this hyper parameter optimization problem and I'm gonna present to you some work from my co-founder of meet tal Walker and a bunch of other collaborators called hyper Bend and if you're a practicing data scientist machine learning engineer and hyper parameter tuning is a pain for you this is work that I think you will find really interesting and exciting it's a it's a novel algorithm inspired by some work than a meeting I did way back in kind of 2013 that he and he and others have really taken you know and very impressive and beautiful theoretical lends to understanding so let's start with a little bit of motivation here so let's let's really put the cost of deep learning into into focus so this is a recent paper from from a friend of ours at Google Denny Brett's an old collaborate of my collaborator of mine actually who works on Google brain and this is about Google's neural machine translation work and so they have this is a single application translating getting a state-of-the-art translator from English to German this is a really important task I'd like to be able to you know have the Babel Fish in my ear and have it be really good and so I think this is a this is a useful application but just around hyper parameter tuning and trying a bunch of models out and getting these knobs just right so that they could get state-of-the-art performance they will spent $250,000 on this problem and to put that into Amazon dollars Amazon will charge you about 90 cents an hour for these for these GPUs on the cloud and so 225 K to train a single application and that's just with GPU time that's not the human time that's not anything else that's kind of the stakes of what we're talking about here and that's one application and you know one you know a few months of research and this they're Google they can do that they've got all the money in the world but for most organizations that's that's a that's a hard thing and so one question is is can we improve this by you know some hopefully in order of magnitude or more and so let's let's talk about this hyper parameter optimization problem in more depth so I'm going to start with a data set and in this case you know our favorite computer vision data set M&S recognising digits and I've got some model architecture this model architecture is something that I might have you know pulled out of young lagoons tech textbook or whatever but this thing has a bunch of assumptions embedded in it so it has these parameters that affect the architecture of the model the activation functions that you choose to use the number of unit units units in each layer the number of hidden units and regularization is another parameter that we might want to tune this is this is really meant to be an illustrative example as we get to bigger and more complex models the number of parameters goes up and the curse of dimensionality really starts to kill us so the model space is divined by these hyper parameters and you know one particular setting is you know these knobs okay so this is meant to look like an old old you know audio kind of piece of audio equipment where I'm messing with the dials to get the sound just right as Alexi did for us earlier in the evening and so we've got one one particular setting of these hyper parameters and I'm gonna feed this in with my training data set to our favorite black box solver call it tensorflow or whatever and I'm gonna get some model out F hat that is this sort of you know converged model here for for that particular setting of the hyper parameters and then I'm gonna feed a validation set that I haven't seen at training time through my model F hat and I'm gonna get some notion of predictive error and so this is the thing that I want to do so I've got you know 95% accuracy or so for this particular setting of my knobs or five percent error on this this particular training set and now you know the task is let's enumerate a lots of lot of different settings of these knobs and pick the best one that I can find right so one with maybe two percent error and the question is how do I do this as few times as possible with as few resources as possible to get to a good answer quickly so we want to efficiently identify these high quality hyper parameters that particular setting that gave me the the least errors is what we call high quality and so we measure efficiency in the amine and others measure efficiency in this work through a quantity of resources consumed so how many GPU hours do I spend how much wall clock time is going into these things how many dollars maybe am i spending on Amazon in order to get these these numbers and quality is going to be a user-defined quantity in this case predictive error but it could be anything depending on your model and and anything is is a little bit vague but I'll get to that so existing methods for hyper parameter optimization fall into the broadly two camps so first is some kind of random search that doesn't really pay attention to the structure of our hyper parameter space so in this case you you look at random search as we look at we think of hyper parameter space as this kind of two-dimensional kind of contour plot here where the colors represent maybe our quality metric and so the goal is to find the things that are that are as good as possible in this case bright red and what we're gonna do is sample randomly from this space and grid search kind of looks like this a little bit different but you kind of get the idea and so we're gonna just run a bunch of models to convergence and then find the ones that look best adaptive methods things like Bayesian optimization are are what they sound like they're adaptive so they might start by sampling a few points from this space and start to develop some idea of what the what the surface looks like and then with that information starts to sample points that are kind of closer to something we might consider a global optimal things that that look better and over time they they end up finding you know the these good areas and in ways that are hopefully better than what we get out of random so kind of adding some formalism to this we're going to assume that a problem has D hyper parameters and that we're able to sample up to n hyper parameter configurations and so there's really kind of two cases that are important here so one is a case where and the number of samples we get to take the number of hyper parameters configurations we get to try is exponential with regard to our dimensionality and in this case we can say nice things theoretically about the performance of random search so we can with enough hyper parameter configurations we can kind of come or cover the space and get a good good sense of things and it turns out that in this regime at activity adaptive methods for this work really really well now there's another case that I would argue is much more important as we start thinking about applying these cases to deep learning is that n is linear in D so if I if it's gonna take me a day or a week or longer even sometimes to sample a particular point in the hyper parameter space and I have these deep learning models that have tons of hyper parameters associated with them fifty hundred sometimes more I'm probably not going to get that many shots at goal I'm not going to get to sample that many different hyper parameters and I'm certainly not going to get to do it exponential with the number of hyper parameters that I have so in this case we can say theoretically we can talk about how random does and we we say that it doesn't cover this space it also limits out apptivity adaptive methods really struggle in this regime so that's not good so this is as I've said becoming increasingly common as we start thinking about this with respect to deep learning and so the key idea with behind hyper band is this insight that maybe we can in some way down sample the then the amount of work we do for a given problem to evaluate more hyper programming or configurations on the same budget so that's the sort of key insight behind the algorithm and the way we we we leverage a special property of deep learning models that allows us to do that and that is that many of these learning algorithms are iterative so we can often check our progress with these algorithms kind of partway through their execution and decide kind of early whether or not we're gonna continue to voting resources to them so let's make that a little bit more concrete yeah [Music] yeah so I think there are a lot of sure absolutely so the question was there these companies like Sig ups that do a lot of this kind of thing in order to solve these problems I won't say anything about sig up we haven't compared to them directly I will say that a lot of these companies do focus on this adaptive kind of methods that I talked about which is is it more difficult to use in the setting where as I mentioned DS is kind of limited or n is limited by D so I you know we have a bunch of Imperial empirical work comparing to Bayesian methods for these problems and we tend to do pretty well and I'll show a little preview of that later and I can point you to some blog posts and papers if you like so let's take this idea of down sampling iterations the kids are somewhere close to the global optimum yep and importantly we can't we don't know very much about the function mapping D to this this target quality so we can't take derivatives necessarily we can't use kind of traditional a lot of traditional optimization methods we're kind of in this black box setting and so we open up the black box a little bit as well as I'll show you here and so the the idea is that you know these algorithms are iterative so it's if you if you run you know machine learning models in in practice before you you get tools like tensor board that might show you something like this the number of iterations that your model has run for versus the predictive error of that model and you'll see that you know in this particular illustration these things are are decreasing mostly and they kind of converge towards some global hopefully some global optimum and in this graph you can think of each line as a hyper parameter configuration so one configuration of these D variables and and you know we would want to find the thing that minimizes this particular predictable in this case it's the red line that gets there at the bottom purple is pretty good blue is pretty good and then the stuff on the top not so good and so the way we think about this is we say you know really the the the stuff at the top right here that's really wasted iterations that's work that I shouldn't have done because I'm just training throwing away GPU hours and models that are never gonna get better than then sort of the things in the bottom here and so that's that's sort of the key insight that we that we leverage in when we use this algorithm hyper band and so really that we you know the idea being that we should focus our effort here on training models that that might get get us better predictive error and it would be great to know to know the shape of these curves ahead of time but of course we can and so one one obvious thing that I asked myself as an engineer and every time I see a new algorithm is what could possibly go wrong with this like there's good there's got to be a bunch of issues so there we can talk about these things in mathematical terms so we can say that these sequences could be non monotonic so they might not strictly decrease over time so my my training loss might kind of bounce around a little bit it's not always going down and they might also even be non smooth they might bounce around a lot so they might these curves in practice might look something more like this than than what I just showed you and there you know obviously there's a lot of noise kind of early on in the process before I get signal later on and so one of the key research questions that was asked at this paper was how can we safely discard a a configuration and so I mean and and others spent a bunch of time and spilled a bunch of ink understanding this from the context of that active learning so they generated a new novel downsampling approach to kind of safely make this Explorer versus exploit kind of trade-off it leads to state-of-the-art empirical performance so you know run on hundreds of data sets and tens of different model architectures and so on really you know does a lot better than than these adaptive methods which were previous state of the art it's also got this night set of properties that in a lot of ways it's provably correct and and so let's let's take a look at what you know a typical result might look like in this space so say the task is we want to do a hyper parameter tuning on an on a neural network and it's a four layer convolutional neural network on this SIF are ten data set that is kind of a classic data set that people use for classifying tiny images the network itself has eight hyper parameters that that we want to tune over and it's for image recognition and so kind of the classic method the this sort of easiest method that you would pick for this problem is random search and over some number of iterations random might look like this obviously there's no labels on the y-axis this is really meant to be illustrative an adaptive method usually gets to a better answer somewhat faster so you know the adaptive methods are actually doing their job but what we see with hyper band is often a curve that looks like this and this has been borne out over and over again and the real you know the reason for this this to happen is that we're able to consider a lot more configurations in a shorter period of time than we are with the adaptive methods so the adaptive methods run these models all the way out to the right edge of these curves and then and then make some decision Hyper been kind of very quickly throws away things that are that are not looking good and so empirically the speed ups tend to be around 50 times faster than random with with hyper band in terms of number of resources consumed about 10x faster over adaptive it also has this nice property where it's a nicely easily parallelizable algorithm so you can throw it a lot of GPUs in parallel and it's and it's happy to keep going it often leads a much lower final variant a final error in in empirical settings and and much lower variance than you get with these existing methods so in a lot of ways it's it's it's a lot better and so then the question is is what can we say theoretically about the performance of this algorithm and so this was really I think where a lot of the lot of the you know kind of academic heavy lifting happen and I take no credit for this this at all but the author's frame this is a multi-armed bandit problem so if for those of you not familiar multi-armed bandit s-- is this idea that you walk into the casino and there are a hundred slot machines on the floor and you know ahead of time some of these slot machines are going to be paying out at a higher rate than others but you have no idea which ones and so multi armed bandits is a way of analyzing what's a good strategy for what what would what should i do what's a good algorithm to follow to find those those high paying slot machines as quickly as possible and so there's this nice theoretical framework around this around regret minimization and so by framing this as a best armed identification problem we can ask things like given n configurations how many resources should we should be required to find the best one and there are a lot of other kind of cute' results that can come out of this so with that i'm gonna switch gears away from hyper parameter optimization i think you know if those if if those of you some of you got really excited about that i'm happy to point you to papers and and and talk about it and now I'm gonna switch much more to sort of the system's east side of the thick of the shop yeah sure absolutely sure so quite often optimization you're doing it on the one that is just a very nice breeze so you've done some position right sure so a paraphrase of the question for the video so the the question if I if I understand it is you're solving I prefer ammeter optimization problems but you're not doing so in sort of a vacuum you've done a bunch of other the same thing before are there opportunities to leverage that data so I think the answer is absolutely and there's a subfield called meta learning that that looks at this particular problem you know I think that there's there's a very natural kind of warm start way you could extend an algorithm like hyperventilate and the adaptive methods are really good at using that sort of prior knowledge to to give you a sense of where to sample going forward it doesn't directly I can I'm happy to talk to you about it offline as well yeah so hyper band the algorithm is is is is freely available there are actually more implementations of it than just the ones created by the authors so I encourage you to check them out we internally at determined AI we have our our own implementation with proprietary extensions that we've made since and you know happy to talk about those as well yeah so the question is how does auto ml compared to this so Auto ml is really about model architecture search and is kind of a I would say more focused on the on the fact that we are training deep nets and that that points in the hyper parameter space are somehow related to each other so as we grow a deep net by a layer maybe we can leverage something that we knew about this the slightly smaller network to to bootstrap our way into it to good performance that stuff is is super fascinating I think it's really you know so far the papers you see it out of out of Google and stuff I mean the problem that I addressed early in the slide where it's like 250,000 GPU hours to to do a hyper parameter sweep the numbers I see there are like an order of magnitude or more higher so actually doing Auto ml and practice seems to be a really hard thing and a lot of those tasks are frankly pretty trivial so far I think it's a really exciting Avenue of research though yeah classify it as a problem we're gonna pick this one I think this one I think this can be very finicky deep learning is often optimizing continuous space yeah so how can you make a cure article guarantee the best single parameter given a continuous space I've ever seen bounded parameters but saying a single best primary in continuous States state search yeah so we we typically are analyzing these things with bounded parameters rather than then in infinite ones there are some in the bound I mean we don't reach yeah yeah so there are some continuous extensions to the classical multi-armed bandit stuff that that are leveraged by this work I'd be happy to set you up with my collaborators to talk about that in more detail reflects in the jail group [Music] the approximate complexity of the algorithm to find the best end I mean it's basically linear with with D so it's it's pretty pretty reasonable cool awesome so great if there are no more questions about hyper parameter optimization and I I think it's a really exciting topic too I'll switch to kind of the distributed training part of the talk so I'm going to talk to you today about this this paper we wrote called paleo and this one I actually helped write so I feel a little bit more more comfortable directly answering questions on it but this is again with my my co-founder of me Tom Walker who's who's both with the company and a machine learning professor at CMU as well as Henke who's a who's a computer vision student at UCLA who's who who honestly did a lot of this sort of nitty-gritty really hard hard core work here so paleo is this performance model for deep networks and the the kind of birth of paleo was kind of again picking on our friend any Brett's here it was really birthed out of this this idea that you know hey it's it takes you know oftentimes days or weeks to train a single model out to convergence and I would love to be able to throw more than one GPU at my problem to to train my models password it seems like parallel computing is is is a very natural way to solve these problems and so let's let's ask ourselves you know is that is this even a reasonable thing to do and so the idea was okay combine this deep learning with parallel distributed training throw a bunch of computers at the problem it takes take us from weeks to you know just a few minutes to train our models and and we should all be able to make a lot more progress right and so I meet and I started looking at this question for almost five years ago at this point and so we were at Berkeley working on ML Lib in Apache spark and you know we built a bunch of the standard sort of textbook machine learning algorithms that you're asking your logistic regression your ALS you know and so on and into that system and you know the next question everybody started asking us was where's the neural Nets where's the deep learning and so we were sitting pretty close to the cafe guys and started talking to Evan shale Shellhammer and a bunch of other folks on working on that project about how do we you know how do we merge these things how how do we put you know deep learning and spark together and and it feels like a peanut butter and jelly situation right so we we sat down and we kind of did some back-of-the-envelope training on the what were then the state-of-the-art models at the time so we we looked at alex net in google Annette and we said okay let's let's take like the first thing you would do to parallelized training of these models in specifically data parallel distributed SGD and we said okay how can we how much could we possibly hope to speed this up this is before we ever did any implementation we wanted to get a sense of how much bang for the buck would we get out of here and we really saw it's a modest estimated speed ups here so I think the numbers you know at that time we were thinking throw you know 100 GPUs at the problem and we get at most like a 3/4 X speed-up which seems like kind of disappointing you seemed like a waste of electricity particularly in the context of hyper parameter tuning it's pleasantly parallel and all that so then in the intermediate time there have been six distributed deep learning libraries just on Apache spark that that have come out and so these are things from some of our close friends and collaborators things like spark net things from commercial companies you know cafe on spark and others and so we kind of said what did we get wrong here let and and started scratching our heads you know if these things are really you know people are investing resources in building these systems maybe we missed the boat on this one and so we've also seen distributed deep learning start to take off in general so tensorflow has distributed deep learning it's it's kind of hard to use but but you can you can build it right you can use you can build your own parameter servers and pin your variables to the right places and and and it actually works and I built I've built stuff in it before MX net is similar and and you know the blog posts that come out of these these big companies that are developing these these pieces of software are making some pretty big claims so tensorflow they say we see a 56 X speed-up and throughput training throughput for inception on 100 GPUs so not quite linear speed ups but a lot better than the 5x that we kind of estimated way back when M X sent they say 109 speed up for inception on 128 GPUs so pretty close to linear speed ups that's so you know is is that good and so we said ok let's let's take that back of the envelope math that we did before where we decided you know I don't think we could really get much from parallelizing alex net and turn it into an analytical performance model to systematically study how distributed deep learning should work and from a perspective of scalability and performance and so you know there are kind of two main axes here that we that were interested in so one is the computationally the computational platform so what type of GPUs do I have what kind of network is connecting these GPUs what about the the topology of that network maybe there's there's some communication patterns that that make these things better and so you know one configuration might be okay I've got an Alex 10 should I train it on 32 ec2 GPUs how much time and money is it going to take to do that versus sixteen GPUs versus 64 and and how do I understand sort of this space and then the other question you might want to ask is for a fixed architecture so I've already bought you know a bunch of GPUs from Nvidia and I've got them running in my data center and I've got you know kind of a fixed sorry that that was sort of the first question the second question is for a fixed architecture what platform should we use so I have you know an Alex net which that's that's one question right the second question is then the converse for a fixed platform so I've bought my GPUs which architecture should I use which architecture is going to be most efficient so is it an Alex net is it a networking network is it inception for training in kind of a distributed fashion if they all get pretty close for in terms of statistical accuracy maybe I just want the thing that's going to train the fastest on my data and so then you know obviously you know you would like to sort of systematically explore this design space and find the thing that's gonna you know give you answers faster cheaper etc so the high-level strategy when we built paleo was to say let's take an architecture let's take a model architecture that is you know Alec's net or resonate or something like that and say that this thing provides a declarative specification of the computation that we want to perform so it is it is you know description of that of exactly that and we want to then map this computation to a specific choice of computational platform so take his input our network architecture and a a particular configuration of GPUs and network connecting them and so on and come up with a model for execution time or memory or energy consumption or so on the details here are quite tricky so you know it might be that that a particular operation in your continent has a number of different implementations so if you're doing convolutions say you know there in the name of convolutional neural network so there must be a pretty important operation there it turns out there like a half-dozen different ways that you can actually execute a convolution and depending on sort of the size and shape of the convolutions things like matrix multiply might be the right thing to do or fast Fourier transform or so on and frameworks like ku DNN will actually make these choices for you dynamically at runtime so figuring out you know which exactly which implementation these things is going to be run sort of automatically ahead of time is is kind of a tricky problem there's also we also want to figure out a way to estimate how much network communication is going to happen when we're doing model updates if we're training our models in parallel and so this is you know okay we kind of know that the number of parameters in the model is a good way of counting that but there might be different communication strategies that we we decide to employ it's also the case that every deep learning framework is not created equal so tensorflow you know when it first came out had this reputation of being slow relative to you know a bunch of the other ones since since it first came out that gap has closed but some that were if some architectures are faster or some software frameworks are faster than others and so when we're developing a model for execution time we kind of have to have a software specific factor in there to kind of pad this estimate and help us get something that is that is more representative of reality and so there's a bunch more details available in our paper from from my clear last year you know encourage you to check it out if you have more questions here but but let's talk through a couple of examples so first example is this Alex net model and so this is the one that we started to look at five years ago and we said okay let's fix the batch size as we scale up to more workers so at the time it was you know folklore that you couldn't train Alec's net with batch sizes over 128 it would not converge ok and that was kind of what everybody told us in the vision community who tried this over and over again and so the other kind of property of this is that as you start scaling to more workers you're doing exactly the same computation from a from a kind of analytical perspective you're you're scaling up number of workers from 1 to you know 100 and your effective batch size remains the same meaning that you get the same answer as you would if you had run on on just a single machine and so I think that's pretty important and we'll come back to that in a little while but the point here is as you go distributed you wouldn't and if you're if you're worried about analyzing these things you it's kind of a nice property to maintain that that we get the same output as serial execution and so we model this out to up to 128 Nvidia K 80s we you know the fastest network you can get on ec2 20 gigabit per second Ethernet we're taking a data parallel approach to training the models and in this case we're modeling tensor flow with the batch size of even up to 2048 so so being generous and saying well if you could train up on a batch sizes up to 2048 with Alex now what kind of speed ups would you get and you know the answer here was something around 5 X so with a provably optimal communication strategy butterfly all reduce with with up to 32 workers that's that's the best feed up we get on this particular thing at least according to our model and I'll point out here as well that you know other communication strategies will take you you know do decently well up to maybe 4 workers but then they kind of dead an out and what you see what kind of an interesting thing that you see here is that you get scalability up to a certain point and then actually communication starts to dominate the time spent in computation and so that's hence the diminishing right sides sides of the curves so really finding that sweet spot is a hard challenge and edit again it varies based on the hardware you're running the network you're running and so on so then we said okay well what happens if we look at this from the perspective of weak scaling so let's eliminate that pesky batch size thing having to remain fixed as we add multiple workers so we're gonna fix a constant workload for per worker you know a certain number of items that were that we're looking at every time we train and increase the number of workers linear to linearly and so again we scale up to 128 K ATS same same setup otherwise but with just thirty two examples per worker as part of the this strategy and what we see is hey we get pretty close to linear speed ups but I I want to stress that the workload here is changing so that thing that I'm running on the right side of the curve is not the same thing as the thing that I'm running on the left side of the curve and you know empirically we don't think that the models on the right side are going to actually converge to useful answers in fact only the things kind of in the lower left corner are going to converge so that's a little unfortunate and so I just want to point out that all of these numbers are estimates so if you were to build a system that looked like this how would it perform that's that's our best estimate so now let's compare that to systems that people actually built so the first one is is a tensorflow blog post and this is from a couple of years ago Derek Murray at Google posted this he leads a lot of the open-source work on distributed tensorflow it's great stuff and they they say okay here's an inception of u3 architecture we're gonna have a 100 nvidia k 20s this kind of interconnect between them and we've got a parameter server approach to model communication and so on and so the Paleo estimates kind of vary based on the communication strategy we use and we think that parameter service that comes somewhere likely between this tree all reduce and and want to all kind of communication strategy and we see that that our estimates scale pretty well with with what they report in this tensorflow Punk blog post again we're getting pretty close to linear speed ups but this is with strong scaling so or with weak scaling that is we're increasing the batch size with the number of workers in our cluster and so it's not clear to me that that it would be really it would have been really nice in this blog post if they had also shown convergent plots alongside alongside their timing estimates so time to getting a model that actually gets to a certain level of performance and so on they didn't do that but we do a pretty good job of guessing how long it's going to take them to run this particular workload in in their system the next example is this is kind of a more exotic thing so just to kind of show you that this model is a little bit versatile so there's this great paper from Alex Khrushchev to the creator of Alex net from 2014 where he's talking about not how to paralyze model training across many computers but many GPUs on the same computer so up to 8 GPUs and he has this kind of cute trick where he recomputes things on certain GPUs to save communication bandwidth and so it's this kind of hybrid mode it's it's again weak scaling but but Alex can get away with it so the one weird trick set up the reported result of this ok well you know we we were actually able to pretty closely estimate the kind of speed ups that that Alex got by by doing this one weird trick thing but I you know the the other nice useful thing about paleo is that we're able to now compare what if how much how much how it was it for Alex to do all this work and come up with this clever new idea and it turns out it was kind of worth it so at the limit we see in the lower right hand corner here he took what we would estimate to be a 24-hour kind of train time for his particular model with 8 GPUs with it just kind of a vanilla data parallel setup down to we would estimate about 14 hours in in reality it took him about 16 hours and so that's a real savings you know something like a 33% increase in throughput and that's that's that's no joke he you know it's really a it's a useful it's a useful thing that he didn't hear but being able to say what it was this work worth it ahead of time I think even Alex would have appreciated that the key takeaways from paleo is that we're able to efficiently and accurately explore the design space of model architectures and computing platforms so help you say for a given model pick the best right number of ec2 notes to use or whatever and I would also point out that we should all be very careful about considering this this notion of strong or weak scaling with respect to to evaluating these systems in the context of deep learning a lot of the the reported results are on weak scaling and I just want to stress that that workload is quite a bit different than in the strong scaling one and I think it's it's on us as a community to start demanding that people also put convergence time plots and final accuracy plots on their on their in their papers when they're describing these systems and I think you know on a positive note I think paleo can really help us develop architectures new neural network architectures and learning methods that have nice favorable kind of computation calming communication profiles things that will do a better job of training in a distributed fashion and get to a good results faster so leveraging this tool we think is super useful so paleo we've got a demo of it available online if you want to you know mess around with a few different model architectures and and different assumptions around how fast is my network what kind of GPUs do I have it too is open source so if you want to use this to you know extend to your favorite architecture or or make some different assumptions feel free to go ahead and or if the code send us a PR and and so anyway take a look so just just to revisit the sort of beginning of this talk I talked about this kind of complex workflow of what does it mean start to finish training all the way down to you know from data all the way down to deployment of developing a machine learning model a deep learning model that we can get out to production I focus tonight really just on the training bit and you know as a as a company and this is the only thing that was gonna sound anything like a sales pitch I hope we really think that there's this natural separation between what a domain expert the machine learning model er should be doing with their with their time and what they want to be doing and that's thinking about the domain problem thinking about the modeling issues and then there's a whole bunch of stuff underneath it the system support that is equally important but something that we think should be owned kind of kind of centrally and managed by a system eliminating a bunch of complexity for for the end developer and so that's the kind of thing that we're building a determined AI alright so thank you I'm happy to take any more questions about the the distributed training stuff for Piper parameter tuning or just talk talk machine learning thanks for having me I hope people enjoyed this yeah so we we looked at things outside of Kahn nets when when we do the architecture evaluation we looked at things like LS TMS and and so on and so the the tools are pretty general in that that perspective they work on our n ends and so on we use image classification because that's the you know the one that everybody kind of kind of knows best yeah so we build a workbench for machine learning engineers and data scientists who are who are trying to get these deep learning features into production and you know the observation is that it takes oftentimes a team of a dozen really smart people with a million dollars in spend in edit Nvidia two years to get these things shipped and we want to shrink that time down to weeks or months that's really what we're we're trying to help people with and so you know we're we're working with teams you know anywhere from you know call it four or five Davis scientists up to twenty or thirty cluster sizes from you know 4050 do you can use all the way up to hundreds in really exciting areas everything from the life sciences pharmaceuticals and and and gene therapies to things in IOT and and and financial services and and so on so I think you know like I said at the beginning of the talk there's a really wide array of applications and I'm excited to see which ones are most impactful services or machine learning as a service or source we so we sell a software platform you know you primarily for folks that have GPUs on Prem we work in the public cloud too so if you've got a bunch of GPU nodes that are sitting there idle and not too unuseful work all the time give us a call see here okay which one okay yeah yeah absolutely you know just to get your opinion where is the current right now it seems like there's a lot of innovation also work along hard and there are these slices just statistically see our efficient anything yeah is is super optimization yeah so the I think the the way I'm hearing the question is hey the hardware is getting faster underneath us why should I bother you know maybe we're still in the in the Moore's law or something similar to it for this this particular specialized application absolutely so that that stuff is all proceeding it at a great place and I would I would like I would say there are kind of two interesting things here so first is you look at every new generation of NVIDIA GPUs that come out and the memory gets bigger and the flops get faster do we get like you know to do our problems go away no the researchers come out with a new grater you know later grater model that fits on in memory and so on and it's a version of Parkinson's Law right work expands to fill the time allotted and so I think that's that's part of what we're seeing here the other thing I'd say is that there's really interesting trade-offs that start to happen as the as the hardware gets faster so kind of counter-intuitively faster faster GPUs doesn't actually help my scalability it actually hurts it so the communication layers that we have underneath us the speed of light is kind of fundamental and it's it's hard to make networking go a lot faster and so there is a real I think opportunity as if we if we view network is basically going to stay fixed for the foreseeable future a lot of work on algorithm development and thinking hard about how do we how do we do this stuff in a distributed fashion so that we can learn models you know in a in a useful amount of time so that's that's kind of where our opinion of that yeah the scalability metrics really are meaningful yeah exactly so it's not meaningless right so weak scalability is really if if I've got to do batch inference on a trillion on a trillion images we scalability is actually a pretty good way of analyzing my system right so it there's no there's no relationship between the inference on one one image or another and spreading those out across a bunch of machines really will you know in kind of a MapReduce fashion increase my throughput right the issue is that when we're doing the actual training time we have this communication step that has to happen to coordinate amongst the nodes and that's where if I were in traditional convex optimization lab land I would not care about batch size I'd say you know use whatever batch size you want because I'm always going to converge to the same answer in deep learning where unfortunately our batch size is an important parameter controlling the speed of our convergence that's where this this kind of effect starts to come in and unfortunately I don't know of any really good theory that that relates the batch size to the the you know the error that you get other than a bunch of empirical observations that say you make back size to bigs in my mom my models don't converge [Music] Jeff Dean however gave a talk at sysml this conference we sponsored a few weeks ago where he made some hints about maybe they've got that figured out so watch google research and and and maybe in a year or so we'll we won't have this problem I think it's a really interesting set of techniques as I said I think it works great in this situation where we can we can really have we can explore a lot of hyper parameter configurations I think that in other settings it works sort of less well and but I think it's it's something that could like actually really be complementary to a bunch of the work that I presented so hyper band where at every round you do an a Bayesian sampling step is a very interesting algorithm and I know some people who are working on that from a research perspective right now [Applause] [Music]