Scale By The Bay 2019: Anima Anandkuma, Next-generation frameworks for Large-scale Machine Learning
[Music] you know I unfortunately could not be here for the other talks but I looked at the schedule and it looks like a great and diverse set of talks on you know how to scale machine learning what kind of algorithm should be implementing what are the frameworks at all levels of the stack how do we improve efficiency and so today I'll try to give an overview of the different can you is a Nokia and overview of you know how to design algorithms with a view of different applications in mind how to design scalable methods but at the same time you know what are the trade-offs is it that the you know typically systems performance and algorithm design don't always go hand-in-hand right they it's easier to be modular and do them separately but if you try to integrate them there can be a lot of gains had and I'll show you some examples of that so you know I'd like to think of artificial intelligence and machine learning not just in terms of the algorithms we use but really about the data and the infrastructure put together right that's why I call it the Trinity because it's the totality of all this coming together that create successful applications many times we mostly think of just the algorithms when especially in research you know we are developing new algorithms testing them on standard benchmarks but in production systems and in the real world they'll be used in new use cases so no existing benchmarks data may not even be available you have to collect data and then how to run that on different computer infrastructure and what level of compute would we need and so that's why we need to think of all of us together right and this current deep learning revolution would not have been possible without the existence of large-scale label data as well as large scale compute from GPUs and it's the flexibility of deep neural networks when you combine with the power and parallelism of GPUs and the availability of large-scale label data that gives us the success of deep learning today and so given the excitement that this has generated a natural question is where do we go next right what are the next generation algorithms how do we design them how do we implement them how do we scale them up and there are some ingredients I'd like to give today so first of all the main shortcoming of current algorithms is that is extremely data hungry hungry for label data right and in so many domains we may not have enough data I mean even in computer vision which is the poster child of deep learning there is still not enough data of in many applications you know for instance for face recognition we've seen issues of fairness and bias when there are not enough examples from minorities as an example right so issues like that where in any application there's always going to be need for more data and the question is can we replace it with something else and that's where I'd like to think of incorporating different kind of inductive bias and structure into our algorithms when we run convolutional neural work convolution is an inductive bias right it means we are incorporating translation invariance in our network design the question is can we now make that more sophisticated what are the kinds of domain knowledge priors and structures can we incorporate in the algorithms and some examples that I'll show today are some are structural others are domain-specific so in terms of structural inductive bias you can think of right the structure in data itself being in forming the neural networks and convolution was one example of that so I'll show you how tensors extend matrix computations to higher dimensions and that can be also a strong form of inductive bias now if you recall neural network layers are currently all matrix computations and in GPUs we've parallelized them very efficiently but what if we take everything to more dimensions what if we design neural network layers where there are higher order computations and how can we scale them up how can we paralyze them what are the benefits so that's something I'll go deep into in this talk and naturally you can think of other kinds of structural priors like graphs if you have relational data then it's natural to use graphs as a form of Prior and graph neural networks have been very popular in the recent times similarly you can think of other ways to use priors although I won't go into details in this talk on my website you can see lots of examples where we can use other kinds of symbolic knowledge you know if you're doing processing on math equations or partial differential equations Sat solvers you know there's so much of bollock rules or axioms you can incorporate right everything need not be learned from data and so the more knowledge you can already embed and then the rest learnt from data it's that intersection that would be powerful so how do we have hybrid systems hybrid neural symbolic systems that can do both right that can reason symbolically at the same time they can you know give you numerical answers so combine the power of style of classical symbolic reasoning with data-driven neural network approaches so we have designed various frameworks to do this another important application is when we go to the physical world you know if you want to launch robots whether it is autonomous cars or drones or walking robots they have to satisfy the laws of physics and this need not be learned from scratch sure in a game like alphago you can have an intellectual exercise of how well will it learn from data right but in the real world we can't afford to just hope that it learns from data and they'll never be enough data to learn that so for robustness and full data efficiency we can embed the laws of physics as much as possible and only learn the unknown parts in one of the applications we showed how you can improve drone landing by only estimating the unknown aerodynamic forces which is hard to model but there are so many other things you can already derive from your model and specifications of the drone so those up you know applications can benefit a lot from reasoning about what's already known and how we can embed that into the model and lastly perhaps the emerging one that I think will be critical in the next decade is the role of simulations right that's never going to be enough real data but theoretically there's infinite amount of data in simulations but the challenges we can never recreate the complexity of the real world all the variations in the real world so they'll be a domain gap between the real and the synthetic data the question is how do we design algorithms that can do seem to real go from the simulation domain to the real domain and know to expect that there'll be domain shifts how do we do that robustly now we don't want an autonomous car trained on simulations to go crash in the real world right so instead we are using the synthetic data to improve the safety of those cars so how do we get the best from simulations and account for domain shifts I think we'll see a lot of this in the coming decade so to go a bit deeper into what I mentioned about tensors so indeed if you're writing deep learning code there is input tensor write your data is expressed as a tensor because you have many dimensions in your data if it's an image it's typically three dimensions video is four dimensions and so on and as we collect rich multimodal and higher dimensional data tensors will become even more important so to begin with we can think of tensor as a multi-dimensional array where you collect the data right but the thing is how do we keep the structure of multiple dimensions as we propagate information through the neural network so the neural network has many layers and currently we only do matrix computations in these layers so can we have better inductive bias by keeping tensor structures in the layers so that's what we can do and think about the tensor layers and indeed if we want to think of tensors they're extensions of matrices to more dimensions as with all abstract math concepts dogs make it accessible friendly and fluffy so your overview of water tents arrays right so point a zero-dimensional our vector is a line with a length and direction matrix has rows and columns tensors now extend to more dimensions a third order tensor is a three-dimensional cube this you can visualize so people ask me what happens when you go to full dimensions I can't visualize I tell them to go read a book called flatland this is a novel that was written in 1850s and there it's an imaginary two-dimensional world all the living beings are in two dimensions and there's a three-dimensional monster that invades the world it's a fascinating book in many different dimensions literally so I encourage you to go over read that and the idea is you don't have to visualize an object in order to process that right so most of the visualizations we try to do in two dimension and two demaree dimensions to get intuitions but really these mathematical concept is applicable to any dimension and the main idea is when you're projecting your higher dimensional object to a lower dimensional one you lose information right so this is easily seen especially you see the cube there and it projects down on the wall you cannot get all the information in the cube just in that - you know 2d and this is the intuition of why maybe matrix layers are not completely the right thing to do are suboptimal for processing data tensors and if you think about what a neural network is doing it's taking an input like an image and coming up with an output like a category or a label right so that mapping is also a tensor itself and so that kind of mapping could be better expressed when you also do higher-order computations as your primitives and so that brings us to the point of what's a new computational primitive I told you that the foundation of GPUs is the blas primitive right blas stands for basically near algebra subroutine and at the heart of it are matrix matrix computations you do matrix matrix products through that you can express essentially the whole array of linear algebra techniques and you can implement neural network layers and now the question is if I want to extend it to more dimensions I can similarly extend the notion of matrix product to higher dimensions and what you see there pictorially is now this three-dimensional cube you're multiplying along multiple directions and we call it a contraction because you tend to contract and come up with a smaller object as the output and so this now serves as a new computational primitive and with this we can build layers of neural network and so that's the basic intuition that we can now extend this to many dimensions and come up with all kinds of ways to express tensor layers in our networks and as an example there is the standard neural network right there's a standard convolutional network the first two layers you see is convolution operation when you're doing convolution it keeps the spatial information because you are processing your image it kinda you know keeps through space and that's how you get the activation map on the other hand when you come to fully connected layers in the standard networks you flatten those activation tensors right so meaning you just make it a one dimensional object you ignore all the spatial information and then you go forward you do reloj and so on but the thing is what if we retain the three-dimensional information throughout the network that to me is the first step of extending this two tensor layers and that's what we did we can do the tensor contraction operation that I displayed in the previous slide so now you're learning separate weight matrices along the different dimensions rather than one monolithic weight matrix in the fully connected layer and in the last layer it's something what we call tensor regression again you can parameterize the weights to be low rank and compact so the benefit of doing this is by incorporating inductive bias that the data is three-dimensional and there is spatial information we can come up with far more compact networks so we can save the number of parameters in these fully connected layers by a huge amount and still retain accuracy as much as 65% in some of the networks and so that's the benefit that we don't have to always have these big bad massive networks right we can now by also make them compressed and robust and generalizable by using better inductive bias and so you can think of this as a more global form of compression by using tensor computations you know when you do pruning or quantization they're much more local right there only reasoning about one neuron at a time whereas this is doing it all jointly and so this is one instance of how you can use tensors so there are several extensions from there we took I mean you can do tensor layers in different parts of the neural network the extension that I'm excited about is for sequential models and if you want to do time series forecasting or other sequence predictions it's hard when you have to predict way into the future the standard recurrent networks have deficiencies in that they only have short term memory and that's why the latest language models no longer are recurrent or els TMS right and on the other hand they're massive they're essentially taking the whole sequence together and reasoning about that so if there's something intermediate in the sense there is still a sequential transition of the hidden states but on the other hand with tensors we can express higher-order correlations better and that's what we did in this world called tensor train RNAs sorry where the idea is instead of one hidden state the bottleneck propagating forward we look at a window of freedom states and finally low rank tensors that Express the higher order correlations between them there are lots of details in the math but the intuition is your better propagating the higher order correlations in time and so the hope is the information in the higher order correlation that matters for forecasting gets used better and we get better forecasting results and that's what we saw in this work where the blue is the LST M the baseline Alice T M and the red is the tensor train LST M and you can see that the forecasting accuracy or in this case the forecasting error it was much worse for LST m especially as you went way forward I don't want to just predict what happens in the next time step I want to predict n time steps and so on this is challenging and this is where tensors can help and in fact the temps to train LST M was a smaller network it had smaller number of parameters than the baseline right so it's possible to get networks that are small and efficient and at the same time having better accuracy and so it's possible to get best of both the worlds and as an extension recently at an media what we did was to ask can we do the same with video prediction video prediction is perhaps one of the hardest tasks now it's high dimensional your input is high dimensional your output is high dimensional if you want to do this you know 10 20 30 frames down the line this is extremely hard right and even the biggest models can't do a great job at it today and so what we did was we essentially did convolutional version of the tensor train the idea is convolution is essential for images and also for propagating forward in time so we built convolution operations along with the tensor operations and what we see is compared to the state of the art as well as the baseline it's able to propagate long-range correlations much more effectively here you see like even up to like 38 frame which is way ahead right it has to generate all the way up to 38 frame forward the last one is the the best one we have and you can see that that's sharp and able to keep track of the correlations all the way through but all the other models just fade away they just don't know what to do that far into the future and same with another data set where the actions need to be recognized and you see like it's still sharp way ahead up to like 28 frames in this case whereas the baseline model just get you know loses track of it and so the idea is you are able to retain long-range forecasting capabilities by using higher order correlations but these are also smaller models these are smaller than the baseline there's definitely smaller than the state of the art models so the idea is let's just keep building big models and we'll get to singularity I know people have told me that is not possible so we need to incorporate the right structure and in that you bias and with this that means we can also go bigger right but with the right inductive bias we can solve more challenging problems and that's the way I view the ecosystem today so testily is an open source software where we've incorporated a lot of these functionalities the idea is it has multiple backends including PI torch and tensorflow and you know within Python so you have these tensor layers already available you can add your layers to it and train into it so as a simple example here if this is in pi torch you can see that you know there is a tensor layer you can express you can ask what kind of tensor representation you want and just define it as a layer and after that it's all like the same as pi torch so this kind of integration hopefully makes it accessible and available to a larger community so like I mentioned so far I didn't talk about performance upgrades with tensors the idea is because you are blocking in two more dimensions you have more abilities to paralyse and hence higher efficiency so ku tensor is a library that I think got launched this week at supercomputing the idea is there is no new set of primitives that are accelerated in CUDA and it's a matter of time where it'll be connected to frameworks like pi torch in terms of flow and we can also get the power of GPUs to further accelerate these tensor operations and yeah so you can see here that three facets again came together right tensors are efficient to represent rich data structures we obtained today whether it's multi-dimensional or multimodal and then tensor algorithms can keep this dimensionality information through the layers of processing in a deep neural network and then on the compute infrastructure as well we can accelerate tensors so the Trinity again comes together very well with tensors so since I have just a few minutes left I wanted to give an other examples of where there is free lunch you know you can get compression but at the same time don't lose accuracy and in this case this was for multi node training if you want to use multiple GPUs and train our model there are two frameworks right either you can do parameter server or you can do ring and we implemented both of them but the idea is if when communication cost becomes the bottleneck you want to be able to compress them and so we asked a simple question what if the machines just sent that sign of the gradient I'm in GP your machines right whatever is a single node of computation so when they only send the sign what happens they don't send the full gradient so now this is one bit quantization and it emerges that the performance is almost the same in terms of accuracy so the accuracy is nearly the same you get no drop in accuracy but of course you get savings in communication and so better throughput so the numbers I won't go into but the main idea is you can get a free lunch with gradient compression and in fact depending on the model some of the language models you can even get more gains by compressing the gradients there's also some nice theory showing that why this is possible I will not go into that but it actually shows that you don't have to do anything just compress the gradients and it will work in most reasonable circumstances and we are also now trying this out on the ml per benchmark which if you keep track of it there was an announcement just last week on how Nvidia beat on the inference benchmarks this time it was the inference the previous ones were training and it's emerging as the de-facto benchmarks for performance in machine learning and so there are lots of like things we can add on top of the current best performing systems in terms of better algorithm design better inductive bias and better also theoretical analysis saying what are the right things to do now the framework that I'd like to introduce is the rapid launch by Nvidia which is open source and the idea is instead of just optimizing a single machine learning algorithm can we optimize end-to-end pipelines especially ETL is a bottleneck in so many real applications right so if we can remove the passage between CPU and GPU and keep it on the GPU accelerated there are tremendous gains that can be had if you go to the website you can look at XG boost and so many frameworks that are accelerated now on GPUs so I encourage you to check that out other frameworks like Nvidia Isacc is where there are robot simulators high-quality physically valid photorealistic renderers and simulators and so with this we can do so much in simulation before going on to the real world and this is just an example of that yeah and same with like autonomous driving we are experimenting our algorithms now on these photorealistic and physically valid Drive simulators and there's so much of testing and safety critical analysis we can do there before taking it to the real world yeah so that's the main principle that you know what I wanted to emphasize was data algorithms and computer infrastructure coming together and as we go to the next generation I believe we'll have more use of tensors and higher-order structures to express our data and then have algorithms that would not only use neural networks but also frameworks like seem to real how to go from simulation to the real world or between domains how to incorporate structure priors and other domain knowledge and also in the computing France not just write faster computing but also more computing on the edge and more energy efficiency and so on so there's lots to come in the next decade thank you [Applause] [Music] [Applause] [Music]