data.bythebay.io: Sanjeev Satheesh, Fast deep recurrent net training
Recording: data.bythebay.io: Sanjeev Satheesh, Fast deep recurrent net training
okay thank you guys before I start off I have a question for all of you how many of us wait for days and weeks for our models to converge right okay well we definitely spend a lot of time babysitting our models and my dog here is fast deep recurrent net training so hopefully by the end of the talk we have some idea about how we will address some of these questions okay so the story of deep speech and probably of all of deep learning is this you start out with a toy data set or a small data set and then you find a model you do model exploration to find a model that fits your data very well that is it has a low bias and then you scale you scale both your model and your data set to really crush that problem go from a toy data set to really crushing that problem but what we often find is when we are training these deep networks at scale it is often difficult to distinguish between bias problems and optimization issues so you'll often see cases where you your models hit a glass ceiling and then you find yourself asking this question oh is this a bias problem or is this an optimization issue what exactly do I mean by that when I speak about model bias that a symptom of that is often underfitting so you take your model and you try to train on your data and like this blue curve here where we are plotting a cost function or against the number of iterations of training you see that this model is just not able to fit the data as well as some of the other models here so this is symptomatic of underfitting and you can say that this particular blue model has a very high bias of course the deep learning solution to this problem is just make the model bigger make the model deeper right on the other hand you have optimization issues optimization issues often show up as Nance you try to train your model and you get just Nance in other cases you also see curves like this where there's just no learning there it's just a flat training curve of course deep learning practitioners are also quick to catch this and say oh yeah this is the case is that your learning rate is too high and you should just drop down your learning rate and maybe your model will look more like that the orange curve but you can already see here the two problems both express their symptoms in the training curve and it is often hard to distinguish between the tool so the point I'm trying to make here is when you are when you're training these deep Nets at scale always keep in mind both the questions we often think about bias and think about making the model bigger and bigger but keep both the questions in mind is this model powerful enough to solve my problem can I make it bigger and bigger and also is there anything preventing my model from learning all the correlations in my data set that is can i make my mock can i make it easier for my model to learn from the data so this talk is mainly about how we are addressing both these questions as we are hacking on deep speech trying to solve the speech recognition problem all the content in the talk is from is based on the amazing work of people at baidu Silicon Valley AI lab my name is Sanjeev Satish if you're looking for me in this picture I'll just say I chose the wrong day to be working from home okay so we will basically start off with a quick introduction of what deep speech is and then go over some examples of how we are trying to solve some of the bias issues and some of the optimization problems and what are the big takeaways high level takeaways for for us to take away from here okay so deep speech is basically a deep network which reads in spectrograms of audio files and the task that is given to the model is to transcribe what's in the audio clip okay in a little bit more detail deep speech is basically a deep recurrent neural network whose initial the input layer is a convolutional layer so you can convulse filters across time to consume your spectrograms followed by multiple layers of fully connected layers and then a bi-directional recurrent layer so the bidirectional recurrent layer allows the model to to communicate information across through time both from the future and from the past immediately on top of that is a softmax layer the softmax layer allows the model to predict one character at a time so remember the task is to go from spectrograms two characters the spectrograms are read by the convolutional layers and at the top is the soft match that predicts the characters the whole model is trained with a sequence level last called as the ctc loss we'll just call it the ctc last for now and not go into too much details okay so that's deep speech version 1.0 what can we do to make it better right of course the one of the great things we can do is to make it deeper make it bigger and see what happens the point of making it bigger is to reduce the bias in the model basically to fit the fit the data even better and better okay so this is what we did instead of having fully connected layers we made all of them recurrent layers recurrent layers that are closer to the convolutional that is the spectrograms allow the model to communicate information at the input level more more data about the audio signals and as you get closer and closer to the softmax its you can expect that this model is commuting communicating more information about the orthography of the language and so on of course you can see with one recurrent layer this what happens and yes and as you add more and more recurrent layers you fit the data even better and better so remember the cut all the graphs here are plotting the cost lower cost is better so if you're fitting the training if you're fitting the training data better your costs will go down this is plotting costs on the training set right see okay so you add more and more recurrent layers you fit the data better and better until you hit a glass ceiling you hit the glass ceiling and then you wonder okay looks like I have so all these models have the same number of parameters so once you hit the glass ceiling you're like hmm maybe I should blow up my model increase the size of my model and add more parameters and see what happens right that's often the first first thing we want to try but the point of the talk is that you should also think about this other question that is is my model running into optimization issues right so let's see what we did in deep speech to address to find out if there are optimization issues in this model okay so the one one thing that we tried is called sorta grad which will go into a little bit detail now the idea behind sorta grad is to improve the stability of training which means fewer man's fewer in costs and so on okay the idea the basic idea is this in the first pass over your data in the first epoch you feed in the easy examples first and as you go through your data set you make the examples harder and harder okay just like how you teach a baby you teach a baby the easy examples first mom dad and as it grows you you feed it more harder examples to learn from the sorted rad idea is very much the same except you do this only in the first pass the insight is this so the ctc loss is directly proportional to the length of your audio clip so the longer your audio clip the ctc loss is just going to be bigger and bigger so you can use the ctc the length as a proxy for how hard it is for your model to understand that clip so the idea is you take the smaller clips first and you go through them in increasing order of length and you serve it in that sequence only in the first epoch after the first epoch you drop back to the standard SGD stochastic gradient descent where you sample uniformly from your entire data set there is no sequence of you know sampling okay so what happens so we took a seven bi-directional model and we only changed the fact that we are now training with sorta grab remember this means only that you have changed the way the first epoch the data items in the first epoch is served this this is this we thought would be a stability inducing thing but it also ends up being an optimization trick which helps your model to train faster and to a better minimum this is important not only does this stability stability in reducing trick make the model converge faster but it also helps converts to a better minima and you will see this trend over and over again ok let's want to the second one the second optimization trick you can try is batch normalization this is a fairly popular method this was first introduced in a paper in February 29 and was useful for getting to state of art on image net on the image net challenge the problem that batch normalization is trying to solve is reducing what is called as internal covariant shift now what does that mean imagine a deep network right and just focus on one neuron okay look at one neuron ask then as the entire model keeps training the distribution of inputs into the neuron keeps changing so you can see here as this model it keeps training the distribution of inputs to this non-linearity keeps changing what batch normalization is trying to do is regardless of what distribution it is it will normalize that distribution to have a zero mean and unit variance okay this is all what barge normalization is doing this makes it so that the input distribution to any particular in wrong at any point in training is consistent so that if that neuron is only looking at a stationary distribution okay what happens here so on top of just as a seven bi-directional model we tried sort a grad and then we tried bash norm on top of it it fits that it helps fitting the data even better right so the cost is going down as we add a banner on on top of sorta graph this again we thought is an optimization trick right this is what we expected this to help the model train faster that is we expected the convergence to be faster but we really did not expect this gap in reducing the bias this is another optimization trick that helps reduce the bias in the model oh the blue lines here are basically saying that we only apply batch norm in the vertical connections you can apply batch norm between any of the connections but we only apply it in the vertical connections yes oh yes there's also a bad norm here good catch I missed that line so in every vertical connection is batch normalized yes okay this is the question we've all been waiting to ask right all the models earlier on have been just using plain vanilla RN ends of vanilla ireland's can be is just having anything that is not lstm or variants of LS TMS we can call them as vanilla recurrent networks the idea behind using these lstm cells or grew cells is to let the gradient flow in some sense we'll talk about that a little bit more now okay a basic recurrent end product looks like this the interface is that it takes at time T and input and outputs a context vector H and which and it also passes it to the next time step so it's like you process an input and you have some state and you want the state to be reused when computing the next time step so this is what is the context vector so this is what goes up and to the next time step you can think of context vector as the state what a vanilla are an end does is that it takes the are the state and completely overrides it this is a problem for two reasons so if the RN and wants to pass state for more than one time step right let's say it wants remember some part of the state for more than one time step it's it's really hard for the model because the model already has this bias that I'm going to overwrite all the context vector and it's very hard for it to remember remember the context for more than one time step lstm and grew cells have fixed that by having a linear connection between the previous time step and the next time step so HT is a leap has also a linear dependence on HT minus 1 and the lstm cells by using this multiplication control how much of my previous context to remember and how much to overwrite so we explicitly model that this model can can can control how much to remember and how much to overwrite and also because this is a linear dependence and not passing through any squashing nonlinearities we also get away from the problem of vanishing gradients you can basically the idea is that you can think of this as an optimization trick right okay we use grew cells here and not lstm cells because the lstm cells typically are more flop hungry they use more parameters and they need more flops to compute to get the same effect so if you can get the same effect with a simple cell you should try it there has been a published research trying a whole bunch of variance of LST ms and they found that the grew cells are about as good as the best msdn configurations okay so okay grew cells you can think of as another optimization trick let's see what happens with that so this is what we had we had a baseline model with seven bi-directional recurrences we added sorta grad better better optimization to a better minima on top of which we added batch rom even better and we sweep we so the RNN cells can be swapped for grew cells we just did that and it fits the data even better right these are all optimization tricks remember these are none of these are really bias reduction tricks which but we get that effect there are a few more hacks like these in the paper but I don't want to dwell too much on it and I let's get to the punch line the punch line is make your model easier to optimize before scaling punch line really is make your model easier to optimize so before so you start out with a small data set you're playing around trying to find out which model fits your data well and then as you scale we will get into this problem that will hit a glass ceiling and we won't know if we are fitting our mo I mean if we are having a bias problem or an optimization problem so the point is solve your optimization problems as soon as possible the so if you solve your optimization problem sooner the idea is they're easier to identify they will show up immediately in the first few thousand iterations in your training and they are very easy to spot because literally easier to optimize means that you have a much sharper slope in the earlier iterations and from all the examples we have seen here they will they will make they will make better use of the same number of parameters to fit the data even better ok so the remember make your model easier to optimize before you scale you can also think of this in a different way given a choice between two more right which one do you pick regardless of which one you pick you're going to hit this glass ceiling and you are going to think oh I don't know if this an optimization problem or a bias issue so if you solve that problem much ahead then it means that at least it's one less thing for you to think about later on the this rule of thumb is also any taxes for reading papers let's say there are multiple papers trying to solve the same problem it's one thing it's easier to ask is is this is this model making it easier for me to optimize on this problem or that model and also figuring out among all you know there's usually many items in the paper and finding out oh this this particular trick in this particular paper is actually trying to solve an optimization problem and not really a bias problem or an overfitting problem right then you can take only that part of the paper out and apply it on your model and it'll you can probably help so those that's the main point focus also on making it easier for your model to optimize not only to which is often a very good way to reduce bias don't look at it only as a way of reducing bias yep so that's it I would you thank you so much I'll what questions do you have oh the thesis is it's hard to distinguish between bias and optimization you will invariably you will hit a glass ceiling and then you won't know if it's a bias Bob well what I mean is that you're saying that your model has some intrinsic bias yes and then you have the option to add some optimizations yes how do you distinguish between saying okay adding GRU cells is an optimization rather than being an intrinsic part of the model I just don't understand how you oh I see so the GRU cells for example are particularly trying to solve the optimization problem of vanishing gradients that's why I would I put it more in the bucket of a general optimization trick than so you're saying it's not an intrinsic part of the not always a addition yes any other questions I was just wondering as far as optimization goes there are definitely architectural issues here such as you know traditional RNN versus GRU versus lstm but have you guys looked into using a non SGD optimizer as in you know at a delta or that basically optimizers that don't depend on the learning rate setting there are plenty of those right now at a delta is kind of an old one by now for the stochastic based solutions but there's also vs GD by n lagoon and no more pesky learning right paper or actually add a secant from last year by Benji oh right so the question is that in terms of optimization there are a whole bunch of other optimizers as well that you could try and we have only mainly stuck to the iterative stochastic gradient descent and variance of that in this particular talk we are using stochastic gradient descent with momentum and nostrils acceleration we did try a few gradients we tried add a delta and autograph they didn't perform as well so we are basically sticking to a CD with momentum I see that's definitely scope for us to explore further we have only tried a few others and put not too much I guess yeah I think actually optimization matters a lot there there was a dog by jeremy howard from analytic and last year at startup ml and he basically demonstrated how they got some training of dead models that they're using in their case cnn's from weeks two hours and by using different optimizers and the convergence was to a much better optimum so it seems like that the whole problem not just of setting the initial learning rate but also of the weight decay is is really a black art and and better optimizers definitely take the headache out of that and it seems like they're doing pretty well so i would think i would trust benji especially on this recent demonstration and even lagoons demonstration shows that you don't have to calculate the whole Hessian to have a pretty much optimal setting for the for the gradient even in a stochastic setting and unlike at a grad which has a continuous weight decay so eventually your model stops learning with vs GD with lagoons modeled a learning grade goes back up if you have a shock to the model so if the cost goes up the learning rate goes up as well which is pretty awesome in my opinion i think i think it's something to explore so just an idea oh yeah absolutely that's definitely a big pool of stuff that we have we haven't yet really explored yet i'm definitely looking forward to one more quick question or yes is that it he do you have time for one absolutely yes right I was wondering if you could describe a little bit you're what software you use hardware ah ok so as we have a follow-up talk from from my colleague shubo later today I'm sure he can answer in detail a lot of these questions as a quick summary we use our own code base for optimizing all for doing all of these optimizations it's returning from scratch using C++ cuda and MPI ok great let's have a hand for Sanjeev Satish thank you very much Andrew Sun do the next talk will be at 210