data.bythebay.io: Shubho Sengupta,

data.bythebay.io: Shubho Sengupta,

Recording: data.bythebay.io: Shubho Sengupta,

so with that Preamble lets me start with a story and the story of how when our artificial intelligence lab was starting off about two years uh from now we were you know a small group of a dozen people and we had this Grand Vision of how we can affect the lives of hundreds of millions of people and we wanted to do it with a technology that almost none of us had any experience working on and and we wanted to do it in a fairly short period of time about a year so you know this kind of sounds like a recipe for a disaster right uh but luckily for us um we were I would say quite a bit successful um so the technology and if you guys have heard my colleague s G stock before that we worked on and developed over the course of last year year and a half is very very good uh speech recognition and why did we start with speech recognition why did we start down this path when we had no experience whatsoever and one of the reasons is because of really big numbers um the big number that I want to talk about is 600 million so there are 600 million mobile users in China which completely dwarfs every anything that we have in the US and we think that we use mobile phones quite a bit in our daily lives uh let me say wait till you go to China I went to Beijing for the first time a couple of weeks back and I was completely blown away how pervasive and ubiquitous mobile phone usage is in China um one example is uh what I call offline to online online to offline Commerce It's Perfectly Natural in Beijing to walk up to aent vending machine hold your phone against a QR code on the vending machine buy a soda and pay through your phone nobody brings out a credit card in Beijing I've never seen it people pay to their phones um this is something that you will never see even in a tech Frontier like the bay area but this is completely natural in Beijing so we as a company Buu we are one of the biggest internet companies in China and our mobile properties and websites are used by hundreds of Millions of people daily so we are very very concerned about or we want to drive user adoption and one thing we should keep in mind about the users in China they're a little bit different from users in the US for a lot of the 600 million people this is the first computer that they've ever owned they've never had a PC they've completely jumped the PC to go to the phone so to drive user adoption we have to make the usage of these phones as natural as possible so we have to think about how do we humans think what do we humans think are is natural and one thing that we want to do is you want to make the interaction with the phone as if you're interacting with a human and one way to do that is to use what we are good at what is our user interface and our user interface the many user interface that we have is speech we talk to each other so you want to be able to talk to the phone like we talk to a um human and this very naturally to somebody who has never used a computer before um so what are the challenges in this natural user interface and there if you want to actually talk to a phone like you talk to user there are many aspects of it so it starts off with speech recognition which is you take in voice you output text output text once you output text you to find out what the mood of the user is and then you have to figure out what does the user really want and once you have figured those out you have to figure out what you're going to tell back to the user what's going to be a reply and once you have your reply formulated um you want to have very very good text to speech a text to speech which has emotional cues which doesn't sound robotic which sounds natural now each of these are incredibly hard Problems by themselves and the one problem that we want to tackle was the problem of speech recognition because it is kind of the first thing in the pipeline without that you can't really do any of the other now if you look at speech recognition speech recognition has been a very very well studied field it's been studied for many many decades there are pipelines like CDI U that work uh reasonably well but these pipelines also have some challenges so one challenge with a CI pipeline it's a very very complicated system it has many stages each of these stages are modeled by somebody who knows the domain really really well for example you would have a stage for say modeling phones another stage for modeling accents so you need a lot of domain expertise to create a pipeline like this and while this is you could say you know it's a source of strength that you have these domain experts you've studied the field they're doing this they're doing a good job and I would say yeah it is they are doing a good job but the downside of it is makes these pipelines kind of inflexible and they don't really learn in the presence of big data so when you have a CI Pipeline and you have a lot of data say from users who have different accents or from different noise backgrounds if you train this pipeline because so much so much of the engineering has been done by humans they don't really learn from the data that easily it is also inflexible in a way that for example the pipeline was engineered uh for a particular accent and now you have data from a different accent to learn this new accent you have to hand engineer this feature into the pipeline uh same goes for a different language say you have a called the pipeline for English um can you use the pipeline for Mandarin which is a very very different language it's a tonal language uh the answer is sadly no so you need domain experts for Mandarin to like actually do the whole thing almost from scratch so we looked at this uh problem area and we decided you know we don't know much about speech recognition at all we are not domain experts in speech but what we do have is a lot of data and we can get a lot of data too we we are good at that um and we're also good at this field of deep neural networks or deep learning so is there a way to combine these two technologies to create something that is very very flexible and very very simple in instead of this very complicated pipeline that we have so with that we came up with this uh what I call our first radical idea which was can be use deep neural networks to go end to end and what I mean by end to end is you take audio from from one end into the deep neural network and you emit text now at this point I should say that the radical idea is not using deep neural Nets the radical idea is really is end to endend so people have tried and successfully done taken a CI pipeline switched out some of some of their stages and put a deep neural net into that stage so you have kind of a hybrid pipeline so what is the problem with this approach and it goes back to like can we scale with data and the answer is not really so here I have a stylistic graph of showing scalability of data of these three approaches so at the bottom in the dotted line you have a traditional ASR which is kind of a CI pipeline um it as you see it doesn't scale with data and then you have the Deep learning version one where people were doing this hybrid approach um and it scales a little bit and it flattens and then we have our end to end approach in the solid line and you see that it scales really really well and continues to scale as we train on more data so what does this end to end pipeline line actually look like so let me kind of give you a secret sauce which is what would call a deep speech 2 pipeline which is uh we did a paper back in uh December uh which is at that point our state-of-the-art pipeline so what it does is you have um audio coming in from one side uh and we're doing it for um say English or any other language I'll talk about it later on um and the first thing we do is take the audio we do a spectrogram spectrogram is essentially a bunch of ffts uh this then goes into a convolution stage we have three convolution stages and if people are familiar with image net these are the same kind of convolutions that imet uses um except that we are doing in time and um frequency and then this data then gets uh the output of the convolution network then goes into the uh recurrent network uh the reason we use recurrent networks is because speech has uh temporal dependencies so what I say now uh is dependent on what I said five seconds from now and what I said 5 seconds behind so you have to somehow capture this temporal dependency in speech and you do that by using this particular kind of uh Network called a recer and neural net and recer and neural net kind of comes in couple of different flavors you have a basic record of neural Nets you have a gated record and unit and you have a long shortterm memory model and you play around with all three of those and you can also go backward and forward in time and there challenges in both uh you can have just a forward only record and neural net which only goes forward in time or you can have bidirectional record neural net which goes forward and backward in time it's kind of like having a time machine and then the output of the uh recer and neural net goes into a fully connected layer um what the fully connected layer does is outputs a probability distribution on the character set every 10 milliseconds every time slice it outputs a probability distribution over your character set of 29 characters this then gets fed into this cost function called the CTC cost function which was in invented by Alex Graves and what CDC does is it takes its probability distribution finds out the most likely text label from this probability distribution Compares it with the ground truth in which case this is cat uh finds out the difference uh between these two text labels that becomes the gradient which is then fed back to the network and we update the weights and we do another Epoch we we take this example back again feed it feed it so on and so forth like you do um neural network training so you know it's all well and good so what are the challenges in in training like this um so the first challenge that we talked about is we have to get a lot of data for deep networks to work data is the key Point data is the essence for deep networks to work and I'll get to how much data we need and once we have the data um we have to explore a lot of models because there's not really a theoretical um framework where we can use to say oh you need to have 10 layers and five of them have to be recent I mean all these number of layers how they're connected where they should be the entire architecture of the network is basically experimental so you have to run a lot of experiments and if you have to have a have to uh run a lot of experiments you want to do them quickly right because you want to get to the solution quickly so let's look at these two challenges one by one so how large is large uh compared to what um what is out there so in this graph I show you in blue what are the publicly available data sets that you can either buy actually most of the data sets you have to buy uh but they're available I mean you can if you have money you can buy them and the data set that we put together for deep speech which is about 12,000 hours of audio compared to 2,000 hours for Fisher so we already are six times more uh than what is publicly available to anybody and not only that this is a this is a number from last December so for our data sets we are very aggressive about collecting more data so this number is has increased quite a bit um and not only that uh we can increase this number even more by synthesizing input by that uh I mean you can take noise so we have a lot of examples of noise and you you mix noise with your input data to even go from uh 12,000 hours to 100,000 RS effective of um the training data so that's um that's the difference in scale of what is available outside and what we train on so the next stage um as I said once we have the data set um we have to train a lot of models quickly because you have what I call a model landscape and it's a vast landscape and you have to explore the model landscape as quickly as you possibly can um to do that um we have to get on what is what I call the idea cycle so the idea cycle looks like this you have an idea you write code or you change parameters in your framework you look at your training curve and you decide whether this is a good idea or a bad idea right uh so why is this I me this sound super simple and why is this hard uh to give you an idea of how uh complicated this process is one run through that cycle back in December for 12,000 hours of audio is 20 xof flops 20 xof flops that's 20 followed by 18 zeros I mean I don't think I've ever worked on a problem which needs 20 xof flops to train so and you want to do the training in about somewhere between one to two weeks like a 10 days is what we're shooting for so if you want to train 20 xof flops in 10 weeks um that is a huge challenge so how do we uh solve this challenge so at this time we had the second radical idea is to rethink How We Do deep learning training so the second radical idea was post deep learning training as a high performance compute problem a high performance Computing is a little bit different how from how traditional machine learning is done a traditional machine learning you do on Frameworks like sparkk or kadoo something like that none of these Frameworks can deal with computational workloads like this so we have to completely rethink our mental model of what training has to be and thinking in this way we have developed quite possibly the fastest deep learning training system in the world I'll go into what are the different components of this deep learning training system is it's completely built in-house um so the first thing that we have to think about is like why did we even decide to do it um if you look at even deep learning Frameworks you have um tensorflow now there's Cafe there's Storch there's siano uh when we started a year and a half back there was no tensor flow but definitely there was the other ones um unfortunately none of the others can even come close to handling this much of compute so we had to build our own so what are the two basic components of this um the training pipeline so at the very bottom we have a very very high performance linear algebra library and this linear algebra library is distributed which means it can work on different nodes many many uh Graphics processing units um and on top of this uh distributed linear algebra Library which we call magel after Majel Rodenburg uh we have a deep learning framework that is efficient for rec and neural Nets because that's what we train um and we train exclusively on gpus or Graphics processing units for computational efficiency we can train on CPUs because they just don't have uh the flops that we need to train on so typically our training runs um train we our training runs use about 32 to 64 gpus uh we can go all the way up to 256 gpus but I'll go into some of the problems that we Face going up to 256 gpus some of the issues with scalability and the other interesting thing we do compared to a lot of the distributed deep learning Frameworks is we use synchronous communication this is very different from an asynchronous communication that you use in parameter servers that Google had pioneered over the you over the years and the reason we use synchronous communication is because we want repeatability which means you train the same model and the same amount of same data you don't change anything you should gain the same exact result and if you don't then you have a bug and we we don't know any other way of debugging these networks uh if you have a bug in your system neural networks are really really good to work around the bug I mean they're really good uh so the only way you can do it is by having repeatability and synchronous communication allows you to do that but why don't people use it people don't use it because synchronous communication means that you have to have a really really efficient communication layer so we built a communication layer inhouse based on MPI and this communication layer uses MPI very very low-level MPI Primitives um to build MPI like system um that can scale all the way up to 256 gpus and for our use case it's 15 to 20 times faster than anything that you'll get like open MPI or any other open MPI uh that you can download and use uh so we've paid very careful attention to the algorithms and our cluster designed to do it so at this point I should tell you what are the problems with scaling up to 256 gpus so if you do what is called weak scaling which means as you increase the number of gpus you increase the mini batch size you're going to reach a point where your convergence will suffer so typically about 1024 G 1024 uh mini batch is where we see uh our convergence suffering so if you have to keep a mini Badge of 1024 and yet be efficient on the number of gpus you cannot go beyond usually 32 gpus because the way the data the gpus love a lot of data so if you have so if you have 10 24 mini batch of 32 gpus you have mini batch of 32 per GPU and below that you kind of fall into the regime where it's just not efficient for the GPU so if you want to scale up the number of gpus now you're scaling up the mini batch your convergence suffers so even though our communication layer can go all the way up to 256 gpus we um usually train around 32 or 64 at the max okay so the other thing that we do is our entire training pipeline runs on the GPU there is no CPU involvement anywhere um this is not that hard because you have um Matrix Matrix multiply libraries now that are very efficient on gpus you have convolution libraries now that are very efficient on gpus what you don't have or you didn't have until January was a very good implementation of the CTC cost function on the GPU uh when we started CTC is a very very complicated cost function and when we started the uh people thought that it is impossible to do a very efficient uh implementation of CDC cost function on the GPU because the algorithm looked really serial uh it turns out people were wrong uh you could have a very Implement very good parallel algorithm for doing the same cost and we did it um and it's 400 times faster faster than the existing CDC implementations on the CPU uh and this project is actually open source other companies are using it for doing their their speech recognition it uh we it has a torch interface so if you use torch you can use it um it's called warp CDC if you're interested in speech and use CDC you would I would highly encourage you to go and check it out um and finally we also wrote A U custom memory allocator and we did it because our memory allocations we kind of know the sizes of our memory allocations because we know the sizes of our matrices uh it turns out that system memory allocators both for the GPU and CPU are kind of tuned for a wide variety of sizes and those are not the sizes we use for example memory allocations are not usually very small because networks are really big so we wrote a custom memory allocator for the CPU and the GPU and the beauty of it is if the GPU runs out of memory the allocator will transparently move data to the CPU the user doesn't even have to bother about GPU running out of me memory so to speak and not only that it's about three times FAS faster than any memory allocator for those particular sizes so it was a big uh boost in our training so you know now we have have the system architected oh and I should also talk about our cluster so our cluster um has um each node in our cluster has eight GPS each so it is a very very high compute dense cluster each of these nodes in this cluster are connected through an infinite band Network for the bandwidth and the latency requirements and we connect with um a fat tree topology what that means is the Hop from going one GPU in the cluster to any other GPU in the cluster um you really only have to go through two levels of switches at most so this is this um very close connection between our Software System and our Hardware system so it's the hardware software Cod design is what makes our system so fast so without this kind of a cluster and this kind of a network our communication layer for example will not be that fast so this is how we kind of did the code design and this is why our system is so fast so now that we have our system up and running we are about six months into a training um and we are training on English and we are getting some decent results in English uh at this point of time we decided to um think about okay so we started out with a premise of deep learning being a very flexible approach how flexible is it can we train on a different language without doing anything can one network to network architecture do two different languages and let's make our lives even a little bit hard will train on English and on manering now English and manarin as you know even if you don't speak manarin are very different languages as I started off with one is tonal another is not one has 26 characters another as I think 80,000 or something like that um we use the 6,000 most frequent characters um and it turns out the answer is yes I mean this still boggles my mind that you have one network architecture that does two languages really really well and a funny story from when we started uh doing Mandarin training was nobody in a team spoke Mandarin we had large and very good Mandarin data set but nobody could speak it but we I mean looking at the training curves we knew that we were getting really really good results so this is kind of I think the epitome of flexibility that you have a group of people with no experience of domain no experience of languages just by the power of data and the power of deep learning we are training these networks for two two different languages um so let's look at how do training differ mean how different are these n network architectures and it turns out they're not different at all it's the same network architecture that I showed you a couple of slides back of these three convolution networks and seven record neural networks works for both languages the only difference is the output layer which is the fully connected layer that's outputting the probability distribution instead of doing on 29 characters which 26 characters for some punctuation symbols uh now does it of over 6,000 characters that's all the difference there is and we have a language model at the end and the language model of course has to be language specific so we have a language model for Mandarin so now we have now we year down the line we are training both on Mandarin we're training on both Mandarin and English um how are we doing how are these networks doing on our various test sets that we have um so let's start with English so how do we Fair on English so you know just like machine Le any other uh machine learning algorithm deep learning is only as good um as our data so our data for English is very skewed towards the US accent um so we do really well I mean at at human level for little bit like not a lot of noise um Clean us accented data um and at this point we realize that English you know is a Kind of a Funny language there are a billion English speakers on the planet out of a billion English speakers only 300 English speakers speak English as a first language I mean English is not my first language um and even that in that 300 million speakers they have quite a bit of difference in accents like a Scottish accent an American accent and if you take take the 600 million speakers who speak at a second language their accents are heavily biased by what is their native language um if you go to India this accent essentially changes from state to state so it's very hard to come up with a very orthogonal data set for English so we are still trying so we don't do so well um on accented speech I Indian accent is one of our uh accents that we don't do very well on and we're trying to get more data we also don't do well if you haven't captured the noisy environment in which the speaker was speaking in so those are the places where we don't do so well so let's see how do we do on Mandarin so in Mandarin we have thanks to being a Chinese company and we have done speech before and using a traditional pipeline uh we have fantastic data set uh it captures real world queries coming into our system people from different accents from different noisy backgrounds um and because of this our ASR system actually outperforms humans in short short queries so by short queries I mean something that you speak uh to Google Now something that about 15 to 20 seconds long and the reason uh we are able to do this is because human beings actually rely on a lot of context to figure out what the other person is say saying like if you're a noisy party even you and I are talking even if I don't hear everything of what you're saying I kind of know what you're saying because I know the context um but on short queries if you don't give me any context I'll actually do very very poorly uh this is where uh machines outperformance um so now that we have these models uh we finally have to do something with them we have to let somebody use these models so then comes the question of how do you deploy these models and now this is also kind of challenging because these models are very very big I mean uh back in December our models were 100 million parameters um and it's gone I mean I I can give you the exact numbers now it's a couple of multiples of 100 million right now so 100 million is kind of our unit that is a uh lot of compute so we need a lot of compute for deployment so at this point of time you might say well I mean why don't you use gpus I mean you been using GPS in a training system don't they work for deployment yes and no uh because in deployment what happens is users really really care about latency which means if you are using an app you want the response back from the app as quickly as possible nobody has any patience now the problem with gpus is they're not meant for this kind of a work Lo so think about a single user query that's coming into our system which is a vector and a neural network is basically a bunch of matrices so you're doing a lot of Matrix Vector multiplies gpus are not set up to do this very well gpus love Matrix Matrix multiplies the bigger the Matrix are better um but they're not good at Matrix Vector multiplies so we had to come up with um the system of batch dispatch which is a very very efficient batching and dispatching mechanism uh what it does is it takes in a lot of of user queries that are coming into our system and then smartly picks a batch that is both kind of efficient for the GPU and which also um is respects the latency constraints that we have um and I should also say that the gpus that we use for deployment are a little bit different from the gpus that we use for training they are they don't need as large of a matrix to be computationally efficient as the as the gpus that we use for training so it's again that Hardware software codesign that we did uh to make it efficient for deployment so now that we have you know we have checked our boxes right we have developed this models there state-of-the-art we ready for deployment uh what do we do well we deploy we have a lot of products that use voice inside of China they're used by hundreds of millions of people um and we are in the process of deploying these models to our users in China and that is how we spent a year and a half affecting the lives of hundreds of millions of people with a small group of people out here here in the US and this is just not our story this is kind of the story of deep learning too um time and again we have seen Fields where deep learning has come in there has been a lot of data and by being having the ability to train on this large amount of data you have gone from zero to like the state-of-the-art uh performance beating humans in relatively short period of time you've seen that in image classification you've seen that in speech recognition you're starting to see some very very interesting results coming out of image captioning where you take an image you describe what the image is saying some it I mean it's almost indistinguishable from what a human would do or even story generation like computers writing stories through record and neurals um it is very very exciting so um I personally can't wait to see how deep learning is going to affect our lives um in the next decade or so I mean we are just starting this journey the data is going to increase our computational efficiency is going to increase we are going to have more compute so it's we live in very very exciting times um so thank you for giving uh me this opportunity thank you for your patience because this was very very last moment for me um and hopefully the talk was interesting for you because uh the subject changed a little bit as well um and I'm happy to um take any questions now or later my emails on there reach out to me i' be very very happy to talk thank you [Music] Chinese lot words like so how do you handle that you kep certain Contex uh so the question is how do we handle homophones which is you have um different characters which sound very much the same so we capture the context through the um the record neural net basically captures the context so yeah so that so that context is good enough for us uh typically I would say this I mean after you know a year and a half of training both languages uh I would say people who invented Chinese invented it for deep learning it is this language is really suited for deep learning English is not uh English is harder I wouldn't say English is not English is actually harder um to do deep learning on than what Chinese is uh uh one of the reasons is because if you go into the depths of CTC um CTC has this conditional Independence assumption which means characters become conditionally independent after a couple of characters so you know uh so this causes problems in nose so is nose K WS or no when you are at o and you're conditionally independent on anything beyond n you don't know um in Chinese because Chinese is so much more compact because the characters encode so much information this conditional Independence actually holds it it it doesn't hold so much in English so yeah so because the English characters don't encode that much information it is actually more challenging uh so I I mean a lot of our training tricks I I don't know if you were in sanjie stock before um there's a trickle srad um which I had come up with uh and and sort ofr the idea came from actually looking at Chinese because the Chinese utterances are much much smaller and our Chinese training is much much more stable than English training and then we figured out that wait a minute we could just do this thing where we in English and for Chinese we'll present the shorter queries first and and voila the training was much more stable so train training two networks has been I don't speak mannering at all but you know you look at these networks and you figure out you look at the training curves you kind of get a sense of where things are going my question I think there's one more question I I have a question I have some I'm a PhD in speech biometric so I wonder what uh do you think might be u a next breakthrough in uh the quality the efficiency of the system maybe you would want to use some additional features not just uh the Spectrum maybe something else H and by system so so the by System you mean the training system or the system in general no no the the way you train the system oh the way we train the system so the one that we want so the question is what what are the next steps for the system and we talk we're talking about it because we kind of brainstorming where the next place to go is um one thing that we want to do is there is obviously a lot more unlabeled data then there's labeled data um can we use the unlabeled data somehow um so that's what we are trying to figure out and also this problem with CDC that I talked about which is this conditional Independence assumption um can we do with a different cost function which doesn't have a different kind of network um that doesn't have to use CTC so those are the kind of main directions that we're looking at is is there any way to gain intelligence from unlabeled data especially for English because you have this accent challenge right um it's it's almost impossible to create an English data set that encompasses the diversity of Accents in English the britishers were great conquerors they conquered all over the world so there we have such a diversity uh there's ton of unlabeled data though unlabeled data probably is 100 times more than we have labeled data for is there any way we can learn features like accents from unlabeled data very very open research problem thank you oh you have a question just ack one I'm not sure I'm not super familiar I read about these uh of recursive neural networks you mentioned C ones before and you touching on like how learn a bit about manding about Chinese structure of it just from of obering netor yeah yeah the conditional Independence we knew because we knew the math I mean we have seen what what it does but we didn't so the question is um what was your question I kind of interrupted I guess have you it's have you seen anything you you kind of learned a bit about the structure of m even though you don't speak it from the only structure I learned is um maning utterances are short and they're short because each character encodes so much information and the funny thing is when we started people said you guys are idiots right man you are you guys are not even doing explicit phony model explicit tone modeling this thing is never going to work uh it turns out we never have to worry about tone at all you we worry about we don't have to worry about homophones at all because because of the data set the data set is really really good uh the network will learn those stuff um so other than noticing that the utterances are small leads to stable networks and the fact that because the characters are so information dense that the uh the conditional Independence works really well the Assumption works really well that's kind of what we have learned from Mandarin like just I'm looking at training I mean you know a lot of Deep learning is getting the six sense from doing a lot of train question yeah so what is the structure of rnns are using very very good question um bidirectional rnns will usually outperform forward directional rnns because they're usually going in a in a time machine right but think about deployment real exactly so we did it for a year and we started so obviously our deployment effort started about eight months after we started training and we started deploy and we were like crap now what do we do because if you have a bir directional Network and you want to deploy you have to wait for the user to finish an user might talk for a long time uh and that's a problem uh so we have now moved away from B directional networks so um we are just working with forward directional networks now but what we do use the B directional networks for is finding out what is the best we can do so that kind of sets the bar and we try to meet the bar with forward directional networks so we are like the models that we are deploying are forward directional networks yes you buch of speakers so let's sayel you might for a few and then yeah so we we did that kind of stuff to when we were starting to deploy B directional we were doing those kind of like trickery that you wait and then decode and you wait and you decode like buff it's kind of a headache I mean we want like simple stuff if you can get away with forward directional model like like doing this um there a lot of hand tuning involved to like having this buffering right just right so the latency works out I mean ideally we would not want to do this so it makes the the the deployment system like too complicated so if you can get away with forward Direction models that's what we want to do and that's what we are trying to do and right now I would say um I can talk about what the best forward Direction models are because we haven't published we usually published end of the year um but let's say that our forward directional models now are like at par with bir directional models yeah comment what's your ACC um gosh I don't know what's in the paper but uh it I would say um it's in it's in low single digits like uh sorry the error rate error rate is in yeah so it's a very high 90s is our accuracy and for Chinese it's like it's very very high so for humans uh accuracy rate is 95 96% that's for humans so for mannering be beat it so you're higher than that uh English depends on the accent and the uh as I said on the noise yeah against we did it for the first paper benchmarking against um apis is actually really hard a lot of apis make it really really hard to do it so I think we had this funky thing we had to wire the microphone to the speaker or something like that um to do it and then when we did it for the first paper the Deep speech version one uh people like not not Siri particularly I won't name the company but a company X complained saying that oh you benchmarked against us we have a better system in house um and then we said you know why get into this political fight I mean we just going of say this is what we did on on on these data sets and um if somebody else has a better number they can publish rather than explicitly trying to bench marence something yeah so talk specifically about kind ofy uh I don't I forget these numbers let me not give you the number because I probably give you a wrong number and uh I'll be publicly vilified for it um uh I haven't worked on the pre-production system much i' I mainly built the training system um so I don't have the number but um it is what is required to make it natural for like Google or or by yeah have two questions one one kind of w and one more of this the W one is is do you or anybody you're aware of uh try to add lip reading or maybe do lip reading on its own but I mean humans are better at listening to the language in the bar because we look and we also have we also look at our eyes yes so face face liing and I'll ask the second question I like to and and it seems that the issues say with English and accents are because you are trying to have a universal model well if you know you're going to listen to an East European accent you kind of twist your ears one way if you know you're going to listen to Indian accent it's the other way the other way so having a short reading sample from the person on top of all the learning done in general show yeah so the first question is of course um looking at orthogonal um features like um I or lips and stuff uh we haven't done it I'm not aware of the research very interesting question um as we reach what I call the the theoretical limits of these models we would probably look at that kind of data um it's a little bit challenging for us to use data in the US because uh I don't think you can use YouTube data without Google's permission and that is the largest set of video data um Chinese we have we have a website called I which is the equivalent of um YouTube um in China U so we can do it uh we haven't done it yet uh um but yeah again we probably we have to do it at some point uh the second I forget the second question second question was training on um um whether we can train on um data sets of different accent and use that as a way to not not train just complete slight modification it doesn't take too much to detect the accent and then you can move into the portion yeah I mean uh nrow the field we haven't tried it we we are accent is something that bothers us quite a bit I mean bothers Us in the sense that that the fact that we are not good at uh we're thinking about it we haven't done anything on it uh hopefully this year we are that I mean it's one of our main focuses this here is to actually figure out a way to solve it seems that universality should not be aimed at do yeah yeah I don't have a very good like this is for us an unknown question yeah we we have we have we have lot more research to do surprisingly no no questions on the systems um thank you [Applause] if