Introduction To Deep Learning

scale.bythebay.io: Alexander Tsyplikhin, Introduction To Deep Learning

Recording: scale.bythebay.io: Alexander Tsyplikhin, Introduction To Deep Learning

you thank you or coming my goal is to help data engineers get a better feeling of what deep learning is and it's and the feeling is that it's really quite simple so how many data engineers are there in the room please raise your hand how many data scientists okay nice I wonder like only half of the people raised their hands actually I wonder what would be the other 50% so I'm a PhD in speech biometrics I've been in data science field for about 15 years and I'm with data monsters we are doing custom data projects with great customers and partners if we're talking about deep learning it very very simple it's basically referring to artificial neural networks that contain more than one hidden layer if you see a network with two hidden layers represented by the green dots here it's basically a deep neural network nothing else in it but there is a lot of interesting stuff around it deep learning was found to outperform traditional machine learning on large amounts of training data for different modalities images text sound and time series and this caused a lot of excitement a lot around deep learning and you would know that deep learning requires a lot of compute power because it's a lot of matrix multiplications basically this is how you can depict where deep learning fits into the whole picture of machine learning and artificial intelligence some people confuse deep learning and think that it's kind of similar to AI but actually AI would be represented by this larger circle and would include anything that helps people to be more efficient or productive in the work rule-based systems knowledge based systems knowledge base is sorry and it goes inside with machine learning the classical one which would include things like linear regression or decision trees and then deep learning is here inside why machine learning is different from rule-based systems the main difference is that machine learning uses implicit rules it builds it is capable of building its own rules based on samples of data you present a set of samples to the machine to the algorithm and it is capable of training itself to make predictions for new samples that it hasn't seen before and it is powerful because it can be very flexible it does not depend on an engineer to create those rules and to maintain them updating for every new business situation so machine learning is one step above rule-based systems and deep learning is even more powerful I will explain why classic machine learning tasks would be these ones to classify predict a class of an object like is this a spam email or not spam based on email body is it a man or a woman on the website now based on their behavior activity so it's one or the other or several classes classification regression is predicting a value for a continuous variable like what would be your sales next month what will be the house price on Zillow for example based on location and other parameters and they do not use photos of the houses yet but that that can be a good prediction factor anything else in this category and also cluster ization grouping similar objects without being explicitly told what to group by you just tell the machine I want to efficiently target my users and I'm ready to split them in to ten different segments and we'll see what their actual features are how they are different so you can give the number of segments and the Machine will split the entire database of users by similarity between those users you don't have to explicitly tell the Machine this is the user belonging to cluster 1 this is cluster 2 it figures out that on its own and also ranking and recommendations sorting the data to maximize the value function like click probability or request relevance and recommendations what movies would the user like based on the behavior of other users that look kind of like this one and classic machine learning is very dependent on the features that you select like you would want an expert to explicitly tell the machine for the user behavior or for the house price you need to look at a zip code at the footage of the house etc you need to explicitly provide a machine with the parameters that may be important the machine will be able to create a box an equation that would combine those parameters and help you predict something on the data and the features can be extracted straight away like each price number of purchases from the data set they key can be invented based on the domain intuition by the experts for example what the customer churn relates to and this can be the number of support requests or we know that the money behaves like ethnic value and we need to lock the money if we are making predictions on sales for more complex objects like images and sounds many for many years scientists have been creating specific features to that domain like sift or H of G or M FCC for speech mail frequency capture coefficients for speech recognition some of these features are patented and they are served to train a machine learning model a lot of efforts of engineers was worth a lot more efforts was spent to come up with these features to train a machine learning algorithm a model in deep learning it's different machine learning model which is using deep neural networks is able to figure out what features they the the model needs to care about automatically on its own and the features are often times better than the manual engineered features those that I've mentioned before for example there is a class of neural networks called convolutional neural networks they learn learn hierarchical features from images like they understand that these shapes these transitions between colors are important to classify dogs from cats for example so as a an engineer as a scientist you don't have to invest your time in inventing these features the network will be able to do it on its own and this is very powerful this is what makes deep learning very different from classic modes machine learning so we can depict again these four categories like that rule-based systems they go from input through ham design rules to an output like this is this fraud or not fraud based on the number of transactions from different different countries for example for classic machine learning the system will need these hand design features to then be fed into a machine learning model that will train itself to predict the outputs for deep learning it come ups with hierarchical features that then are used to train a deeper model and to to predict the outputs so it is called an to and learning you get from pixels to categories for images like that for example and this this was revolutionary with deploying the the deep learning is a game-changer in many use cases because it can provide higher quality it can help the data science team to iterate faster sometimes with less investment as I was saying it can use raw data as inputs pixels sonograms words characters whatever and this is opposed to some cases where creating the features is extremely complicated and you cannot use some of those creative features because they're patented and it gets you directly from end to end from pixels to a category from sonogram to characters like by do speech recognition did or for translation Roman text English text to German and that means it can be less data preparation and labeling the AI history actually began 72 years ago in 1943 where when McCulloch Pitts introduced binary threshold neurons and then hub learning have researched how the neurons is adapted inside our brains during the learning process then 1950 Marvin Minsky and Dean Edmunds they've built the first neural network with 40 neurons artificial neurons and then Alan Turing with his famous test and other brilliant ideas like reinforcement learning so the the approach is quite aged something has changed since then now that we are able to use these neural networks in a new way and also the algorithms have been around for thirty years or so like back propagation was formulated in 1970 the cnn's convolutional neural networks were invented in 1980 are announced LS TMS CTC the connection is temporal classification which is very popular now it was all invented a while ago the difference so and there is a quite a history for neural networks and different approaches to solving these problems the AI has been seeing a lot of ups and downs and now we're kind of in an up over height state of the market and of the industry but as you would remember many years ago like 1956 John McCarthy said we think that the significant advance can be made if more in one or more of these problems if a carefully selected group of scientists work on it together for a summer when 1965 machines will be capable within 20 years of doing any work that a man can do so we did not see that that resulted in AI winters several AAA winters throughout the last year's I believe we might see a new AI winter now we need to be very careful about our promises what AI can do and cannot actually do AI meaning deep learning in this case but a lot of people are attracted to this field and that inspires a lot the difference is that the large datasets became available approximately after 2001 and throughout the whole history of computer computer science we've been focusing on algorithms and now we have seen that having more data can be even more powerful than creating a very efficient algorithm and this table shows that the availability of the algorithm on the right side stands like 18 years apart from a major breakthrough in this area while the data set is only three years before a breakthrough breakthrough appeared so the it'll appear to be really really important for many of these breakthroughs for example in speech recognition technologies that used classic machine learning hidden Markov models and Gaussian mixture models were not very good at using lots of training hours of data while with deep learning the difference from small training set to a larger data set training set was significant so we have found that data is important for deep learning to be efficient this is a recent study by Google they've trained a deep learning model on their scale using three hundred thousand three hundred million images fed into the machine learning model and they have found that minimum mean average precision was growing with the increase of the data amount in the data set which means and there is no plateauing nothing like saying that we'll get to a limit somewhere so we need more images to be very efficient with computer vision and that's why Google allows you to upload your photos and Google photos with for free with no limits I would say this use case this article in 2011 was the first break significant breakthrough for deep learning and this one was with road signs at at that time deep learning well committee of convolutional neural networks performed better than humans at recognizing what road sign is this in the picture and that was incredible nobody expected that we will get there that soon so it was two times better than humans with the error rate of zero to zero point 54 percent or rate the next one was in two years later we are recognizing house numbers on Google Street Maps Street View Maps humans did 2% error rate machine was nine point ninety four percent and then the image net contest million images human performance was at about four percent and in 2015 the researchers came up with quite a complicated neural network with one hundred fifty two layers that outperformed humans in this task and that was one of major breakthroughs so these are some of the examples that the of the images of classes of images that the neural network was able to recognize so I'm saying that I would not be able to ever recognize these three images I don't know what kind of dog it is but the neural network was trained on it and was efficient some of the mistakes that it had that it made that GT stands for ground truth the real label given to the image by a label or human and we can see that well it actually looks kind of similar to what the Machine came up with acoustic guitar altar wine bottle instead of restaurant like we were expecting to see restaurant as the tag but there is actually a wine bottle something in this series is also very very credible plate meatloaf ice cream kind of ice cream yeah so we have seen that the machines are quite powerful there lots of data and new architectures of neural networks we have also seen that they can be easily cheated in some cases for example this image of panda if you are able to engineer the noise that you would add to this image with a very very small weight which visually results in an image absolutely similar to a human I would trick the machine while giving a completely different label to it so we need to be very conscious about using machine learning and deep learning at for predicting things and there is a big movement in in this area like should we what should we trust and Trust to the machines etc so these are the three things that make deep learning very successful the deep neural networks the new ways to combine mathematical entities together availability of hue jimin's of data and the compute power the GPUs so this is an example of how more data and more compute power made a jump from the lunette case which had which was trained on to to power of seven pixels with these many this many transistors and in 2012 alex net was using much more pixels and much more transistors to be trained and we can still compare it to the human brain Alex Knapp was less than a million units and human has 100 billion neurons so I think we have not reached the limits of these algorithms the important thing was in that in 2010 in Switzerland they've issued a paper finding that training neural networks on GPUs gave a speed-up of about 50 times 54 speed-up and this is because GPUs can perform a lot of computations in parallel because they have thousands of cores in comparison CPUs have like a very finite number of it like 8 so deep learning has a lot of matrix multiplications inside of it and GPUs were designed to do that very operation so Nvidia is now very happy because they can serve a whole new market need whole new market segments with their GPUs built for a different goal and well many players are trying to catch up but it's challenging we can see this is a slide from NVIDIA GPU tech conference from this May last May GPU compute performance is growing fifty percent per year CPUs only 10% because GPUs can leverage those parallel computations much more efficiently so Nvidia is in a very good position and we are as well now we can leverage GPUs to do a lot of computations for deploying and get to very interesting results some of the tasks that deploying was strong in our speech images and video for example classification and localization what do we see on an image and where the object is located the boundary boundaries bonding box of an image and with a lot of data neural networks outperform humans for these tasks and also enhance do semantic segmentation pixel-perfect segmentation meaning which pixel on a in on an image belongs to a specific category this is broccoli this is road sign this is a car this is a pedestrian then style transferred making your pictures look like an artwork and our piece image generation and robert perception emerging area I would say is text and NLP natural language processing and we have seen word embeddings and different language models to recognize speech from audio to text than sequence to sequence learning for machine translation and also the things that is emerging now is quench question answering and dialogue systems and tag sternum generation these last two are not solved yet would be exciting to see this salt and also reinforcement learning robots games like you've seen Atari go and dota 2 recently and alpha go zero and the things that we I think we would be seeing soon is combining several modalities for example images and text doing image captioning for example or searching the web by image we can do it already video descriptions like translating sequence of images into text what is happening in an image that would be very useful for security purposes for example visual question-answering in an image given an image the machine should answer to a question formulated as text speech recognition from video that enhancement by articulation of a person in in the video and many other things like extracting depth from an RGB image well I think we can expect a common representation for several modalities embeddings not only for words but also for images or for thought vectors so that we could be using similar algorithms to train the machine to understand the world and make predictions on the world in general not only on a narrow task of problems and we are also seeing that convolutional neural networks that were created initially for images are now used for text and vice versa are Nan's created for time series and texts work well on images for example on videos so this transfer of ideas between domains is very powerful one of the new challenges with deep learning and neural networks is that instead of engineering the features you now have to engineer the architecture of these neural networks creating the right architecture will help the network to be efficient for a specific task or a set of problems so people are Korea getting all kinds of architectures for the neural networks and they look beautifully and differently if at your company will be thinking of doing starting into deep learning you may encounter some barriers to entry you will need lots of data like thousand data samples per class for images lots of compute power the more power you have the faster you can iterate on your models and tasks and you will need expertise in deep learning deep neural network architectures because it can be important in many cases and some of the an advice well my advice would be to overcome potentially those barriers by using transfer learning because it can it means that a system a neural network can be trained on another data set on another set of images and then then you will apply that pre-trained network partially to your data set your categories like you need to recognize one competing brand versus the other competing brand and you would not have hundreds of thousands of samples of that kind of products you can train your neural network on another data set publicly available and transfer learning will help you solve it for your specific case and also many data sets are publicly available there are marketplaces for data sets you can acquire some of those from different vendors and you can collect your data and hire a company to label the data for you because deep learning well is typically trained on labelled data in most cases the need for lots of compute power also can be tackle by using transfer learning and also if you do not have a need to run a lot of computations continuously you can just rent a GPU in the cloud and two weeks ago Nvidia and AWS have announced the availability of their invidious new GPUs Voltas in the cloud their top line of GPUs so this is really powerful and overcoming the the lack of expertise in deep neural networks can be also easy in specific cases many real-world tasks can be solved using standard architectures well and if you have lots and lots of compute power you can Google had shown that you can replace expertise in deep learning with a machine that will be finding the optimal configuration of the neural network to train itself instead of having a data scientist do that a little bit on how it works inside so it was inspired by the neurons in the brain and a neuron has then writes the input signals that come in to a cell body get trance translated into an output signal somehow and then go through the axon to be transmitted to potentially other other neurons or muscles so this is a simple a simplified model that helps us understand what's what's important in the reality it's a little bit more complicated but we can forget about it for now and try to be simple to keep it simple so we model the neuron in the brain by collecting the inputs like an accent from another neuron comes in and this is the the feature that we observe in the in the world and we have several features X 0 X sub 1 X sub 2 coming in and we apply a weight we just multiply the input value by the weight that will have to learn somehow we combine it we sum it together seem as simple as this and we add some bias here can be just 1 to put it together like this and we apply an activation function to this value so it's just the sum of n weighted inputs and a function to calculate the output so the output axle is the value of that function and there are different activation functions for example this is the sigmoid function it goes from 0 in Europe in minus infinity to 1 somewhere here and it equals to 0.5 in the middle and there are different ones like that binary threshold neuron 72 years ago is like this there are also all kinds of activation functions invented and tried by humans why do we do this because we want to find an efficient configuration for a specific task so one of the acts one of the important elements of the architecture for neural networks is having the right activation function for neurons so we have the hidden layer so-called hidden layer for a neural network we have a set of inputs they observed features of a neural network and we have the output layer just the terms that are typically used here and it works like this you if you have several layers it's just as simple as with one neuron but instead of getting the output from right from this neuron here you get this output and use it as an input to the next layer so all the artificial neurons on the previous layer get connected to the next layer and so on and so forth so the output function here will be the sum of inputs at this level before the output layer of the neural network and there is a very efficient technique that made deep learning possible it's called forward propagation and backward propagation it allows us to learn these weights that we use to calculate the activation function from the examples for a specific sample that we data sample that we have observed we we based on the weights that we came up with maybe initially it can be just a random set of ways we measure the distance between the output that we have calculated with our weights and the desired result like we wanted the house price to be predicted as a thousand we got a hundred so we would use that Delta and we we will propagate it back and update all the weights of the neural network so that the next time the network is given this example it will get closer to the desired result and there also a third important element is the loss function how you measure the distance between the desired outcome and what you actually observe it can be very efficient well a specific loss function can help you get to the result much more efficiently it has been found that creating deep networks where all the neurons are connected to the previous layer is inefficient and networks with three layers are typically well four and so on do not get you any performance boost in accuracy so other types of neural networks have been created like convolutional neural networks it basically calculates a weighted sum from pixels of an image and it uses this matrix to calculate the activation function for every neuron on the next layer so instead of 100 weights it just uses nine weights 4 3x3 cube to calculate the entire next layer and it comes up with different features for example for an image of a cat it shows some activations on different layers which seem quite random but the parents it's looking for these are weights weight functions for these matrices that if the network comes up with they are they seem to be meaningful so and CNN solve object classification object detection semantic segmentation and are used for sale driving cars there is another kind of neural network that help has a notion of time the neural the neurons the artificial neurons on the hidden layer get connected to the neurons on the next time step within the same layer and you can unfold the recurrent neural network like this so we have an activation function at this time step and this function gets inputted in this the same layer on the next time step and the recurrent neural networks are good at dealing with time series but they have a problem of vanishing gradients so the network in its outputs tend to forget about what's what happened some time ago and to solve that European Schmidt Ober had came up with the concept of LS TM a network that keeps its internal state with the ability to remember it when needed when something important occurs and it has so-called right gate to write down a new value for this internal state so this is the forged gate forget so it zeros out some of the values of internal state this is the right to update the state and this is the output gate so it changes zeros once and in between to modify the state and output the values and it solved the problem of vanishing gradients and the some of the use cases for neural networks work like this given a set of phrases that network can reply to a question where is the Apple after some movements happening to the Apple or what kind of trees in the background are given an image machine translation is getting to the perfect level in many languages because of be directional LS TMS this is a an example of machine creating caption for an image group of people shopping at an outdoor market an image chlorination coming up with a color for two pixels in a black and white image train on previous images alphago still driving cars dance captioning what's happening in an image with a very high detail level making photos look like van Gogh pictures coming up with new volcanoes or new kinds of birds based on description of this bird is completely red etc and robots I would not be seeing this and some of well I I guess I do not have time but some of the industrial use cases are really inspiring like this one in sports track players or create automated video highlights in tennis or something like that so the conclusion is that we can think of deep learning as of automation on steroids if you have a lot of data and the accuracy for your use case is critical you should be looking at deploying and the more compute power you have the more ideas you can test and faster you can iterate and in this slide I have a set of links that you can use to dive deeper from like self teaching part to following the thought leaders etc and please email me for the slider calyx of data monsters calm thank you [Applause] if there are any questions we still have it I'm fascinated by the fascinated by the adversarial penguins of pandas and so from the human Ally you actually can't see the however apparently there's some very subtle well whatever just a picture of the adversarial Panda yes so that's a that's a good way to augment the data set like you try to trick trick the system and you use that new data samples to train the system and say no that's a panda so this is another way to get more training data by using pictures that look similar to a human but would improve the performance of a machine however it will be still possible to come up with an adversarial example even if you do a lot of those artificial samples so for a lot of the tech space you know input to neural net CSU's weird effector to convert words to so the vector space but it seems like it's going away from that - just straight characters I mean yeah are there still cases were like using or detecting sense definitely yes well we're Tyvek is a one of the techniques that made embeddings of words very efficient there are others they can account for different other aspects of the words not only taking care of the neighborhood of a word but also some what me aspects related to the meaning of the world word and also the words can be misspelled and also there can be new words combined from separate pieces of words like those suffixes les ness so there is the right projects that take those engrams of characters into account and they augment were two vector kicks to account for unknown words as well and there are Studies on that thank you [Applause]