Adam Gibson, DeepLearning4j on Spark and Data Science on JVM with nd4j, SF Spark @Galvanize 20150212
okay so I'm Adam creator of deep learning forj as Alexi has kind of already said um first of all I'd like to screwing to me man hey come on all right so one one thing I'd like to announce is I'm writing the O'Reilly book on deep learning deep learning a practitioners approach with my co-author Josh Patterson he present he co-presented with me at Hadoop Summit last year speaking of which we're also going to be present we're actually going to try to present deeping on spark this year so we did yarn we prototyped it got some good stuff going and now we're going to be doing deep learning on spark so what I'm going to present to you here today is mainly going to be focused on the architecture so getting you familiar with neuron Nets because you know actually let's let's stop there how many you guys actually know about neural Nets like okay how many of you guys have actually coded with them and actually used them in production woo all right more than I thought okay so how many of those were just back propagation or did you actually use deep learning okay yeah back propagation welcome to the 80s guys all right what I'm here today I'm here to cat you up to 2015 so I'm going to cover today one the core architectures that involve different kinds of deep learning Nets and then I'm actually going to kind of bring it back down to the code so you're going to see visuals paired with code so the idea here today is to get a practical understanding of neural Nets how I apply them and what in my opinion is actually a modern machine learning architecture that allows you to plug and play with different kinds of knobs and everything's configurable so the problem with current machine learning libraries you know this could be WCA this could be MLB yep I'm trash talking um you know things are brittle things are preconfigured and in general things don't tend to be you you can't do anything like you have to submit a PO request for spark whenever you want to update it this is as a spark 1.2 if there's any ml guys in the audience bring it let's you know inform me teach me something because what I did was I I went straight to building on top of spark I obviously I'm an implementor of a deep learning framework um and I specialize in neuron Nets I've done machine learning for a number of years I've used the algorithms but there's been there's been distributed machine learning algorithms around for a long long time spark is just faster which you know obviously it's the deao platform today for any sort of map produ Paradigm Computing so that being said it's worth building on top of um you know but I have my own opinion about what an architecture looks like and that's what I'm going to present to you here today so to start I love the jvm I've been doing it for a number of years um you know distribute Computing and where the the data lives that's all on the jvm you know there's basically you know if you think about the Hadoop ecosystem spark it's basically a fork of Google's ecosystem in open source with something that you use on a day-to-day basis Hadoop is now over 10 years old spark has been around for quite a few year now few years now as well it's now a top level project in the Apache foundation so things things are great for the jvm the jvm is still the future um so what I'd like to do is kind of bridge the gaps between into Computing in research with gpus like what a lot of these guys emphasize and Native Computing which you actually have in your data center all brought back down to your spark or Hadoop clusters so to start um so what is deep learning you know why why why should you care about this what what's going on here you know most of you guys are doing machine learning you guys are probably specialists in some domain where you know every single little piece of your data you understand what the features are you understand what things do you know but you know at the end of the day if you're trying to do text if you're trying to do unstructured data you know there's no there's nothing to make sense of that if I'm an NLP practitioner which that's that was primarily what I did before doing deep learning you know you got to you got to think about well if there's a suffix here it begins with a capital letter or maybe if you're doing images you know there's all sorts of crazy transforms you have to do you have to build Edge detectors you have to apply filters you and it gets brittle you know so imet for example before deep learning came along with convolutional neural networks by Lun group and everybody else at Google and everybody else um they you you basically had a series of hacks so you had uh you know you had all these different Auto feature engineering libraries you see open CV you know open CV is just an abstraction for a lot of different kinds of feature engineering deep learning actually gives you an elegant solution to feature engineering with unstructured data all in one all in one algorithm so the the problem with neural Nets though is they're slow they're really slow they're impractical for day-to-day use so what I'm here to do today is to place an emphasis on distributed computing and actually make it so that you can iterate on models and use it in your dayto day because the you know the problem with neuron Nets is that it they also require a special pipeline so while MLB is amazing you know it's fast real time there's streaming there's all sorts of algorithms neural Nets require a special kind of pipeline the features of neural Nets are typically output from other neuron Nets so I'll I'll actually demonstrate that here today and kind of give you an overview of like why you should care how you should think about neural Nets and then applying that back to distributed deep learning so now that we've kind of seen deep learning what is so let's let's talk about this feature engineering thing so why is it hard so you know humans perceive cats humans perceive but computers only see numbers you see pixels 0 to 255 what do you do with that you apply Edge detectors you smooth out the image you get rid of noise think of it very similar to photoshop what I'm here to do is actually show you that deep learning basically has a built-in Photoshop system that actually knows what filters to apply and when so the the core idea with deep learning is unstructured data and automatic feature engineering so what's it good for basically any machine learning application focused on unstructured data while deep learning is great for other algorithms like or other you know other kinds of problems you know it won the Netflix prize it was part of the Netflix piiz with recommenders it's broken records in image and speech analysis both of the systems that are you know in like in speech recognition in face detection all records broken by Deep leing all with automatic feature engineering so you know think of it as basically a subset of machine learning that gives you automatic feature engineering over unstructured data so what can deep learning do that other applications can how about image caption generation this was the output of this use the output of a convolutional neural network trained on images neural word embeddings from word DEC I'm giving you a bunch of terms to Google search I'll be more than glad to answer questions later and what's what's called a recur recurrent neural network uh or a long short-term memory neural network so a convolutional neural network was trained on photos a neural word embedding was trained on a corpus and then you have a recurrent neural network that learns to actually generate the captions so I'm going to go into what each of those neural networks are this is actually this was some this was actually a fresh result just from last November and several several actual companies have actually been able to produce results with this so if you want to automatically describe images if you want to augment search you use neural networks as features so how does deep learning actually work so deep learning focuses on this automatic representation learning you start with raw data so focusing on Vision because that's the canonical use case we have pixels you know we have cat pictures like I mentioned earlier and I want to train it to think about cats so we'll start by first building an edge detector so you have one neural network that allows you to kind of detect edges you know it kind of figures out like you know just direct patterns then you have the next layer that actually learns well oh there's there's like these edges actually form parts of a face or parts of a um any an animal you know then you finally have the final layer what actually which actually figures out what a full cat looks like and it's what basically what you do with deep learning is you have three or more layers that's the Deep part of it you initialize you initialize it on the raw data with one neural net you take the output of one pass it to another it learns the next set it learns the next set of representations and so on and then all you do is you tag logistic regression on the end of it and you have a classifier the key here is that these neural nets all do a form of feature engineering the automatic feature engineering works with unstructured data learns the patterns and gives you a good representation that you can then use for Cutting Edge classification so just to give you an example so this is actually from deporting 4J so this is actual this this was run on the labeled faces in the wild data set the neural network was a restrictive boltson machine and it actually outputed actual faces these are actual neurons the the weight M the weight renders you see are actual neurons it actually learns to reconstruct the data so in general this is done with standard convex optimization techniques gradient descent uh lbfgs conjugate gradient anything that basically minimizes an error with respect to a loss function a neural network itself is a universal approximator that can learn any function in this case it also does a nonlinear transform on the data to give itself a representation that it can then use for feature engineering so now what I'm going to do today is I'm actually going to open up the box on on on neural Nets in some Frameworks you do Val type is equal to a new deep learning what does that even mean what I'm actually here to show you today is what the different neural network architectures are how they work and I'm here to open up the black box so that you can actually understand what kinds of neural Nets are there what you know how to think about deep learning and how to actually build your own applications in a practical context you know part of the problem with deep learning today is that most of the Frameworks are either requiring you to derive your own equations or you know sometimes you have an automatic der derivation engine you know if you're looking at theano or you know you you can't scale it out so you can't work it on a distributed system so what I'm heing today is actually kind of show you how to bridge all these gaps and actually give you something comprehensive that's practical and actually realistic for deep learning applications so the first is the canonical feed forward neural network a a feed forward neural network has an input layer a hidden layer which is the nonlinear transform think of that like the sigmoid function or any of these other kinds of algorithms and the output layer it's a series of logistic regressions that basically output yes or no that's that was that was the first representation that thought of the brain where the neurons fire you know so but you can in general think of neural networks as a 2d logistic regression you can actually use that analogy to learn most about neural networks so what I'm here so you know so we've seen this there's feif forward and back propagation so that's the standard way that's the standard that's the only neural network most people see so one thing I'd like to tell you about is that there's actually a two there's actually a feedforward architecture you can use for automatic representation learning so for those of you who are familiar with standard back propagation neural networks you have an input on each neuron you have an input you have a connection weight and you have a bias what this actually get so what happens if you add another bias though so that's called an auto encoder there two biases one on the visible that's attached to the input that knows about the original data and the output the output is what you think of with the normal bias on a back propagation neural network what it does is it minimizes the difference the in this case for those of you who want to Google search something K Divergence or the Reconstruction cross entropy of the the the original data and the output so those faces that you saw were actually the connection weights turning on and activating on faces so I tra I had a visual bias that was attached to the original image and I had the output and it minimized the difference between the input and the output which gave you those reconstructions for vision specifically that's actually a debugging tool for neural networks you don't actually debug them just by changing a number running an F1 score debug them visually so the problem the problem with working on a cluster is it's not it's not exactly obvious how you how you troubleshoot something especially with with something as visual as neural networks where there's all sorts of knobs you have to worry about you you want to debug them visually deting 4js architecture actually allows you to prototype the algorithm in local just like you would in Python and then scale it out in a cluster without rewriting one line of code so debug visually and locally work with Vector work with vectorization in either spark or in deporting 4J and then scale it out in a cluster so another kind recurrent neural networks so just like I mentioned earlier so recurrent neural networks are for time series so therefore for learning they cannot so they they learn sequences so that image caption generation you were seeing was actually a neural network that pretends to be a hidden markof model so it actually learns a sequence so what it does is I feed it features in and it learns how to it learns how to predict future values um this has been done with this has been done with all sorts of algorith like time series text anything where there's a sequence of something you can actually use so it's a for those of you who know machine learning it's a generative model so again it also learns features so the next one is a convolutional neural network a convolutional neural network is the best thing for vision it's also broken records in sentiment analysis as well as recently by Yan Lun neural like uh word embeddings so you can actually train a neural net to recognize individual characters and build a better language model so a convolutional neural network the basic idea is it breaks a space up into Parts called a feature map it and what it does is it it actually it it almost like uses like a little ey and then it just and then it trains each part of the eye to identify different parts of the different parts of an image or different parts of a problem so those feature maps that you're seeing then get aggregated or pulled and then what you get is basically an aggregation of all the different uh parts of a problem into one space Not only do you get reduced not only do you get reduced space reduced memory but you also get a generalized representation that's position invariant the other one this is a little this is a little academic uh for some people another one is called a recursive neural network a recursive neural network is basically mainly for text it's it can also be used to break apart a problem for vision the idea is that you have you have a tree you have a binary tree specifically and you basically take two parts the children you combine them you compute a score take the next one so another another child and you take the output the aggregation of the two combine them and then you go up the tree and then what you get is a representation that gives you something that learns sequences or individual contacts so this is called back prop this is this uses something called back propagation through structure so other than that let me show you one more visualization of scene parsing and then let's get into some code so one one of the things I wanted to do here today was kind of get you familiar with like just the different kinds of neural network architectures in deep learning 4J I've implemented all these architectures all these are all these are the core representations that you think of as a layer so that's what I'm going to kind of show you next is actual some actual code now so this this this is breaking AP part of scene just like just like what we represented before so from here what I'm I'm going to do is I'm actually going to show like different parts of the code now let's see so so the first thing to think about is that everything is a layer so when you're when you're thinking about these different kinds of neural networks and you want to mix and match them to do different things everything's a layer so these layers then do B these layers basically have all this idea of mini batch training so you'll see you'll see like activations inputs outputs it's just like what you see with any any neural network so any recursive neural network or any of these other kinds of neural Nets will actually just be this representation so you combine what you typically do with deep learning is you combine them together into a multi-layer network that you then use to run your classification or do your generation or what have you so think of everything every like just implement this interface if you want to create your own layer so if you want to create your own neural network it's actually a plugin that's it so basically it's a general purpose architecture for neural networks so rather than treating the idea here is that rather than treating them as a black box learn them combine them and use them learn about some of these new applications because the key here is that neural networks are used in combination with with one another to produce a lot of these results that The Cutting Edge results you're seeing today so let me see so the other thing okay so the other thing then is this idea of a multi layer Network so the multi-layer network is how you combine layers so you combine layers by taking the the output of one and putting it into another so this is how you build convolutional neural networks this is how you take the output of like a recursive net and pipe you know pipe that into the you know another neural network and then you have some form some form of final output layer you then use you then use that in in combination with other neural Nets to I don't know you can do class ification with it you can generate image descriptions doesn't matter key thing here this is actually how you aggregate other neural networks so basically the the idea is this is basically a support class for you to implement your own neural networks you have a general purpose architecture that you configure in core and then you scale out with spark so before I get into any other code are there are there any intermittent questions or anything in particular people thing people want to see yes no one okay one pass so okay yeah say something please can you give it a white backg little EAS uh yeah I didn't I didn't expect that um hold on let me see um actually you know what let's go to GitHub all right so it's open source people just remember that so incore okay better okay all right good now okay so here's here's the thing a multi-layer neural network is literally a series of layers the way I designed it every neural network is a plugin that's what I was talking about you take the output of one and put it into another um did you did you guys want me to show layers again or did you guys get the general idea I mean basically think think back to the visualizations every one of those is an is every one of those is a layer that you then use for the multi-layer neural network Okay so let me go ahead and actually go back okay I can't see that one second all right so conf okay muler configuration all right okay so so like I mentioned earlier the other thing is this idea the other thing everything's configurable so neural so basically what you do is you actually create a multi-layer configuration this configuration is what basically allows you to instantiate a neural network you then basically how it works with spark is you basically take this configuration you use the autocomplete the IDE so remember Scala and Java are statically typed languages use the tools presented to you use the tools that you're given allow allow the ID actually help you actually discover what the parameters are and what the knobs are and allow yourself to ask questions so the multi-layer configuration is basically built in the IDE serialized to Json and given to spark so basically in core you prototype everything locally you run the visualization pipelines that I was showing earlier in this case working with the renders if you're doing Vision or if you're working with text I actually have something called t neighbor embedding built into the framework that actually allows you to group and visualize words so one of the one of the cruxes of deing with text and why a lot of people haven't done it is because they don't actually understand how to use neur word embeddings um I'm not going to go into that rabit hole right now but I'll say this you basically use you can you you debug neural networks visually um images and text are by far the major ones but another thing you can actually do with neural networks is you can also debug them with histogram so one of the things one of the things you one of the one of the tricks in neural networks is you want to actually monitor how fast they're training how fast they're learning if they learn too fast with too few iterations they're going to overfit and you don't have an accurate classifier so do everything visually prototype it locally um the the histogramming support I have right now is currently uh I'm not I'm not I don't like this but matplot liit so we're actually using that to actually render uh a lot of a lot of the visualization tools um if any of you guys have any jvm based visualizations or web based uh I would love to hear from you um so in general prototype locally scale out with spark so what I want to do real quick is just show show so basically show some example usage now so I'll just pull this down here oops so what does this actually look like then see so actually I have all right all right so what I'll do is let me let me show this actually related to spark now that's what you came here for right all right so what I actually have a sample I actually already have a sample application on spark uh that gives you that gives you kind of an idea what to do so let's let's just kind of go through this now so uh can you guys see it good so in spark you set up your configuration you know so in this case I have I set I set a few different options in here I'm just using arcs 4J to parse the results and basically all I'm doing is I'm setting up a configuration so like I mentioned earlier you use the ID autocomplete to basically give you kind of like a a discovery mechanism for figuring out what options are present how do I configure neural network you know so neural networks one thing to note here is that all of them have this idea of hyper parameters neural networks have over 60 hyperparameters and you have to understand how each one works and and how they interact with each other there's a variety of there's there's an infinite number of configurations that you can do so one of the things I want to one of the things I want to mention here is that you'll notice how I have this idea of a layer Factory so Factory just you know sounds Java I I come from that world uh you're you're more than welcome to give me a poll request for something Scala based um but basically you have a layer Factory that knows how to create a layer so that layer is basically how you create a series of Stack neural Nets that learn features so remember deep learning is hierarchical you take the output of one it learns the raw inputs take the output take the output of the first one pipe that into the next one Etc and then you build your classifier so basically what I do here is I take a I take a layer Factory I say I want a restrictive bolts machine a restrictive bolts machine is that two bias neural network I was talking about that basically did automatic feature engineering so it learns the features it learns faces it learns OCR handwriting whatever you can think of it learns those things and then gives you the and then basically what you do then is you say Okay I want logistic I want I want logistic regression so what I do is I say okay override only the third configuration and give me softmax so for those of you who aren't familiar softmax is basically a probability distribution over a set of labels so in this case I'm saying give me softmax let me let me basically minimize the error with respect to the guesses or the likelihood of a particular label and then the output the the true labels are a binary Matrix where you say the True Result is one or if I have three labels and it's label one it's one so 0 1 0 and then it says 30% 60% 10% and then you minimize the difference between the two so that is basically that's that's softmax regression that's multinomial logistic regression um what's really weird here is as of spark 1.2 I didn't actually see uh I didn't actually see multinomial logistic regression in there I saw some talks no code though kind of strange um the other thing I want to mention here is that you know you know spark is spark is awesome as a runtime but one of the I have a problem when I see classes logistic regression with lbfgs what you know why can't I just say give me an output layer and the optimization algorithm is lbfgs why can't I just pick which one I want to use why do I have to change why do I have to change the class completely in order to give myself something that allows me to run lbfgs with logistic regression so I emphasize a configuration based approach here and then what I do is I scale this out with Spar so I minimize multiclass cross entropy that's the difference between that's the TR that's minimizing the true minus the guess and then I say okay now let me create something let me create spark dl4j multi-layer so all I do is I pass it a configuration and a spark context the spark context is basically your drop in point in spark so you specify your master or you run it locally you create your spark context you give that then to spark dl4j multilayer in this case I'm setting up the network um you know you can use you can use Sparks rdd support for shuffling uh train test split and all those these other machine learning algorithms you do um or you know or you can use dl4j built-in support for a lot of this already there's a lot of overlap and I I encourage interrupt between the two so I actually have an ml Li util class that gives you your familiar label points allows you to predict with vectors and all the things that you're familiar with so underneath um for those of you who are curious basically this is this is basically me using nd4j here so basically I actually could have a backend that delegated debris or in these other linear algebra libraries so dl4j basically just has this idea of a data set that's basically just a pair of input and output matrices and then all I do is I say fit data set or in this case I also Al have a I also have the static indication that's just like ml lives we can just say do train spark context labeled points so I have that as well so both both work um both will give you basically both will give you spark dl4j multi-layer back that then underneath has a multi-layer network so you're like where so like where does this come from then so so what I'd like to do now is go into the scaleout part of this so one of the things that's not apparent is I'm training so let's let's think about this for a second I'm training on core I prototyped my neural net I got my good F1 score I got my good training training accuracy on you know a certain kind of small sample size now I want to train on the whole data set what does that actually look like so hold on so see I'm just going to do this real quick apologies for the switching it's like code and this and that um so one of the things I want to demonstrate here today is parameter averaging so that's something that that's something not a lot of people have seen um or or even really think about because they just use whatever whatever Library they're they're given so one of the ideas with one of the ideas with with spark is this idea of workers so basically this is this is our algorithm called iterative reduce so iterative reduce was co-created with my with my co-author Josh Patterson he's been he was employee number 34 clera and new distributed machine learning um and he came up with this uh in collaboration with other with other researchers as well so basically what you did so this is a this is a variation on Hog Wild this is also done at Google with Google brain all you do is you train a mini batch so what's a mini batch a mini batch is just a subset of your data set so I say if I have 100 examples and I have 10 core I send 10 of those examples to each core I then train a neural net on each core it minimizes the error with respect to that mini batch only and then all you do is after the after that particular worker's done training it basically sends it up to a master so you say output of all those reduce the reduce is the average part and you do this for n iterations so in this case I train everything on I train everything on multiple cores in spark and then you could also do this in gpus or what have you but you either way you have a neur net on each core you have end models you break it up you partition it you train you average and then what you get is essentially a modular neural network framework that allows you to scale out on spark so in this case prototype oncore think of one neural net that you're training oncore as one worker and scale out with spark where you instantiate a neural net per core and you average it's like it's literally that simple this is how Google brain works um you know so in this case this this Maps really well to the functional Paradigm so what does that actually look like iterative reduce itself in spark is only a few lines of code it's actually pretty easy so what I'll do is I'll what I'll do is I'll show that now all right so oops all right so let me just uh this all right cool okay so and then after after this I'm going to I'm going to kind of bring it all together and then I'll and then I'll be open for about 10 minutes of questions okay so okay so this is the actual so opening up the black box here let's let's let's just let's just make some observations here so first of all I have my spark context so SC is basically like you know that's that you see that in all the all the examples sc. text file sc. whatever that's your spark context I I use that to access whatever data that you're trying to train on so then I have my configuration and then basically what I do is I I basically have the resultant multi-layer neural network that was taken that was basically a averaged so those are the main those are the main things to be concerned about I guess one one other thing of note is the broadcast here so I don't know I don't know how many of those you are familiar with broadcast but broadcast is basically this thing in spark that says let me only send data to a particular node or computer once so what I do is I I create a readon copy of the parameters so one of the neat things with neural networks is that basically you can Tre you can treat a neural network as two parts a configuration and a layer architecture and parameters so the parameters are just each each Matrix lined up as one long Vector so I then what I do is propagate a copy of the parameters after initialization of of the multi-layer neural network to each node and then I only send it once and then what I do is per worker I'll instantiate it so again the parameters are just one long Vector so I'll break that down as well so all I do you know dep you know obviously with Java it's not it's not as awesome as some of these other languages I have to do Constructor overloading to give you kind of you know like less verose construct rors so that's that's that's all this is the core here is you instantiate with with a spark context and some sort of and then some sort of configuration and then it handles the rest all right so here's the here so here's the underlying fit so or here here's one way to do it you can just read in like a text file and in this case we have our points here so the points used something called so this this implementation uses something called kova which basically Kenova is our vectorization framework that basically allows you to take like an image or a WB file and turn it into a CSV so orm or you can translate between SPM White And livm So It basically gives you one interface for handling all kinds of for handling all kinds of data and then you can basically instantiate it then with a data set or in this case Kenova itself is neutral to the kind of data type so you could actually translate into a doubles array or what have you but it's basically a vectorization framework so you know in in the usual spark fashion you have a function that basically does everything for you so I do lines. map I align over the data set I return the points and then I do fit data set here so fit data set is the core okay so this is this is the me so let let's let's break this down so I get I basically get the number of iteration ations so each neural network basically is one convex optimization problem where you're minimizing error you terminate after an iterations you have the count so it's like what is the number of examples I have the batch size so I say okay given a batch size let me evenly partition this across the cluster I have an I basically I basically calculate a batch size make sure that basically each neural network has an evenly distributed part of the data you know because you don't you if you if you want the average you need to evenly partition it h then I use then I use Sparks rdd repartition that repartitions the data into even even batches and gives you something that will basically make parameter averaging work well and then here's basically here's basically the instantiation of the parameters so I create the multi-layer neural network you'll notice I'm just using core here it's actually pretty simple I'm I'm using core I'm in stti in the neural network so that same configuration you were using in core when you were prototyping you can scale out with spark so I created a neural network with a configuration that you specified earlier I initialize it that gives me my parameters it sets up like all the architectures debugs it make sure it's set to go then I I basically grab the parameters as one long Vector so it goes through and it says rbm1 2 and three become one become one layer or this could be a convolutional neural network or whatever whatever configuration you set set up it basically squashes it into one one parameters and then it broadcasts it to the cluster so it says given given given these parameters let me send this to each worker then just some validation here nothing crazy and now this is the interesting part so I say all right just back out a little bit here okay so all I'm doing is I'm saying given given my partitions run iter to reduce in this case my parameter averaging collapse all the vectors and give me back basically the average so I I get back all the results all the all the different all the different parameters I only send the vector that's the key thing here I don't send a bunch of other neural network architectures and all the Prof I only send the original Vector back I send the original Vector I get that into an rdd and then I average I mean it's literally that simple I accumulate them all with that ad function and then I I set up I I set up the average or the the the initial Vector the the first one I sum them and then divide by the number of partitions and then that gives you your average it's actually literally it's literally that simple it's not not anything crazy so and then in this case all I do is I say set parameters so set the average and return that neural network and that's intive reduce um dead simple especially with Spark's uh functional programming paradigms nice and clean you know not too many lines of code um automatic scaleout tricks and everything that a distributed systems engineer needs all in one go so the key thing here now is prototype locally you know I made deting 4J such that I hand derive the gradients myself and and I basically used I basically I I I basically hand optimized each worker to basically to I don't know to run as fast as you can think of um and then you scale that up that's literally it so now the so now we so now we have the distributed system so we know how we know how to scale it on spark so what about uh what about GPU what what about spark with Cuda how would that work so in this case this is spark with Cuda too this is spark with Native just like Alexi mentioned you basically all you do is you include a jar file that says I want to use native I want to use Cuda it's just like cross linking with these native libraries without actually having to compile all you do is change a line in your your pom.xml your build.sbt depending on which build system you use and you're set to go and then you can you you can actually run Cuda with gpus or if there's an if there's Matrix Library n plus1 that comes out that will also work because all I need to do is create a backend for it without having to rewrite my algorithms so you basically have core with a set of algorithms you scale it with iterative reduce this will work on any runtime mesos spark yarn any of these that you can think of it's all Google brain style parameter averaging thank [Applause] you questions mention that there's like problems that you want to do like like in spar with Scala that you haven't yet tackled but this looks fine what's the problem so oh so you sorry rephrase your question so what do you mean you mentioned earlier that like you you're going off on like doing this more installa and Spark and that there's fors of like ml in so I guess I guess what I'm saying is so spark spark spark itself is amazing what I'm saying is is I prefer the configuration based approach that I'm doing where I I don't have to say logistic aggression with lbfgs all I have to say is multi-layer configuration change one character and I have all the optimization algorithms available to me so I don't actually have to re keep recreating like uh softmax regression logistic regression with lbfgs or gradient descent or so I prefer my configuration style approach that's all I have a question regarding the performance right would you compare with other well so in this case like okay sorry um so he was asking in so he was asking performance uh with respect to other tools so benchmarks in theano and torch have mnist bar mnist on four cores uh takes about a data train that's just 60,000 images I take 10 minutes with the parameter averaging approach so I can scale out to any number of cores or I can train I can train on each core once and get the results back um so what I would like to do is I I want to actually release comprehensive benchmarks that actually demonstrate uh like how fast things are how it scales linearly among other things so I I need to actually I I need to actually like run like full-blown benchmarks that I can actually back um so those have yet to be released but that's that's actually what I'm working on next I'd like to be clear that this version of spark is a technology preview um you know there's there's a lot yet to be done and I think the API could be cleaner um and I'm welcoming input into the community uh based on like what we see here today sorry uh what what was the biggest model that that you able to train like how many terms so a neural nut on average can be over a million parameters with just a simple mnist um so obviously like you know depending on the box you could you could scale out to 100 million parameters or more um like I said benchmarks need to be done yet um I've optimized N4 where you can handle like you know floats as data types and a lot of other things I would actually like to leverage uh Sparks uh new spark new features as well as like their their distributed storage because one thing we're one thing actually we're going to implement is the parameter server uh done by Google so they actually scaled out to billions of billions of Connections in in the neural networks um so that's just that that's that's actually just a logical extension of iterative reduce actually so you do parameter sharding um so I'm so there's there's actually things on the works scale to billions of parameters but not everybody has billions of parameters so I'm just I'm making some assumptions right now how do I missed I don't know what you talk about it how did you do parameterization oh so in this case that's also configurable repeat the question so yeah okay so he asked how I do how I did weight initialization so that's another hyper parameter so you typically you typically you typically do it with respect to the input and output sizes that's one configuration um you can also specify your own distribution that it samples from as well so that's all in the neural net configuration question is because the air surface Network are periodic right you're going to need to have a a warm up step otherwise the individual batches they settle down different right eventually run yeah so I mean well so that's well that's that's where you have that's where you have tricks like Dropout constraining the gradient making sure that making sure that you don't overfit on any particular many batch actually has a warmup step to make sure that you're close to where you're going to be right in terms of the the actual so so I'm just referring to the fact that the paramet stage you're going to need to have you're going to want to have at some point a Serial St just before seral yeah so we're going to do uh so I want to implement sand blaster as well that will have that initialization step all the examples that you're showing are about uh image so you must be showing us image analysis do youy to analy some other problem that so we've done we've so yeah so fraud detection well so what kind of fraud so he's asking about fraud detection so or just other problems I've solved so while I'm while I'm demonstrating image that's the canonical example that most people can relate to um you know you can do you can do anomaly detection in this case I've done it with long short-term memory um so uh so if you're or you could do binary classification whether it's fraud or not fraud um so it really depends on the kind of fraud you're solving um just like any neural network framework you you set up the objective function and the classification error and all that to give yourself your binary classifier but it's relative to to the data that you're throwing in um you know I also you know so we also support word DEET glove nor W embeddings um time series analysis it's all in there um I just want to be clear that the main the main feature introspection tools right now for visualization are mainly for image and text um but you can you can actually Le monitor the parameters and histogram them to get an idea of like at least how fast it's training with other forms of data oh so yeah so the debugger oh sorry um so the debugger is are those things that I was showing earlier with the like the like so in this case the histograms the filters I can I can show those real quick so hold on I'm just going to run my one example here I think see so I need to open that all right so what I'll do is I'll actually show you the interactive uh debugger now so pardon me while I pardon me while I'll just run it just to make sure set everything up here and then I'll run through it okay so okay so so right now right now we're warming up and then what it's going to do is what the first thing is it's going to do is it's actually going to basically pipe some output to map plot lib that will show me the histograms here so um so the in this case in this case the debugger while we're setting up here what we're doing here see just um well so what we're waiting sorry oh um I don't know what's going on with today me try to run on a simp simp example all right one second am I not seeing it or the the dual screen is me before oh there it goes so sets it up all right okay cool all right so so basically the first thing you do is you actually you actually run it and in this case this is this is a restrictive boltson machine so you plot the activations so you see that I have the weights the hidden bias and the visible bias so the weights are the actual connection weights the hidden bi in this case the hidden bias is is what you'd think of for the forward prop and the visible bias is which attached to the data the key things here is you need to pay attention to magnitude so that's that number at the top there make sure that that's not too large and that things are uniformly distributed which you can which you can also see here so let me just and then you close it out and then and then see and then then what it's going to do next is actually bring out the images so these are actual faces oops so these These are the actual faces so what it'll do is it'll run the histograms and then it'll it'll actually it's really difficult to see if you want to see a close-up later I'd be more than glad to show you it's just what I was showing you earlier with those faces so you debug you basically debug them visually and then they this is the activations so basically what you do with a neural network is you see how sure it is so and this this case in this case you can see that it's relatively sure so it's if it's white that means the activations were closer to one if you if it's black that means it's unsure and that it's more it's uninitialized the typical the typical flow here is you want it to start dark so dark and gray make sure that it doesn't make sure that it doesn't overfit you let it run for a few iterations and then if it's basically if it stays uniformly distributed with the histograms and it doesn't and it stays it stays great throughout and slowly over time gets brighter you know that you have good you actually have a basically a good uh hyper parameter configuration so that's that's the G so that's the general flow with kind of like the debugger that's why I suggested debugging visually because otherwise neural networks tend to be black boxes does that makes sense code yes so this is yeah so all you had so I actually have built-in tools that do all of this for you the only thing you have to do is say I want at every n iterations uh plot and then I have I basically have hooks for every individual little debugger that You' want to build intell go ahead uh I have a couple of questions uh the first regarding the size of the models I saw that you save thank you uh I saw that you save all the parameters in a progress varable as well right mm uh do it limited the size of the model by the this currently right now it does um so again if you have if you have a billion parameters send me a PO request um so be really problem because basically we imagine that we have like 1,000 workers right and at the same time they're sending the gradi right gradient back yeah so and so it's so in our yarn version in our yarn version we save it to dis I said this is a technology preview because I wanted to get I actually wanted to get feedback from the community um I've been I've been talking to Sean over over claer and he G he kind of gave us an initial pass um but there's definitely still more work to do uh on yarn I'm aware of that and save it to disk uh that one's a lot more developed than the spark version right now okay I see um the next question is regarding the the configuration of the network you saw you show um So currently as far as I understand that if I want to change the configuration I have to change the code mhm is it so you so you can you can actually serialize this to Json and and then just change and then just change a property file instead oh I see yeah so very similar to Cafe I just okay it cool so you mentioned that part of deep learning is actually figuring out features from from neur data right uh could youor how that works and if you sure yeah so what you saw the output that so those faces were basically were basically the features so the automatic feature engineering is just learning how to reconstruct the data so if you have a feed foret you're using an auto encoder or a restrictive boltson machine you minimize you have two biases one attached to the input one attached to the output and basically what happens is you you compute the activations with respect to the output and with respect to the input and you minimize the Reconstruction cross entropy or or k diver con of the data and what you get is a shared weight Matrix that then represents the original input basically is able to reconstruct the inputs so this works for this works for recursive Nets this works for normal feedforward neural networks convolutional neural networks whatever you're trying to do just he ask you lab your data sorry your data or yeah well okay so like I said deep learning is not Magic it's it's just like normal neural n it's just normal like normal machine learning if you're doing supervised learning you have labeled data um so the key here is to use the key here is to use J like uh the rdds in spark to handle like labeled points or what have you for your data um if you the key one thing I want to mention though is that a lot of a lot of a lot of search based algorithms uh and clustering can be done with just the output of neural normal neural networks um so one technique that not a lot of people know about um is this idea of semantic hashing deep Auto encoders it's not used it's not used as much anymore but I know a lot of companies that are basically use it as a form of compression for storage uh this is actually what's done with IM this is actually done in image search as well as text to compress a bag of words um so you basically train a neural net with just back propagation on the on the original on reconstructing the original data so you have a feed 4 decoder that you train you do pre-train pre-train pre-rain just like the normal neural networks and then what you do is you create a you you create a transpose of that that's a decoder and then you minimize error with respect to all with with with respect to all the neural network like all the layers and what you get in the middle is basically a neural network that know like a coding layer that actually knows how to compress data so you can actually create a good approximation with reduced dimensions on your website you said that you starting uh running the code for LS specific what long shter memory yeah do I have it it's done it's in there okay I see I need find the link on the website oh so actually so we need we actually need we're actually going to update the website for the next release so um right now right now we just implemented the new API that has like the layer architecture Fano style uh module you know modules and layers and the parameters and all that as well uh so expect that here soon um so yeah so nd4j I can I can do I can give you like a quick rundown on that um you know the the major thing with nd4j is that you have a number of backends so so n4j itself let me just bring up GitHub here uh so nd4j itself is a very simple project it's very similar to what you see with um with numpy and mat lab so basically what I do is I have a series of components that give you a met lab like interface to any sort of scientific Computing library on the jvm so Juda is supported J blast is supported and we're going to be we're going to be rolling out support for Intel mkl as well as open CL here soon as well um so all those are backends without you having to rewrite code so every time I every time a new Matrix Library comes out if you you have one core you rewrite you write your Arin in nd4j and then you delegate to a backend so that's what that's what these modules represent here um I so I support netti Bloss uh this is the this is the beginnings of the scale API I have J Bloss in here and then a number number of K Bloss modules so the API is the key here so one thing what basically what I have is one test suite for all the backends that's represented here so okay so this will this will also show you example usage as well so basically there's one ND array so this is how you create an ND array so just like just like everything you think of you have linspace you have um ND arrays you can you you have you know np. create or what have you sorry yeah here so so all you do is you have an indd array and then all these all these backends Implement that interface and then you delegate to any any back end you choose and then the key here you know you know netl netl and some of these other things have multiple backends but don't give you ND arrays and one interface for uh one familiar interface for scientific Computing so if you want to work with 40 tensors you should be able to do that you know not a lot of libraries support this though so you know so here's like so here's like a basic example so as you can see I have like nd4j do1 nd4j do uh linspace like basically if if it's a numpy I've implemented it here in nd4j so it's a very similar initialization and again this is one test suite for every back end that you could think of um obviously i' I'd like help expanding this because God Only Knows God only knows what else you run into but like the core idea is there so you have reshape like it's exactly like numpy um so one of the things I did though is I I actually I actually I actually took some cues from J Bloss and actually um made some operations on matrices more explicit so subtracting row vectors with broadcasting that's something that's that's a pretty common operation and something that should just be explicit um I have rows and columns um so rather so you don't have to use the axis interface for everything unless you're actually dealing with any arrays um so there's also there's also sorting so just like just like just like Arc sort with numai you have that there as well all that I mean this this is it I mean it's I mean this this at a scale of interface with operator overloading could be you know would look exactly like numai save some save some differences but you know in general it's there I mean beyond that yeah this this is it I mean there's there's nothing much to it like um so like see swap axes uh you have slicing um I mean it's a matrix Library uh I have I mean I have um I also have like I also have indexing so ND array index gives you intervals gives you Boolean indexing all that's supported as well um I mean a lot of this needs to be documented a lot better I I I'll be honest with you I primarily created this for deporting 4J um you know I wanted I wanted an interface that was familiar to me that I used in scientific Computing so one of the things that's not very familiar to Java developers or Scala developers you know maybe some of you but you know functional programming but vectorized programming so actually like working with vectors making it easy to do transpose operations and thinking in terms of symbols so one of the things I one of the things that I'm looking to implement and um if if there's any mathematicians in the room uh there's I'm I'm actually going to be looking to support symbolic computation as well very similar to what Fano does with automatic uh derivation of gradients um other than that I mean I'm pretty agnostic as to what Matrix Library I use I'm going to use whatever is fastest I just don't want to have to rewrite my code every time something new comes out that's like the idea here um beyond that um I've also implemented some I've also implemented some of the operations that only exist in theano so Dimension Shuffle is one of those um this one this one this one was fun um beyond that Dimension Shuffle is like a more kind of academic but you can basically do a lot of crazy reshaping tricks so I mean as you can see it's it's it's literally it looks it looks more or less like numai you know you have lens space reshape you know you can get you can get rows get columns um you know I I support uh tensors which is all in here I mean this is this is in my opinion how uh scientific Computing should be done I mean you know it's nice and simple everything's a vector I can say sigmoid I can say um tan H whatever whatever whatever linear transformation you're looking to do it's all there um I don't know Alexi is there anything in particular you wanted me to show I mean that's that's really all it is you have a number of back ends and you use one interface for matrices sure so I can um here so one thing I can do real quick is go into okay so one of the Core Concepts um is the Bloss wrapper so the Bloss wrapper itself is basically the the Primitive you use to delegate to different back ends so Bloss is Bas Bloss stands for basic linear algebra subsystems that's actually the spec for fast Matrix operations on a computer so it's been around for a long time it's a standard set of algorithms and it's a standard implemented by a number of vendors including including Intel Nvidia among others so the Bloss wrapper is actually how you delegate low-level operations uh actually very again very similar to nump um I expose but I expose the raw Primitives this is so all you do to use a spe specific Bloss wrapper is include a particular jar file and then nd4j will do a look up at runtime to figure out which one it should be using and then when you call nd4j create it then we'll create the particular back end see El wrapper linear um scalers wrapping different kinds of vectors tensors I mean there cool so oh so the other thing is coming up is also the ability to inject your own convolution and fast foror transform implementations as well so if you're if you're in if you're in Native and you want to inject your own implementation of that you can do that as well the idea is to have one jvm based interface for handling all for handling all your scientific Computing and then in this case with spark all you would have to do to use it is run you know run your run your map and run your map operations or flat map ored or whatever kinds of other algorithms are there and then all you do is you have to make there's ant there's going to be an extra step you have to make sure to set up the particular back end on your cluster so if you're using spark ec2 you know it's not it's not going to be quite as uh it's not going to be quite as automatic unless you have a little bit of assistance so there's going to be a little there might be a little bit of manual setup so that ideally will there there will ideally be some utilities to handle that but uh for now just just be aware of that so Spark by itself comes with God I don't I don't even know how many machine learning libraries or Matrix libraries has Breeze in there there's some NET LIVE I know there's J Bloss so a lot of a lot of things I believe are supported already I've I've used J Bloss uh without a problem on a big cluster pretty easy use Sparky C2 submit your jar file via submit jar and you're done all right cool go ahead so with the with the current state of the code um if I go home tonight or tomorrow on my lunch break and I want to run a simple example where's the best place to go to follow uh a basic like mest or something like that example so deporting for.org has a number of examples on single core um we need to put out updated examples for spark and Spark Hadoop and some of these other things though um the API the API has been Rewritten since 3.2 and I need to get uh I need to get some better documentation out there right now which is why I'm saying it's a technology preview I have a lot of early adopters on our mailing list that that have been helping that helping me along though we're getting there yes yeah so all the all the codes there if you want basic example usage the tests are going to be the most up toate thing I mean now that the um I'll say this now that the now that the now that these now that the API is stable and modular and you can plug in different components and it's composable I'm pretty I'm pretty comfortable with where it's at and intend on actually like basically writing psyit learn document style documentation for this what's the difference torch sorry what's the difference with torch so well okay torch is written in Lua right you know it does I said torch is written in Lua so it cannot be in the same process as your spark cluster without a lot of without a lot of custom work whereas this whereas this integr where whereas this integrates with built-in tools I have I support nor w i support different uh neur W embedding implementations and everything you see is distributed so everything will integrate natively with spark in the same process and you can deploy you can deploy it with just a jar file so the the whole point is to work with a familiar language in a familiar environment versus versus torch I mean torch has torch primarily works on gpus and Cuda um it's it's a bunch of SE blobs which no one likes to playing um you know you know I I mean well you know so it's it's hard it's it's harder to manage I mean a lot of a lot of a lot of a lot of systems are jvm based the day-to-day data scientist doesn't know Lua the day-to-day data like you know I want to you know I want to I want to support a lot of these different ecosystems but you know it's like I'm I'm primarily focused on practitioners here so you said this one is already distrib already yes yes how do so that's parameter averaging so we use that's called parameter averaging we use something called iterative reduce for scaling out to multiple cores and then we are also going to implement uh sandbuster lbfgs as well so those are both scaleout algorithms that involve parameter averaging where you have a number of workers you replicate the model to end cores and then you basically send the results back up to one master node that controls the aggregation of the results just a quick question you compare this library and with other libraries like Java or so those are those are all backends for me and I have I actually support ND arrays so you know B Breeze has three and 40 tensors but doesn't allow you to scale out to like six and 10 um and I give you I give you well that that's actually a realistic workload like if you're working especially if you're working with convolutional neural networks where you're analyzing video that's four and five Dimensions um so I give you a transparent interface that looks exactly like numpy and people who use vectorized libraries could then use and then I delegate to a number of back ends rather than so if you if you there's one takeaway it's a facade it's a facade for any number of libraries that you would want to use uh with ND arrays and then there's also I also want to add that there's supplementary algorithms in here it's not just a Bloss implementation you can plug in your own ffts you can plug in your own convolutions or any other native based operations done okay all right thank you [Applause]