Adam Gibson of Skymind, Q&A with Alexy Khrabrov of SF Spark

Adam Gibson of Skymind, Q&A with Alexy Khrabrov of SF Spark

Recording: Adam Gibson of Skymind, Q&A with Alexy Khrabrov of SF Spark

hello everybody this is SF spark uh I'm Alexi krabo the co organizer of us of spark metup and we're at on location at galvaniz University and today we are speaking with Adam Gibson uh who is the creator of deep learning for J uh uh the open source toolkit for deep learning on jvm and also scientific Computing on jvm including Library such as and d4j which we're going to be talking about today uh and also a founder of Sky mind uh company which commercializes uh applications of deep learning and uh all these U interesting insights so let me ask you Adam first uh I watched your video I think the the the way I found um Sky mind I found a video where I was saying that a few years ago you did not actually practice machine learning much and you you uh ramped up by watching a lot of videos immersing yourself and now you are you know producer one of the most advanced packages so this is really amazing and you know this is what a lot of folks getting data science aspire to do how what was your motivation to get into the field and how did you manage in such a short time to basically learn so much how did you do this it's funny so in general I mean I dropped out of school I've always been selftaught uh so it mainly comes down to the fact that you know you just need motivation and grit it's seriously it was just a combination of me reading papers and implementing Concepts and repeatedly making mistakes until I learned what I needed to learn um so I have like an implementor perspective on a lot of things just like I just I like to tear things apart uh very similar to any engineer really and just figure out how does this work what does it do and decomposing it um and just kind of going from there um so what fascinates me in general about machine learning it's just this ability to have computers think on their own so machine perception has always been something I've been curious about because it's like well you know if I don't have to tell a computer with an if statement what to do what what are the possibilities and then you know you just kind of expand from there so why you bluring right there's a lot of different things what pick your interest in machine learning and specifically in De bluring well so in general I would say first of all the difficulty of it was alluring to me you know so they're diff you know neural nuts are difficult to tune they're slow to train so you know obviously distributed systems and you know in this case more complex Hardware such as gpus you know were a great engineering challenge for me I wanted to build something that not a lot of the industry would dare to tackle because it's not practical or it didn't it didn't make sense for their use case and you know I like I like jumping down rabbit holes and figuring out how things work uh and something as complex as deep learning allowed me to do that that's great you know a lot of people kind of you know uh avoid uh rabbit holes but and and and uh it's uh uh it's it's kind of it's a black box system right in the way um um deuring you know produces some some uh uh solution but it's hard to understand so that's it's a lot of work to to make them work in practice um so I wonder how do you deal with this fact that that that this is a very uh complex system which gives you a result in some way but but you don't know how this result is achieved so how do you bug uh something like that frankly histo so visually so the trick with neural Nets is that you don't actually just run them and hope for a result I mean not even grid search works with neural Nets what you do is you actually do things like histogram the weights and figure out you know is it changing too much is it overfitting is it changing too fast you can also do what I like to call monitoring machine dreaming so you can actually for vision you can actually plot the weights and see like actually it draw faces and reconstructing things so you can actually see what individual neurons tend to activate on which gives you the ability to figure out well how well is it rendering this and that that gives you an idea of how well it's converged also the ability also I would add the ability to plot the activations and seeing like you know relative to each iteration seeing like you know how dark it like basically the darker it is the more insure it is and then like you know you can see it converge when it's basically like all mostly white you know so it's a way it's a actually it's a way of actually just figuring out how fast is an on that and what is it doing another one is text you can actually with these neural word embeddings GL glove word DEC you know there's more than just word DEC out there more than just word DEC get you know get get familiar with that nor word embeddings themselves um basically allow you can actually group the words with something called TASC neighbor embedding so you can actually see a group 1 2 3 4 5 over here and then maybe Spanish words over here so there's actually you just need tools to actually visualize it that's great that's great uh and it's actually it's funny because you know now that I think about it it's kind of ironic because you're using your own neural net right visual processing to get an impression of how the artificial neural net is performing right and that's probably kind of I mean of course we're using it for all kinds of cognition but that actually seems a like a visual way is a very fitting way to evaluate this uh so for J is kind of you know it's I think it's a play on the classic you know sell for J and log for J right so this is used in kind of java Community to signify some kind of Hardcore Java focus and and uh it's a set of conventions basically for for gvm so so you made this choice to run on gvm compared to a lot of data scientists who run on you know Lua and Python and Ruby and R and whatnot and and uh I wonder like how did you make this choice in the first place and how do you see this uh comparison with other folks evolving like why gvm why do you stay on gvm and what does gvm have going for it that you know we should as data scientists kind of uh work more with jvm well well think about it what are production data Stacks written in they're written on Hadoop they're written on spark they're written in Scala the modern data pipelines are on the jvm why not be a first class citizen right where the data lives in the same process there's no reason we can't do that a lot of problems though with machine learning up till now on the jvm has just been the fact that there hasn't been a library that gives data scientists familiar syntax so you know I drew a lot of comparisons between me teaching uh python data scientists as well as screening other experts on Java you know like I drew comparisons and figured out what the jvm was missing and one of those things was yet another Matrix Library so you know we we see the xkcd you know standard plus one but at the end of the day what I what I the the approach I took was why not just take you know every every Matrix library in existence and put it all under one interface just like slf4j so why not solve it once and for all and give them give data scientists an an updatable Library that's future proof and give them familiar syntax so I wrote that I used N I used nd4j my library to write deing 4J and so I future proofed the I future proofed it and also made it agnostic to a particular runtime as well so I can be where the data lives but also be you know get access to more you know faster libraries like Cuda and some these other things yeah Cuda is actually another amazing example because you work with gpus and uh your software allows actually to choose uh GPU implementation if it's available right so I think that really Bridges the gap which a lot of uh things like R and python have going you know with sea linkage right we can actually have an access to to gbus as well from from the jav world and I think that really makes I think it's distributed both in in in the multi-course set up and also across the cluster which kind of brings me to spark I don't know if you know everybody is familiar with spark but maybe you can kind of uh reiterate the points why uh deploying forj and Spark makes such a killer combination well so one thing that not most of the industry is doing is this idea of Jeff style parameter averaging so the ability to take a neural net train it on several cores or workers and then average of results and the this has been proven results that Google uses in most of their machine learning infrastructure so just like Hadoop you know modeled map reduce why not model you know with iterative reduce our you know the kind of our take on parameter averaging you know why can't we scale that out with spark it's a perfect you know it's a perfect model where you just you you you scale out with spark and you just average the results and you can use you can use spark storage and rdds and all that to basically build an infinitely scalable system yes yes yeah I think that's you know and being in memory that actually gives us a speed up uh right which you know was missing in the regional K architecture so that's I think that's the timing is really perfect and given the spark uptake right with the spark metap all this 300 people are coming tonight to to to learn about this right I think that's a great opportunity uh to advance scientific Computing on jvm so you know being the founder of skymind I got to ask you right so you see these Technologies you also are connected to the startup Community where you know do where are you taking this like what is your vision for skymind how you going to kind of uh uh take these Technologies and Implement them so we're an already profitable startup that mainly works with Fortune 500 companies so the most interesting problems are where there's a lot of data um so we actually typically work with more you know with you know modern Enterprises who have Hado they have some stack and they just need they have a certain problem they need solved and so what we do is we facilitate problem solving right now we're not primarily working with startups I mean you know mainly because like rest apis and all that there's there's plenty of that stuff out there for normal developers we're targeting the Enterprise developer who doesn't have access like who doesn't have access outside the firewall and they need something that's a first class citizen on their cluster so we work more with Fortune 500 companies over uh startups in this case you know there's plenty of that out there there's plenty of exess JavaScript accessible things we bridging into Python and some other libraries you know deporting forj would be a great server as well but you know primarily uh Hadoop Hadoop oriented uh companies great and so uh in terms of domain space right so if you're one of 4 500 companies what kind of problems what kind of data makes more sense to use with deploying for G well so just like we think just like we can see things we can perceive them right so deep learning is good at actually making sense of things that we normally we normally see and just learn from so you know humans are really good at pattern recognition with site reading things you know basically ingesting patterns you know deep learning is specifically unstructured data video image text you know things that things that we see that maybe a data scientist has a hard time trying to pick apart the features for so I help I mainly help fortun 500 companies make sense of unstructured data uh and usually they have a lot of it in you know the H file system and then they don't know what to do with it we kind of help them bridge that Gap yeah and the text I think is a great example right because vision is a classic uh application of deep learning but it turns out the text makes actually very great domain for de learning as well and that's that's a huge Focus uh in in the data mining Community now as well well that's great and uh uh I guess uh uh the question I would ask because this is an engineering Meetup and we have a lot of hype about deuring but here we have the folks who can actually do this who can take a piece of Open Source and understand this so if I am a fairly good engineer who is interested in deep learning and specifically I want to learn it by looking at De planning for J call base right what like is it is it something we I should just plunge into like uh how can I essentially get my hands dirty with the play FJ understand how it works uh do you have some projects I can just take and run with them so we have a we have a few example projects out there but I would still highly recommend playing with core and actually understand the architecture so there's actually two there's actually two major modules there's core which is the core algorithms where you can run them play with them I usually just run the tests um what I want to do is actually put out more curated examples for people to just play with right now I just have a lot of examples derived from the unit tests you know not highly tuned but you know it shows you example to hear that you have not everybody has unit test right you know so I mean you know we have a very well unit tested code base and I I recommend just diving into cores core and understanding the layout understanding where where the layers are the core API and then working from there scale out is just parameter averaging so it just averages whatever whatever you're going to do in core yes uh okay great and since the as I understand the project is currently mostly in Java you know and uh I also help uh organizers of schola communities so uh what is uh the challenge to scholar developers where can we come in and kind of uh help with dsls and kind of apis where would you need most help from the scull folks nd4j because you guys you know you guys want functional programming you want operator overloading you want you want a lot of the fancy features so I actually I've actually already put out an n4j scalea API module that people can just modify um so I've started I've actually started work on a wrapper interface for the Java so after we get that you'll have essentially scientific Computing and Scala with Scala s syntax and then from there I would I from there I would want more you know in the algorithm space I would want more functional oriented deep burning algorithms as well great well you know if you're a scull developer uh watching this and uh especially in skull I would say let's jump on this because this is a huge chance for us to advance scientific Computing on gvm and have a clean nice access from Scala and basically experiment with data and techniques much easier and much faster have all the convenience of python and our repple but backed by spark cluster maybe some of the notes running Cuda right so we have we can have the most performant uh platform in the world running the most advanced algorithm so you know let's get to it guys and uh we'll see you at the next SF spark and SF scull mtops thanks