Tensorflow and Swift

Scale By The Bay 2018: Brett Koonce, Tensorflow and Swift

Recording: Scale By The Bay 2018: Brett Koonce, Tensorflow and Swift

thank you yeah the title of my speech actually got changed slightly since the slides went out so now we're going to talk about tensor flow and Swift today I put my slides up on the internet if you want to download them and run through but don't spoil my ending so to speak so here we go earliest year I went to the 10th to flow dev summit where Google announced tensor flow for Swift and intensive flow Jas a couple weeks later Chris weakly invited me to come to his spark and tensorflow meet up and talk about tensor flow yes I was gonna make some demos and stuff like that but my speech sort of mutated in the why you would even want to use JavaScript to do something serious like machine learning and so I tried to make sort of a persuasive argument for why you want to do things that way there was this guy in the front row who well basically I made a joke about how the functional programmers needed to get rid of their silly programming languages and jump on the JavaScript bandwagon this guy in the front row got really worked up he's like well you really are you really gonna do that do you really really believe in all this and I was like wealth to be absolutely honest if he was me I'd be using Swift and so anyway he said well then you have to come to my conference and talk about that so that guy was Alexi and Here I am very broadly this is how I think about a lot of this stuff at your most basic level you have sort of math and algorithms doing all this machine learning stuff the next level up then I would say is like basic virtual machines Jupiter notebooks stuff like that the next level up there I think it's cloud software which is usually UNIX in some form another where you're sort of like coordinating multiple machines multiple GPUs to sort of produce results the next level up then I think is the edge this can be thought of just sort of taking cloud logic and you know running it locally in some form or another the reason I think the edge is above the cloud even it's oftentimes with the cloud you can sort of cheap against your problems so you can say give me a server with a terabyte of RAM and WHAM your problem solved or is if you're doing stuff on mobile devices you often have very hard memory constraints very hard processing your strains and so I think it's another level of difficulty what this talk is gonna be about it's largely about custom hardware we're gonna look at the Google's tensor processing unit the Bolton Architecture from the Nvidia there's a whole bunch of machine learning startups and stuff out there right now so people are bringing custom Asics and stuff to market in next year - this is sort of an area that's very hot right now people sometimes ask me how to get started machine learning this is roughly how I think you should approach it if you were to start today at the very least I think you need to know Python I know this is a scholar conference but I think Python is basically defacto a language of machine learning right now I put up real add up here I think you need to have like a basic understanding statistics but if I tell you to stay statistics you'll often read a book about like probability distributions and stuff like that and I don't think that's very fairly practical so to me if you go to Vegas and put a chip on the roulette table and you actually know what the odds are I think that's like really good real-world understanding of probability and statistics the same thing I think you need to understand like the basics of calculus linear algebra I don't think you need like the full-blown calc 3 college level course or whatever I think you need to be thoroughly familiar with derivatives integrals and limits and stuff like that same thing I don't think you really need all the advancement this stuff but you really really need to understand how to multiply matrices the current course I think is the best way to go it's the fast a 2018 sequence this is all of Pi torch I've been doing PI torch a lot this last year I'm a big fan of PI torch I think this is you know if you do that course forward and backwards you can basically go out and you know go anywhere from there the next level up Tim did you're gonna have to start reading papers or other people's code we're going to go through a bunch of papers here today and then finally you need to start practicing kaggle is good for this what I tell people to do is take on a cattle challenge from a year ago cuz then you can go out and github and find other peoples code and see how they tried to solve the problems and then finally I think you need to get out into the real world what's exciting to me about this whole field right now is basically every person this room has some problem that you understand better than everybody else and so you can take that problem add a little bit of machine learning on top and basically you can become the you know the world's domain expert in that little niche and so I you know a lot of people make big deals about em phd's and all this other stuff and I think that's play the wrong mentality you know I think everybody should be able to do this stuff okay we're gonna do a little bit of JavaScript here at the start just to get warmed up this is a comic strip sort of make in front of JavaScript for its we'll say a DD personality the basic gist of my presentation I'll say I did a couple months ago the sort of that JavaScript has this whole creative destruction element you know it's always changing always moving and I think that's ultimately why it should be on your radar so here's five demos with JavaScript that I was just run through real fast here just illustrating some sort of different concepts from the machine learning world I like this demo whenever I show people CNN's I always start with the in this dataset I think that's better you know the best way to get going so this is just running here in the browser it's running on the in this thing you get your training curve your accuracy curve here at the end it's going to show a whole set of data in those predictions and you can sort of see like you know what the machine is thinking so like it you know like it got this three right here and it said it was predict it was a seven but it's kind of not really a well drawn three so you can kind of give the categorize err a little bit of you know leeway here this is a demo that's just running the yodel object detection Network on the browser against my webcam right here so you can see it trying to pick me up just kind of a cool real-world example of this stuff what oh yeah as well it's yeah it doesn't have anything trained perfectly this is magenta this is a project out of Google I think it's kind of cool because it's you know a lot of machine learning stuff is oriented towards you know extracting another few percentage points where the performance I have things whereas like this whole domain of say art and computers is really interesting to me so it's like making a sort of some quasi ambient idea music here or something if you listen [Music] this is a cool demo it's sort of visualizing how the gang training process works how the distributions how it learned them this is something I don't really have a good conceptual model of how it actually works so I thought this was kind of just a cool way of looking at the whole thing it has this button here that lets you restart the whole thing I thought this was cool cuz you can kind of see it introduces cluster of pixels you know and then slowly start stretching them out to sort of map into the latent probability space so just cool stuff and then this is a demo I saw a couple weeks ago I'm not going to run it for you but they use transfer learning to sort of learner to guess truce so basically you can sort of shadow kick and fight your new controls of Mortal Kombat character so you can sort of shadow kick and quiet refer your friends just using a webcam okay so a couple weeks ago I did another talk on tensorflow HJs I attempted to put together this demo I didn't quite get it working so I got a working last week so I thought I would show it off to you all I'm not gonna break down how exactly it all works if you want to look at the code you can jump to the steps but very broadly we're using mobile Nets which is a computer vision network using tensorflow a chance to run it we combine all this code together into a docker / node container and then we use IBM and open wisk to upload it to the cloud where we can use as a function and then finally I'm just running it locally by using a kernel command and then we base64 encode the image and into a post so here's the actual command so to speak just base64 is this picture then we have this sort of key and then we have an API in point everything runs against so we ran it on this picture this bird that I have here locally on my computer and it thinks that it's a indigo bunting or whatever with some 85 percent probability we'll say we'll do it again but with a panda picture and this one that recognized this pretty well and things that we definitely have a parent a 99% probability so you're sort of demonstrating a you know a different way of doing a machine learning problem by using a little bit of cloud magic we'll say okay now we're going to get into the sort of immediate presentation we'll do some Swift this is a picture from a PI torch Developers Conference relation here got to meet Chris Lander very proudly look at what a tensor is what do we mean by a flow what do we get whenever we combine these two concepts together we'll review the current state of the art for image image recognition training and scale then we'll look at how I think Google is positioning LLVM and Swift to sort of take things to the next level and at the end I'm gonna try to give you a glimpse of where I think things will be at in the future so what is a tensor I think a lot of people are comfortable with matrix math a lot of people are comfortable with algebra but I think whenever we combine these two concepts together it starts to get a little bit fuzzy even there's picture here which I found on the internet you know which just sorta tensor as a collection of matrix objects even this you know they're not putting actual letters into here you know ABCD or whatever and thinking about in terms of like properties of it as a whole you might take like a linear equation and form of ax plus B then you can play games with it you know you can make linear transforms you can go for one transform to another if you sort of start to build this up you get this whole set of rules for combining rules and eventually you have sort of a formal algebra Ford a matrix math this in itself is a lot of work but once you actually have an algebra then we can extend it with the you know to say the algebra or graphs or the algebra over type theory the big story I think of this last year so the machine learning has been like pi torch and this concept of these network flows fundamentally the core concept is just that everything is fundamentally a directed graph which means that if we can model the graph as a whole and we only want to change one variable or whatever then we can recompute just that one little thing that got changed as opposed to adding a rebuild our graph the whole time Google is wholeheartedly embraced this concept eager execution is part tensorflow now for their tensorflow to world map I mean this is that's they've clearly indicated that's where they're heading as well so I think this is one of these key concepts that you need to know if you're going to be working on this going forward this is a paper from Ian grades from the year two ago he built up a formal architecture for modeling neural networks what I think is powerful about this is basically you can think of all the neural networks instructions as being a combination of these sort of blocks will say you know you have to input data you run through some sort of convolution and ax plus B sort of steps and then you get your result and you can think of like a full-blown neural network or whatever as being a large collection of these sort of bots you know you might literally take a thousand these blocks and put them all together the reason I think it's important to think about this is because if you really look at the original tensorflow paper it's not even really about GPU training per se as this is this gigantic sort of Gantt scheduler for performing these operations whenever you have a whole set of different things going on it's it's much less about the GPU per se as is about coordinating across all these CPUs and then giving the end result out at the time and doing so using the least amount of you know a long time so to speak the second big story then the last decade or so I would say is that hardware there's been this relentless battle in GPU space to improve things and this is a cool graph because it's basically got every single GPU for the last decade or so up here what's wild to me is basically everything below the red line is now obsolete netting includes like some Intel things and other stuff like that I marked to these devices with stars up here this yellow star up here is the V 100 the volta which we'll talk more about here in a second and then this blue star over here is Nvidia snooty for GPU which we'll come to at the end so just remember those ones for not a random question does anybody know where Google's TPU would fit on this graph it's a trick question because the TPU hooks are hidden no no actually the TPU doesn't do floating-point 32 so actually it's a zero in the TPU doesn't even register on here the reason for this is that while I think most people conceptually can follow that hardware's getting faster and faster the second piece of the puzzle for the last few years has been using smaller and smaller operations if you go from say a 32-point floating pit the 32-bit floating-point object down to a 16-bit one and you can still do the same math then that means you can do two twice as many operations in the same time then the second trick then is they've introduced these various hardware devices so that this whole steps to be done together even faster and faster so we have the traditional floating-point 72 here we have the floating-point 16 the ant 8 which is what the original TPU is based around which gives you a 16 X speed up just by a jumping you know reducing the precision your mouth when you finally got the m4 portion of the the new turn chips pay attention to FP 16 because that is one of the important tricks that the volta does so next we'll jump over to the summit supercomputer this is that overreach former site of the Manhattan Project what's up on time but nowadays their games you know they do a lot of high-performance computing the summit supercomputer is so yeah this is a paper that came out last month where they're trying to model large climate systems using this computer so to me the coolpix part of this graph is simply an X axis that's 0 up to approximately 28,000 GPUs being thrown at these problems and then to me what's very cool is basically all this work that's got an intensive flow there's basically linear scaling you know the more GPUs you can throw at your problem the faster you can get a solution the architecture of the the summit architecture is using a custom idea of architecture there's been a lot of work that's gone on under the hood very specifically within deep link in order to deliver unified memory architectures the actual computer is not made up of Intel dgx ones will say but you can think of this cluster is being roughly equal to three and a half thousand dgx ones if you're on Amazon and using a p3 - 16x large that's a DG x1 so to give you sort of a plain reference so most of us probably cannot take our problems and you know throw 20,000 GPUs Adam but all of us can take advantage of the difference between the red and blue lines here the blue line is demonstrating FP 32 training and the red line is demonstrating at a P 16 train so if you can take your model now these are two semantics that segmentation there works but they modified the math to run on the lower precision operands and so you can see like this one maxed at around 200 kind of flops but by reducing precision in half they were able to get to 500 petaflop the same thing for this deep lab over here the max down around 300 petaflop using the FP 32 whereas whenever they went to an FP 16 they were able to get up to somewhere in the range of 900 pen flops so of honing in on that exaflop territory [Music] so what do you need to do in order to convert your models FP 32 FP 16 so earlier this year I took the fasting a course in personal downtown and we worked on a submission for this fast sorry there's dog men's competition out of Stanford our entry came in second in the imagenet Porsche and I can basically give you our high-powered secret recipe for getting those results number one algorithms if you spend any time on anything spend your time on algorithms we had quantized hardware we used both the boxes pizza these are sorry yeah a piece sorry a DG x1 and then we had quantized software pipe art steam upstream had done a lot of work in order to get Bolton working on these devices we had to do a bunch of coding in order to convert the fast day I codebase to support the path precision math and then finally we did a little bit of distributive training on top by using a GPUs to solve each problem this picture is a little blurry but it's this guy at Google named Jeff Dean who was talking about how by implementing the algorithms that were introduced in the competition they were able to reduce the cost of their training in half so that's kind of cool so for us in order to implement all this we basically had to do a lot of elbow grease we'll say we did a lot of custom coding and whatnot you all probably do not want to do that and more broadly you know that shouldn't be like how you have to do what you have to do in order to get good performance at your models so about five six years ago a company named Apple had a similar problem and so they were sort of locked to their GPU providers for their mobile devices which is the iPhones so they wanted to change GPU providers but they didn't want to lose any performance for their existing code base so they hired this guy named Chris Latner and he came in and he introduced LLVM which the project he worked on at the University of Illinois to Apple so by doing this basically they went from having to write a custom GPU code for every model that they needed to implement to just sort of writing you know writing a way to get coded Bellavia and another way for LLVM to get the code out for a specific GPU architecture so then adding new jeep used to their pipeline was simply a question of writing a new transpiler for that particular GPU so as a direct result of this they were able to switch their GPU manufacturers but a long time over time I mean these key concepts start to make this make their way into the larger iOS ecosystem so the initial phase they sort of introduced clang and LLVM into the objective-c code base and they started introducing memory and thread safety checks where the compiler could actually catch stuff long before it actually got pushed out in the fields and what cost rash was there they were sort of retrofitting these changes on top of C and they basically push that up as far as they could so the next step was to introduce Swift and bring full-blown functional programming into the iOS world something a lot of people don't know nowadays is that you don't even actually submit iOS code anymore what you actually submit whenever you're using Xcode today is byte codes so it's sort of a transpiled JIT sort of thing up to Apple and then from there Apple has servers that actually recompile your code for each individual device in the field to generate the optimized runtime so you push it up they push down at a different model for every single say iPad or iPhone or so on and so forth what's wild about this approach is that if Apple comes up with say a 10% improvement for coding optimization tomorrow basically they can add that to their server fleet and then recompile all the iOS code in a while if you think about this concept then and you look at say the tens of comprehensions paper which came out of Facebook earlier this year I think you can see that the GPU people are moving in the same direction this top is just illustrating the basic pipeline that they have but they take GPU jobs run it through CPU or cluster of some form they profile the results say is this better or worse than the previous one if it's better than it to their tuning database and they repeat the process over and over again but I think they're always a good spot for humans in the loop for this search strategy we'll say but the other biggest learning I think in the last year is to has been this introduction of evolutionary strategies and stuff like that in order to have computers improving the programs that the computers are running the second part down here at the bottom this is a picture of how the tens of comprehensions actually takes input graphs and then generates optimized I think if you look at this and basically replace this green polyhedral transformations we'll say with tensorflow and Swift that's basically what Google is thinking about introducing to the whole machine learning ecosystem in the same thing even if you could say today generate a perfectly optimized model for your computer vision problem will say or whatever in the real world you're probably gonna be running your job on a cluster with other machines and other jobs you know have different devices and so it's very easy for me to think of say I take my job run at 80 percent of its normal speed you take your job run at 60 percent of its normal speed but the computer can find a way to combine these two jobs together to get to units work and say 1.5 units of time this is a graph from the TPU paper not a Google couple years ago what's wild to me about this graph is this number over here in the bottom right corner which is five percent so cnn's are pretty much what everybody thinks of whenever they're thinking about neural networks to Google that's like you know that's not even less I and then half of their CNN usage is actually alphago look at this paper so very probably like 95 percent of what Google does isn't even CNN's which is what most people think about what ever think about neural networks there so then what what what new capabilities will the tens of flow and live and Swift combination unlock so much people are shipping a new hardware to market in order to tackle these different problems in eight like I mentioned before part of the TPU the new r-tx cards have to have hardware for this be float16 is a lesser than data type but that's actually the native data type of the TPU twos and threes Intel and just trying to bring the hardware to market and they're going after this data type as well it for is part of Turin we'll talk more about Turing here in a second and then there's been some academic papers and stuff that's been done all this stuff so ternary networks negative 0 and positive we're a big thing a couple of years ago this summer there was a paper called sign SGD which made heavy use of these sort of terminator operations in order to reduce the communication between nodes in order to speed up distributed training so that's worth looking at people have gone all the way down to one bit neural networks so binary neural networks bike that was a paper earlier this year where they built up a language model using one bit neural networks it's a very interesting approach this graph then is sort of demonstrating how a traditional see a tip you might think about that and then you can simply think of any binary neural network as replacing all the reels in the graph with integers so if you have a one bit binary network that's a B and n will say if you have a two bit binary Network they're no longer be n ends they referred to as quantized neural networks so this top one is graph demonstrating how this is the graph that sort of gave me the idea for the dip on I drew here but this paper by Bertrand in the Netherlands basically they're able to target say arbitrary model complexity or model size in order to reef you know if you have a very specific constraint basically you can pick that and then you have to you know you can work backwards to how much better you're willing to accept it to your problem the second graph down here so a lot of people are doing quantities in their own networks but they're basically taking say like a FB 32 model and simply converting the individual operands down to integer math so they're not technically training a quantity neural network so to speak so this paper is very interesting because they introduced some tricks and stuff in order to have stable loss and training validation curves for both resonant and seafaring so the demo I'm going to do is basically a demonstration of different queuing ResNet 18 variants on c5 so we have our traditional floating point three to one here on the far left from there we work our way down from 8-bit seven bits six bits I've bit four but three bit too big one bit models as you can see in each step by reducing the amount of data we have available to the network the accuracy of our network goes down slightly so here our controller whatever we'll say has like roughly a 98% accuracy we go to 8-bit the next one we like go down to 97% accuracy and in each from each step we lose a little bit more quality the flip side though is that say if you go from 32 to 8 we have a significantly smaller model you know lose a little bit of quality but we have like literally 1/4 for this large of a model and we can use this integer 8 math run even faster at run time which is very important for doing stuff on edge devices I didn't write the code for this this gentleman named Elon Hoffman did this demo so I simply modified his code to output these different results and so we're going to demo is a 4-bit quantized ResNet 18 running on the CFR data set here so here's my code we all right I'm VP and into a Google Cloud box right here so this is fairly boring I would agree but it's just the basic training room for doing a image recognition okay okay okay okay yeah so what's cool then if we run our MB DNS why I'm running this on a Tesla t4 in the cloud Google announced this about a month or two ago I got an email that I was accepted into the Alpha program yesterday so this morning I gone was working so this may be very well the first tee for Google cloud demo in the world so fun of that so yeah shout out to them there for helping me out and set if you let it run and keep on running for a while it'll end up with your network work around 80% accuracy which really isn't that great for CFR but just remember that and we'll come back to here in a second so recap we reviewed the current state of the art for hardware and software for doing image recognition and training we looked at how people are moving from floating point 32 to floating point 16 and now people are starting to explore in eight models in order to speed up things even more I talked about how I think Google is thinking about integrating LLVM and Swift in this whole thing in order to make this process simpler in the future and then finally we demoed a four bit qnn resident 18 network training and then running on actual 4 bit hardware ok so at the beginning I promised to try and give you a glimpse in the future so here's my attempt to do so I've shown you the hardware and software working together all that exists here today so let's imagine that we have a cluster of these T 4 GPUs for real world problems are probably implemented about the 27:27 r-tx but they have about 260 T ops for this import process so let's imagine that we had a cluster of 256 dgx threes what does the DG x3 well I just emitted but a DJ x1 like I said over there there's a P 3.16 next large d GX twos are brand new but they're simply a 16 GPU version of this thing so imagine that we have 256 of these dgx tubes we rip out the GPUs and replace them with these t4 chips Amazon charges you about $25 an hour for a p3 today so let's say that we roughly speaking this would cost you about $100 an hour I think this is probably like a reasonable it's probably on the high side but it's a reasonable number to work if so I would mean that for one hour it would cost you about $25,000 to run this cluster the the the hint result though is that you will have about somewhere in the order of one extra op of computational power to the director problem so if we think about a DG x1 today is delivering roughly one petaflop then if in say in two years we'll say roughly all this stuff could be reasonably considered to be working all together so conceptually in two years for the price of we'll say a decent used car you'll be able to afford as much computational power as currently the most powerful supercomputer in the world if you're willing to modify your algorithms to use this F for math and if you can find a way to you know start working on getting that working now so anyway when I started programming a long time ago basic computer very small amount of RAM so my I'm glad that joke but I started out with an 8-bit computer and now after all these years I've moved on to a 4-bit one so thank you all for coming I put my put a bunch of links or different things up here if you want to jump these papers and whatnot so it's all on there any questions in return