Scale By The Bay 2019: Brett Koonce, Machine Learning and Mobile

Scale By The Bay 2019: Brett Koonce, Machine Learning and Mobile

Recording: Scale By The Bay 2019: Brett Koonce, Machine Learning and Mobile

[Music] hello I'd like to thank you all for coming thank Alexi for inviting me here today we're going to talk about machine learning and mobile at high level we'll look at this problem through the specific problem of image classification or recognition on mobile and edge devices we'll review the current state of the art for this field then we'll sort of zoom out look at where things are going and then I'll sort of do a demo to try and tie these concepts together and at the end we'll do a quick recap what is the edge I think phones are a really good example of this something everybody has in their pocket more or less but I think of it more broadly than that any really any any moving computing platform will say so I like autonomous cars this kind of an example but really any sort of sensor out in the real world is sort of what I think to be part of the edge we might even think of like a satellite up in space is being able you know some remote server and we can ping so fundamentally then I think whenever we deal with edge devices we have the following assumptions fundamentally our compute power is significantly less than the cloud fundamentally our bandwidth is limited at best and usually unreliable we might only be able to talk to a server once a day and it might be on a random schedule such as when a person pulls the phone out of their pocket in general we have to be very efficient with our power we can't run things at 100% and generally speaking we need to make decisions quickly we have some sort of concept of bounded or interactive decision time so to use the example of a self-driving car your cars at an intersection it has to pick a direction to go it can't consult an Oracle so given these limitations why do we do stuff on edge devices people talk about like security or privacy or things like that but I think they're kind of like walking past what they really should be saying so we take ResNet or imagenet or any other classical computer vision networks their input is a small swatch that's 224 by 224 pixels last year Google published a paper called G pipe where by using sort of a cloud supercomputer they were able to get this input size up to almost 500 by 500 pixels wide meanwhile the latest generation of iPhones generates 4k video at 60 frames a second so to me then why do we do stuff on the edge the answer is very simple that's where the data is people talk in machine learning about how data is everything you got to have the data so in theory working on edge devices we have access to an order of magnitude more data than the cloud people do and by extension we should be able to do things that they can only dream about so first we'll go through some of the existing solutions in the field quorum mouth this came out of Apple a couple years ago they've worked very hard on this iOS 13 came out last month and a quorum l3 is a very solid update to this whole platform they have a Python tool called core ml tools that lets you export models from XG boost scikit-learn and caris then Apple has this other set of tools and a lot of people have not seen called to recreate basically it's sort of taking the concepts of these above packages and throwing numpy into there and they're sort of rewriting them all into a package that runs on top of metal which is Apple's graphic programming language the basic limitation of core ml is very simple its iOS only the flip side though is that Apple has spent a lot of time optimizing it and so it runs really fast so if you're new to this whole field and you have a little bit of swift experience I think this is the best place to get started I did a couple years ago on how to build core ml models just using in distinct Aris is this basic example so if you're interested in seeing how that approach works you can look at that tensorflow light if cross-platform compatibility is your biggest issue then this is what you should look at they have libraries for iOS Android they have like a Raspberry Pi library and in theory you can get it working on pretty much anything you would like at a high level the process of building a tensor flight light model so we start off with some sort of tensor flow model in that world we have our graph definition and then we try to compare our graph into tensor flow light operations if there's a one-to-one correlation between your attentive flow model and tends to flow light then this process is easy if not then you're going to probably have to start making some trade-offs and start looking at the actual you know neural network operands themselves I was gonna demo I was gonna run one they tend to flow like demos here but the my Xcode is acting up so I'll save it for the end but basically we have the video feed off the phone and we have the mobile net computer vision Network from a couple years ago just running on the device last month pipe porch came out with version 1.3 where they've now finally added iOS and Android libraries that you can run yourself so at a high level you take your pipe or model you run a JIT trace on it it produces this dot PT model and then you can run this PT model on device using the local libraries I've not played with this a whole bunch but I got it working and I was really impressed with the speed and stuff so if you're over in the land of Pi torch I would definitely recommend that you do this so we're just lending a simple image classification Network on the phone the standard thousand image net categories but the speed is really impressive so 30 milliseconds and this is a pretty old phone this is less of the world of phones but embedded Linux should always be on your radar basically at a high level you can bring whatever hardware you want and then run whatever software you need this is nice for a lot of reasons the simplest is that like you can very you know if you have specific RAM or CPU requirements basically you can make the device speed up device to be fast enough to do whatever you need OpenCV if you're gonna give me one computer vision trick I would say just being able to type import OpenCV is a really good one there's basically everything under the Sun is in there people have made all these various libraries D Lib I've still seen people compiling MATLAB down to custom C++ stuff and basically you can run whatever shims or whatever you need to do on device in general I often use this as a prototyping platform I'll write a demo or something other and say four to five hundred lines of Python you know make sure I have a high level understanding the problem and then I'll start to think about how to actually get it to run on the phone a lot of times you sort of have two problems which is one is can we do this and then two is how do we make this fast and often times figuring out the can we do this part will allow you to either simplify your life significantly because you can avoid going down false paths and then finally more and more custom hardware is starting to come to market now people able to target say arbitrary integer and floating-point depths there's custom Asics and Max coming to the floor to the market we might even view Google's quantum computing announcement from last month as sort of seeing the dawn of probabilistic processors starting to become a reality a lot of people in space I think have a mentality that they can build the fastest chip or whatever that developers were flock to their platform and I don't think that's what's gonna happen I think fundamentally on day one whatever a new hardware you bring to market you need to have a way for people to get their existing workflows and patterns onto your device so to me while this hardware is cool it's just really exposing the new limit which is software so here's the picture here just sort of illustrating sort of the differences between say the CPU GPU and TPU style devices I thought this caption was hilarious because this is CPUs GPUs and CPUs have different on memory memory architectures and compute primitives this divergence must be addressed when generating optimized code like there it is that's all so now let's sort of look at the whole software piece here's most tensorflow is that today we have various high-level languages of wiffs switch switched is just one we generated graph of some form and we have these various will say routes to get your model out and actually run it on to some sort of device this is just sort of me speculating we'll say that if you look at the star right here it's my belief that google's long-term vision is to sort of get everything to go through this LLVM ecosystem so tea pews will have some sort of lvl ovm thing to generate code for them TF light will have some sorta hello view code to generate that we could maybe even think of WebGL as being a potential output of the LLVM runtime I think in a short term this is going to be a painful transition because they're gonna have to basically redo years worth of work to get here but I think long term this will make the whole tensorflow ecosystem extremely flexible and they'll be able to work with other new whatever new devices come to the market so take a look at this grappler thing up here because that's what we're gonna look at next so grappler is a tool to sort of speed up runtimes this is a couple of pictures just sort of illustrating how it works at a high level we have sort of these data conversion steps at a high level that are needed at the graph grappler basically then looks at the graph as a collective whole sort of does like a minimum spanning tree style approach and then ultimately it produces this simpler graph over here the practical upshot of this then is that things run much faster so what's the thing this sort of as being like a high level top-down style approach on the other end if you go down to the LLVM level you'll have sort of like you're trying to align your memory hits right a lot of this machine learning stuff isn't really so much compute bound as it is bandwidth bound so if the compiler can sort of make sure that the data is getting there on time it can speed up the results significantly so you might think of this as being a bottom-up style approach and so then we might try to meet in the middle somehow and we end up with like polyhedral compiler techniques I talked a little bit about this last year as sort of like a here's something for them to implement but I was kind of underselling how hard of a problem this is I think we could literally be spending a decade on this particular slide here and this is very much a hard unsolved problem there's a lot of people in this field looking at this stuff I talked a little bit about like Gantt charts last year so I liked this picture which came out of the globe compiler paper but effectively they're sort of taking the graph as a whole and then using it to optimize their runtime to just sort of figure out what order to schedule things so I think this is just a interesting high-level approach to continue that thread though we think of scheduling at a high level graph level but it's also very important to be able to schedule things on the device level this is from the mesh tensor flow paper which came out last fall basically they're exposing these sort of GPU primitives to the programmer so that the programmer can sort of manually manage where their codes actually running this is a powerful approach because it allows you to scale up your code to large clusters but the flipside is sort of you're making the programmer do the work that the compiler in theory should be able to do so to continue that thread so people are experimenting with using evolutionary algorithms in order to sort of find the optimal optimal layout or optimal layout or where the graphs are running on device so this is one way paper where they were using reinforcement learning to try and split up things across four GPUs the upshot of the paper was loosely that the human expert the domain expert realized that they could sort of put everything on the GPU three we'll say here and that actually ran better and so but to me this is sort of like a not a limitation of the reinforcement learning it's just that perhaps the reinforcement learning isn't medium in a fair shake here I could imagine the scenario in which we told the reinforcement algorithm that the only rule it had was that each compiler has to be doing something different and so it could figure out the same end result and so this whole area of compiler exploration or sort of using machine learning the speed-up machine learning is very interesting to me in general here's a couple of graphs from the TVM paper which came out last year so effectively they're running this thing like 500 times we'll say in order to start producing significant speed ups for a one-off problem this is probably overkill and you wouldn't want to do this but if you're doing large machine learning jobs and stuff and you have a very static problem then it would be well worth it to spend this sort of time and let the compiler figure out a way to get your code to run 50 percent faster for effectively free or the cost of testing running a bunch of tests this second graph is a roofline plot the basic idea is that this thick blue line is the theoretical maximum performance of this particular video card which was a Titan X I believe so the basic thing that this roofline plot is showing you is basically this ResNet 18 architecture is fitting neatly into both memory and compute onto this device is not going it's not hitting the roof so to speak the flip side of this though is that you can see all this sort of white space between our ResNet operands and the actual roof which is to say it's not running it optimally so here's an area where you can see where the evolutionary algorithms so are these like efficient net style approaches are able to more able to sort of impact the problem into these little holes and by extension get optimized runtimes the second player this stuff that is the future this roof is basically going to move right it's going to go up into the left and so run times that we have today the networks that we're using today you know we don't think of ResNet as being like a a what's the run times are running today are you just gonna change over time and so the assumptions are the popular networks today that assumption may not hold tomorrow so at a high level the we don't know what the future is going to look like is basically what I'm trying to say and then I don't really know what the future will look like but I think that if you can sort of have everything in one programming language whatever that programming language is you'll be able to adapt to it so to speak so here's a picture from the MLI our demos that are in the toy AST things but you might think of being able to do whatever domain-specific language you would like to have you get it in ml ir at which point we can start adding all sorts of optimizations on top and then finally we can output it to whatever device we need so to me this is where things are going so now we'll try to get out of the realm of theory and then there's something a little bit more practical so to speak last year I did a presentation on split for tensorflow and I did a theif our demo that notably had no Swift or tensorflow in it so my goal for this year was to sort of actually use this tool to sort of do this stuff on device so the demo we're going to do is we use Swift to build and train AC file network and it will actually run it on a device that is to say my phone up here however in order to do so we're gonna have to jump through a few hoops at a high level we'll train a C you'll use Swift or tensorflow to train a c far model from there we use Swift for tensor flows numpy bridge to sort of export our model to Kharis where we can then save it as a h5 file from there we can load our h5 file into a tensorflow session and freeze the graph and save it as a protobuf from there then we can convert our protobuf using ml ir into a TF flight file and then finally we can run the TF flight file on device so at a high level I'll demo how we get from swifty Kerris h5 to protobuf part of roughed em LIRR to TF light and then finally the modifications needed to actually run it on device here's my CFR model if you've not seen any Swift this is how the you would do a simple convolutional neural network in it this is just a basic sort of vgg esque style architecture with two layers of 3x3 convolutions a max bull and other two layers of 3x3 convolutions a max will do densely connected layers and then finally a categorization layer at the end there's a little bit of magic here we'll say basically at a high level swift for tensorflow doesn't support saving models right now so we're sort of manually adding it ourself Swifter tensorflow has a Python bridge so any Python trick is in theory available to us so our first line is pretty simple import Kerris the first top half of this basically we're constructing an identical Kerris model to the one we built in Swift and then in the second half we're just manually setting the weights that caressed model base out the results of our Swift run and then finally at the end the magic happens this little model dot save call from there we can load our h5 file into the tens of float Kerris importer and then just combine the variables and the graph together this is called freezing the graph and then we can say about the result as a protobuf so that the top is just sort of the high-level tensor flow code to do this at the bottom is the just basically a - B - C this is probably not a tricky of seen but ml IR is not very easy it's not theoretical right now you can actually build and download it today if you go to the tensorflow source repository and run basil on this first line this will give you the actual mo ir runtimes from there basically then we have this man line tool that's what the second line is here we have to do some bookkeeping for our network so we sort of manually specify some input and output layers and stuff like that and then finally at the end we export a TF light file oh sorry yeah I'm doing all this on top of the custom C far demo from the Swift models repository I just basically replaced their model file with my own and then I also removed the normalization step from the data so then finally what I have here is simply the dental flow light image classification demo from their source code examples and I've just made the following modifications we put our file in here I manually made my own little C for our labels file right here we did these I simply just point the model reading code to point to my file we changed the input and within the heights of our image input to be 32 to match our C for our model and then finally I added this ugly little line right here which will print out the results to the console of what the network is seeing so we have here a picture of a dog so we're now running our CFR model on device it's spitting out the results in the console here as you can see so now I'll hold it up to this dog picture and it should say dog so it's same dog with 70 to 80% accuracy okay so to recap our goal was to explore image recognition on edge devices I showed you the current high level approaches in the field I talked about how I think hardware and software is going to get closer together in the future and then finally I demonstrated using Swift and ml I are in order to build a TF light model which we then ran on device so that's about all I have for content I'd like to thank the ML IR team they've been working very hard on this all summer the Swift for tensorflow team has been making a steady progress for the last few months as well I've not demoed any fast AI tricks but there's a new version their library that's due to come out shortly and I'm looking forward to that I'd like to thank two people in particular Nia rosendorf this is gentleman over in South Africa he had a really nice blog post on how to do this numpy bridge with Swift for tensorflow and I really found that useful so I wanted to thank him and then a gentleman over in Taiwan named constant ton he's done a number of interesting technical presentations but he did one on ML IR and tensorflow earlier this summer which is what made me realize that doing things this way was even possible so thank you to him if you're interested more in this subject there were a number of interesting presentations at the LLVM Congress earlier Shir if you're interested in polyhedral compilation techniques I stole a number of slide today's slides from this presentation by albert cohen but you should look at some more of his papers and stuff he's been working this field for almost a decade or more than a decade and then finally LLVM is not the only game in town and so I think there's some interesting stuff going on in TV em and glow as well it's worth keeping an eye on so with that thank you for coming thank you so we have time for one or two questions anybody none all right thank you again Brett thanks thank you [Applause] [Music]