SF Scala: Brad Heintz, PyTorch 1.4 Release Update
Recording: SF Scala: Brad Heintz, PyTorch 1.4 Release Update
[Music] my name is Brad Hinds I am a partner engineer at Facebook I work with the AI team specifically with PI torch and if partner engineer is an unfamiliar job title to you that's not unusual but basically my day job is making sure that developers get the most they can out of PI torch whether that's in you know research and development or production or wherever if you are kind of new to PI torch and wondering what it is it's an open source framework for your machine learning and AI projects that will take you from research to production it has a number of sort of pillars to it design wise it offers both eager and graph-based execution I was one of the first frameworks to have like a really mature eager computation mode where it would actually track your computation graph during your forward pass and then automatically compute derivatives for you during a backward pass but we can also you know skip the automated gradients and executing a graph mode as well for production inference PI torch neural networks they're written in Python so they can be full of dynamic they can have complex control structures and you know loops whose length you don't know until runtime and all that good stuff we have tools for distributed training we've had data parallel training for a while and with PI torch 1.4 we now have experimental support for model parallel training hardware accelerated inference so where everything's all a magic and PI torch doesn't really happen in Python it happens in Lib torch which is a compiled optimized library written in C++ and targeted for CPUs and video GPUs and there's also experimental support for google teepee use so if you want to mess with your PI torch notebooks and collab feel free to try that out and finally simplicity over complexity we try to make the happy path the short path there's kind of one thing that I keep hearing about pardon me probably torture in a while is that developers seem to get a kick out of using iteration times are faster and you don't have to learn like a separate domain specific language to build your computation graph and all that we try to make it as straightforward as possible to to implement your ideas and try them out it's a pipe torch is an open source project there are over 1,200 contributors from around the world from companies like Microsoft Toyota Research Airbnb Salesforce institutions like CMU MIT Caltech etc so this is not just a facebook project this is a worldwide open source project we started we've seen huge growth in usage over the past year as well as growth in the community in the PI torch form on PI torch or on reddit groups and Facebook groups slack channel etc the stat at the bottom of the screen there is one of the interesting ones to me of archive.org papers that mention a framework machine learning framework pi torch had like almost two and a half acre or two and a half x growth last year in citations which kinda I think speaks to the graphic grassroots enthusiasm before the framework so a few weeks ago we released our 1.4 update every three months or so we do another point release and January was one point four and so I'm gonna share with you some of the newer stuff that was in 1.4 I mean I look back to a couple of 1.3 features we've improved on as well I think the big things to highlight with this release are first off pie charts mobile so in 1.3 last October we introduced the PI torch mobile runtime which I'll talk a little bit more about it's basically like a you know a drop-in library you can put in your iOS and Android projects you can feed it a PI torch model and you can do inference performant lis on your mobile devices and the big improvement in 1.4 is selective compilation so now if you're very sensitive to binary size you can set it up where only the operations that your model needs get compiled into the library when you ship so that's kinda nice I mentioned distributed model parallel training so I think a lot of us are familiar with data parallel training or every one of you the nodes in your cluster gets a copy of your model and they each get different batches of data and you kind of collect their answers at the end model parallel training which is becoming more important with models like GPT - is where your model won't necessarily fit in memory memory on its own and so you have to kind of slice it layer wise and put different chunks of the model on different nodes so we are offering experimental support for this now which is enabled by the RPC facilities baked in a torch not distributed now by the way when you hear me use the word experimental that doesn't mean we're just messing around with it everything I talked about here we're gonna we're committed to in some form experimental really just means the API isn't frozen yet it's still kind of evolving so what else we have java language bindings so as part of our work on android for pi torch there is sort of an opportunistic side project someone took on to make Java bindings for pi torch it's inference only linux only right now but if you're a Java shop you might want to check that out learning rate schedulers now they used to have a race condition if you try to use two of them at once like if you try to use a step and an exponential decay in the same same topic of the same epoch only the only one of them would win but now you can kind of compose them in a natural way and is anyone here still on Python to anyone sorry you lie [Laughter] yeah poor run out for Python - this is the last release that is going to promise any kind of support for Python - now besides the PI torch core framework we also offer what we call the domain libraries the domain API is for computer vision for NLP and for audio work these are torch vision torch audio and torch text all of which also got their point numbers bumped in torch vision models support quantization now I'm gonna be talking about the quantization feature in a few slides ahead all models are now able to be exported in torch script which is sort of our graph computation language a graph mode language for production deployment we're talking about that soon as well and masks are CNN is now exported to onyx so if you're doing you know instant segmentation with masks are CNN and uni onyx to deploy you can do that torch audio also had some improvements more and more of the the high value operations and transforms from call D are being rewritten and optimized for torch audio there are new data sets in torch audio liver speech and common voice and actually all the datasets in torch audio have been revamped to use the data set parent class from PI torch which is great because now it really reduces the memory impact of loading up that audio data set and kind of regularizes the abstraction we use for data sets across the libraries so yeah new data sets and improve data sets in torch audio Sox is not mandatory dependency now and again more operations within torch audio support torch script for performant deployment and torch text new bindings for tokenization for sentence pieces more new data sets and yeah blue scoring metric bilingual something understudy what's the e oh yeah but it's sort of like becoming a de facto metric for language to language or you know translation between human languages and that's kind of that's baked into torch text now for measuring the performance of your translation models so I'm gonna go a little deeper on a few features that crowds have told me our interest before starting with PI torch mobile so the general idea here is if you have a model that you would like to you know improve someone's experience in your mobile app you know the kind of common way in a lot of cases is to actually like take the input do a network round-trip to your model living on a server and get your answer back that way deploying a model to mobile unless you're using one of the the like OS specific frameworks is kind of a pain we've tried to make that as easy as possible here you would build and train your model as you normally would in Python on cloud hardware or whatever you'll export it to torch script again more about which later and then load it up in your native environment inside the PI torch runtime I actually ran this at a demo at a booth in an at scale in San Jose and people wouldn't believe that that the mobile net model I was running was actually working that fast like I'd would defy my shoe or a computer or a coffee cup and yeah all happening on device is kind of cool you know questions are the size of the binary where the the binary or the model itself actually I don't know the answer to either on top of my head it was mobile net B - so you can probably look up how many parameters that has quantized it down to 8 bit int and make an educated guess I don't know the number off the top my head though okay anyways to include the mobile runtime is literally one line in your pod file for cocoa pods and iOS or your maven or Gradle file if you're living on Android so model parallel training I'd also mentioned earlier so this is becoming more important with the big NLP models I feel like right now we're kind of seeing the the NLP revolution that happened to a computer vision a few years ago and but it seems to always be happening with really really big models and so to the point where a lot of people are finding that they have trouble fitting these models into memory on the hardware they have so we're offering you the option now to slice up the model and distribute it across nodes and have the computation graph work across physical hardware so this comes with an RPC framework that you can use EVs are not doing distributed training supports running functions remotely reference and remote objects all the auto grad magic that I happen to bout that happens under the hood where you do your forward pass and it computes the backward derivative derivatives for you that is that works across nodes with model parallel training and there are examples on PI torgue to show you how to do this with LS TM and reinforcement learning Java bindings I'll just mention briefly we have a Python front-end we have a C++ front-end and now we have saw this experimental Java front-end as well only for inference only so no autographs support for training and only on Linux at this time but expect the Java support to grow with time besides the core framework we also have some tools that we've developed within Facebook they're you know Facebook AI research or AI Applied Research have built these one is classy vision so if you're doing an image classification problem and you're trying to figure out okay what is the right model to use the right loss function etc classy vision might be a great tool for you because the the workflow for an image classification problem is pretty well known and so we basically sort of commoditized and parameterised it we're now even the model type can be a parameter to this classification training run it's also integrated with PI torch elastic training so if you are you know wanting to have an elastic cluster that will take up as many spot instances as you can afford it'll do that for you as well and this is actually something we use inside Facebook for weekly supervised CV training urge you to check it out also refreshes detector on the Tet run is now detector on - it includes latest models for the main tasks all the sort of masks are CNN and its descendants you'll do a pan optic segmentation as well dense pose etc you have the idea here is that the texture onto is sort of your one-stop shop for all your segmentation needs so that's detection on quantization so this was a feature that we rolled out with 1.3 last October but it's the performance and operator support has been improved for pi torch 1.4 the the basic notion here of course is that we do most of our work with 32 bit floating points but could we get similar almost as good results if we went down to a bit intz and goes an answer to that in the next slide but besides just the memory savings so you know going from 32 bits to 8 of course you'd expect for exchange in your memory footprint both you know resident and bandwidth but a lot of hardware architectures like avx2 include optimizations for integer math and so you'll tend to see a compute speed-up as well if you quantize your model so to give you an idea of the effect of that these are three different you know well characterized models that we're trained and then quantized with different different workflows but you can see you know for we have what - M - CV and one translation problem here minimal changes in accuracy but significant speed ups in compute time so continuing the theme of tools outside of the PI torch core also with 1.3 we introduced captain and this has also been approved as part of the 1.4 rolap so captain is a tool for model interpretability and so you can see here on the right hand side there is a picture of a zebra and a question what's in the picture and to the right you can see that it has highlighted all the most zebra ish pixels for us as well as highlighting what it thinks are the most important words in the question so this is an example of captain acting on a multi modal model to tell you what were the elements in the input that gave me this prediction of zebra with 0.6 confidence now captain will help you attribute your output to two features of your input you can attribute your output to the activity of hidden layers within your model and you can even attribute the activity of individual neurons within your model to an input or two earlier layers currently we're doing just a gradient based approaches but if you want to use like perturbation or other methods of you know assessing the significance of an input you can do that no it's so we have visualizations baked into it for CV and for text and so you can see here weren't the visualization for text yeah where it's saying you know I think the blue is contributed to the answer and the red is these words pushed away from the answer is it work but there's also a for generic inputs there's also a sort of stock visualization for that yeah you guys can have it tonight attention mechanism right so what it's actually doing is tracing backward through the graph and understanding for a particular output like what are its most interesting gradients through the graph and what input features do those relate to so it's not attention and captain itself is not a model it's a tool for interpreting models and telling you they're helping you understand the activity within them not just the weight but more it's more about the computation path through it was that if the weights or what it examines but it's actually kind of the the gratings yes absolutely so these two slides I showed you how captain actually came from a longer series of slides way they show code for doing this with your model and the outputs and yeah you can get numeric answers out that you can do your own visualizations with or whatever okay and so we're close to the end of what I have for you I want to point out some resources if you want to go and learn more the first is pi torch org this is the one-stop shop for all things pi torch it'll tell you how to install it full documentation tutorials including on most of the topics I talked about today like quantization distributed training all of it if you're interested in class II vision that's got its own website classy vision AI likewise for captain detector on to lives on the Facebook research github so if you are going to github.com slash Facebook research you should find it easily and then if you want to get your hands dirty with the code itself of course there's a github for that that's what I have for you dude does anyone have questions I can answer before we move on yes sir yeah absolutely you can you can have multiple oh sorry the gentleman here was asking about distributed training and whether or not a single physical machine with multiple graphics cards could be used with distributed training and the answer is absolutely yes you can have multiple training nodes on a single physical machine if you have the hardware to support it what have I seen at school lately I mean some of the stuff that Autodesk is doing I don't know if you use PI torch for the yet but there's a generative design tool that they have that I thought was pretty dope I turn to the hood you know okay no that was plate or so it was pretty cool I know all the coolest stuff all the coolest stuff I know about is still under NDA so I'm sorry but if you go to the if you go to a pie-chart org there's a model hub there where you can see like some of the models that people are putting up for others to use a good place to investigate yes sir yes I haven't had a chance to mess with it myself yet but pipe torch through the PI torch 3d library for like doing ml on 3d geometries yep that's new and very cool do your question alright I'll check that out anyone else yes well captain makes available multiple strategies but right now all them involve measuring the gradients through the computation graph so gonna yeah basically from the steepness of the gradients for a particular output you trace the path back to the input features that are most interesting that get to your question yeah I think I'm prepared to go much deeper on today but a weekend of another conversation if you want alright anyone else yes sir Oh oops classy vision detect rot so classy vision is the library that sort of parameter eise's your image classification problems and then detector on to is your one-stop shop for all things image segmentation anyone else yeah that is correct but they're built on top of pi torch and things that the communities has found useful for classy vision specifically you know it's funny I was just messing with classy vision with this so classy vision comes with this sort of its own versions of a lot of the models you find in torch vision so you know your resonates and your VG G's and and all that good stuff it doesn't have the pre trained ones but there's a workflow where you can take a pre train model or wrap it in a classy vision style model and feed that in so the answer is yes it takes a little extra work so the compressed version that question is will Facebook train our models for free so the slightly longer version is you know tools like collab allow you to like do some work on GPUs or GPUs are we gonna offer anything like that the answer is right now no but of course you can run PI torch in a collab notebook in fact most of the tutorials on PI torch org have a collab link up top if you want to go in there and mess around I think here I'll do long training runs that way because collab will kick you out if you're not paying Emily will have all your cloud computing answers all right anyone else all right more so yeah I mean top my head no I think get your hands dirty with both I think that's what everyone should be doing when they're picking a tool for a project get your hands burger bill get your hands dirty with both and make a choice that's my level of confidence in the products all tell you to try the other one - all right what put it one more and then it'll actually wants to move on yeah I understand the appeal I don't think it's quite there yet longer conversation but yeah it looks like thank you all for your time [Applause] you