Scale By The Bay 2019: Paige Bailey, Swift for TensorFlow: Machine Learning with No Boundaries
[Music] so thank you for coming to the presentation today I'm really excited to be here in this beautiful venue to talk a little bit about the project that I've been working on for the past several months as well as tensorflow in general if you have any questions about the framework so to get a little bit of a feel for the room how many of you are Python developers couple cool how many of you are as machine learning engineers a couple more hands all right and then how many of you are just interested in deep learning because maybe you saw some salaries posted somewhere and it looked pretty interesting cool so hopefully hopefully there's something of use to you and this talk Swift for tensorflow is a little bit different than tensorflow in general it's it's kind of a reimagining of the project itself and I hope that I hope that there will be some some good questions towards the end so a little bit of background my name is Paige Bailey I'm currently a product manager on the tensorflow team at Google brain working both on tensor flows api's particularly its core api's as well as Swift I've had a little bit over a decade of experience doing machine learning at places like NASA and Chevron and just before Google at Microsoft and my background is not computer science at all other than some master's work it's geophysics and applied math so coming at it from from a bit of that perspective I also did not have any experience working with Swift prior to starting on this project so it's been a little bit of a roller coaster ride how many of you have used Swift before or I've seen it ok so a little a few more hands than Python so hopefully hopefully this will be fun so an outline for today what is Swift for tensorflow and why is it different why is it particularly useful for deep learning and then also some ideas of how you can get started if you would like to so deep learning has really taken over in the past several years right it's gone from being sort of a research project to being something that's ingrained in every single application that you use in your life right so you you're able to take models easily port them to browsers to mobile devices to do things like computer vision tasks but also really interesting things like natural language processing to do automatic text generation to be able to do speech recognition or speech transcription and this is all happening in near real-time on your devices so this is great it's wonderful that sort of machine learning and deep learning have become so ubiquitous but it also means that you have a lot of constraints that weren't necessarily apparent when this field first started for your deployment targets so being able to port something like Bert to a mobile device that is not that is not a light challenge like these models are megabytes in size is even just kind of the the weights or the graphs themselves and finding ways to quantize them efficiently and make them performant is really really trying we've also seen an explosion of interest in the research space right so not just these production and and very lucrative business II sorts of deployments but also the number of archive papers per year so here you can see that we've actually surpassed Moore's law as of 2017 and if I had extended this chart to 2019 it would be even more impressive right like we're hitting these exponential growth curves that are sort of delightful to see but also a little bit staggering because some of some of the papers that are produced are not necessarily immediately applicable to to us in our everyday lives but many of them are our models that are highly interesting and could offer substantial business value if you are able to incorporate them into your web app and so what we see are people who want to be able to convert these models as quickly as possible and get them into production we've also seen rapid accuracy improvements these are some examples of image recognition and and sort of object detection models and you can see the step change performance improvements over time but it's it's becoming harder and harder to get these without doing sort of interesting algorithmic fusion techniques so being able to take being able to take sort of traditional methods like Monte Carlo tree search and marry it with some of these deep learning techniques and of course deep learning is central to everything that we do at Google and alphabet so you can see here on the left the number of unique project directories with deep learning models at alphabet over time it's the trend continues through 2019 and we've started incorporating models into every single one of our products so everything from YouTube to drive to to Android so for example if you have a pixel device you might have noticed that just recently speech-to-text if you speak into your phone got very very speedy and that is largely due to tensor flow light models being embedded directly on your mobile device so instead of having to speak into your phone and then go and ping a server somewhere with the data and then come back now that is done directly on your cell phone so your data is stayed secure and also you have much lower latency we've also seen an explosion of specialized Hardware right so here you can see some examples of TP use over time our first-generation TPU was just for inference so just to be able to get results from machine learning not from training and then we've continuously built out the scale and the capabilities of these pods so at the bottom you can see the latest version of our TP use that are currently available on Google cloud and in Google Data Centers these allow scientists to be able to run models that historically would have taken weeks in just a matter of minutes or hours and if that's the case you suddenly get into this state where instead of having you know a delay and your your experimentation process you're able to truly do rapid prototyping and if you get results immediately you know if you're if you're able to set off an experiment and then go and get a coffee and find out if what you decided was right or wrong you can really start innovating much more effectively so all of this is great right like why why Swift why is it useful particularly for these particularly for these scenarios that I just described so these these more novel these sort of novel deployment targets specialized hardware these this need and desire to be able to do things like algorithmic fusion and for this one of the examples I always like to give is this is from a book called Python interviews and it details a story from when Google was first starting out it was trying to match the speed and the productivity of one of the startups that was operating in the Bay Area and they they were so shocked that this startup this tiny this tiny team was able to iterate so quickly that they had and maybe they and Google had the same idea at the same time but this startup was able to ship features very rapidly and Google who was using C++ at the time was cranking out features much more slowly and then after they acquired the company now called YouTube they they realized that the reason why they were able to crank out these these enhancements so quickly was because they were using Python as opposed to C++ and even if the the first bit of performance wasn't necessarily as great as you would get from C++ you could fine tune it as soon as you had a Minimum Viable Product right like you get the you get the first thing that works out the door and then you can start doing these performance enhancements and the cool thing is right is that for those things that I mentioned most of the development work is done in languages like C++ right late is what people that is sort of the the norm for people who want to deploy things to to embedded devices maybe C instead if you want to deploy models to mobile devices if you want to be able to do these these very sort of bespoke configurations for distributed training you're probably doing them in C++ and C++ is painful often like I mean it's great in the sense that Python is also painful for different reasons but C++ takes a great deal of time it's often difficult to understand for people who don't come from that particular background and it doesn't give you the affordance of being able to rapidly prototype Swift gives you all of the performance benefits of C++ so we continuously see performance just as quick as C++ or C even but it gives you a syntax that's very friendly and feels very similar to Python so I'm not sure how many of you have seen Swift code before that's an example on the screen you know it might have some let's sprinkled in and a few more a few more curly braces maybe or some indications of types but if you squint a little bit it looks a lot like Python right and and so if you have a situation where you're able to do these very sort of low-level tasks in a language that looks like Python the world is suddenly a bit brighter right and so here you would see an example of an image classification model using swift for tensorflow you have a model you have some layers that you define up top and then in order to actually differentiate through those layers all you would have to do is sort of decorate your function with at differentiable and this is capability that we're up streaming to the Swift programming language itself so any function literally anything not even if it's machine learning related you would be able to differentiate through it to get the to get the gradients to get the change over time with just this this short prefix we're integrating differentiation and these you know things that have traditionally been thought of as very machine learning specific directly into the language because we believe them to be that important if you want to train your model all you would have to do is define some optimizers you know maybe some distributions and then set a for loop and you're off to the races all right so it looks just as you imagine it's very straightforward syntax almost identical to how it would be in Python but you're still getting the the performance benefits that you would have using a language like C++ and this is just another example taken from a recent a recent course that was taught by Jeremy Howard who is the creator of fast AI you can see an implementation of a model and Swift up top and then the same model implemented in Python down at the bottom it's actually kind of funny in that the Swift model is a little bit more concise than the Python version but they look again pretty similar and and pretty straightforward to understand another nice thing so how many of you have looked at assembly recently not very many hands but if you did want to go take a look at God bolt I strongly suggest it it's it's a lot of fun to see kind of how the different languages that you use every day sort of translate themselves to assembly code and what you see on the left is a function implemented in Swift and on the right the equivalent assembly code and that's about is that's about as sort of efficient as you can get right like every single thing that you see happening on the left is sort of directly correlated with what you see on the right if you tried to do the same thing with Python you would you would probably be a little bit surprised but yes so God bolt org lots of fun to play with the next bit that I would talk about specifically how Swift is afforded to mathematical operations scientific computing a machine learning in general is value semantics so with Python you have sort of the concept of value semantics for integers right or or for variable assignments rather so so quick question if you print B on this screen what what is the output just shout it out yep I heard it in the audience so it's three right because Python for this for this assignment follows follows something called value semantics but if you have if you have list right so if you if you do the if you do the same sort of concept but with a list what you're going to get is a little bit different you're getting reference semantics and that is not how math works right so Swift erase follow value semantics for everything so so you have you have a similar situation and they it's with Swift you get precisely what you would expect and that's the rate they're sort of rationale for this why this would be beneficial is because that is how math works and if you're able to have a language that operates in the same way as you would expect mathematically it'll life is a is a great deal easier and variable handling especially is much better automatic differentiation and Swift so I was having a little bit of trouble with the internet a little bit earlier so I'm not going to go through the I'm not going to go through the exercise of executing all of these cells in our example notebook but we do have this available on github.com slash tensorflow slash lifts if you would like to take a look and try it out yourself how many of you have used collab notebooks before or Jupiter notebooks alright so several hands if you haven't ever experienced them before I strongly suggest taking a look they are really fun ways to sort of understand to understand models that you create and then also to immediately get responses for the for the work that you're doing and all of these cells that you see there either markdown or or language cells that are immediately executable so you can see here a little play button you can also hit shift enter and each cell execute this is all done for free in collab using CPU GPU or TPU you can actually toggle between the the runtime types in the notebook as you work but here you see an example of differentiable programming using Swift you have a function some gradients and you can see here an example of sort of doing doing that differentiation on a function particularly square roots and creating a custom a custom derivative for square roots and this is this is partially because everything in Swift like even the types are fully customizable so if you want to have a custom type example be float16 which is what you would need to order in order to run on TP use you're able to to create that directly within language this is also an example of sort of the C Interop which we'll talk about a little bit later you can import any any header or a library from C or C++ and use it exactly as you would expect and again all of the we have a huge collection of demos available on github strongly suggest taking a look and trying them out yourself so Swift for machine learning it is cross-platform and this was something that surprised me when I first started with Swift I had always had kind of the understanding that it was an apple language maybe like it worked on all of those things that were prefixed with the I like iPhones iPads maybe the things were prefixed with Mac too but it actually can go anywhere C++ can go so that means Linux Mac OS Windows Android iOS and even embedded devices there there have been a number of sort of thesis or theses and projects recently showing that you can get the binary size of Swift reduced down to something like one like 1.4 Meg's and that means that you can start running it on really interesting deployments targets and syntactically it's very similar to Kotlin so if you are an Android developer and you wanted to be able to use Swift inside of your Android app you could just compile your model down to a dot ISO file and call it as you would any other library right so so that's really really sort of interesting to understand it also means you can write your model once and deploy it anywhere so I having to instead of having to create a completely different pipeline for data ingestion data pre-processing model building and then sort of the post-processing step for your model for the you know if you're operating on iPhone or if you're operating on a server if you're operating on an Android device you can just have a single a single pipeline for all of it and deploy it also gives a lot of productivity and customizability right so with typed API is static detection of errors you also have nice things like semantic aware autocomplete and jump to definition and you also have customizable abstractions and user space and it's also being integrated or in the process of being integrated as syntactic sugar for something called ml IR so how many of you have heard of Emily are excellent so ml AR is a new project from Chris Lattner who is also responsible for LLVM the switch programming language itself really great guy to work with a very sort of bright and focused on community but ml IR is a new project that's specifically focused on taking the pain out of having having complex deployments so what does that mean that means so for example right now if I was implementing a model and I was using something called like scikit-learn maybe which currently only works on CPUs if I wanted to deploy that to a server or to a phone for some reason that we would run into the constraint where maybe some of those ops would work on each target and maybe they wouldn't right or to be to be maybe a little bit more concrete if I had developed a model using tensorflow maybe the Python version some of those ops might be gpu-accelerated and if you deployed to a target that doesn't have sort of that afforded to it there's no guarantee that though sort of bit of model building would be punted back to the CPU to run as opposed to attempting to run on GPU so you might have a silent fail you might just get bad numeric responses or you know just bad things would happen like it's problematic and that you essentially have to hard code everything that you build to work on to the places that you want to deploy them to and when you think about the mobile device space with you know thousands of different kinds of thousands of different kinds of devices all of which have different kinds of hardware very difficult to know like what your user is like what what to expect like does your user have a mid tier device a load to your device a high tier device if you do deploy a model like what is the what is the confidence that you have that it'll actually actually execute in the way that you expect it to it's just very problematic way too complex and so M lar is building on top of something called lob m and it allows you provided there's an MLA our dialect to write your model once and to deploy it anywhere and not have to worry about the complexities of those targets it's also an open source project that was recently open sourced to the LLVM foundation and they have open design meetings every Thursday at 10:00 a.m. so if you would like to learn more strongly suggest sort of subscribing to the mailing list and staying informed Swift is also interoperable I think I mentioned this little bit earlier but there are no wrappers you can just import libraries directly from C and call it so here you see importing G Lib C we also have c++ Interop so for this is something that an engineer on our team named Parker has been working really diligently on you're able to pull in any arbitrary header and then use it directly or extend it in ways to meet your use case and this is really exciting particularly if you're operating on a company with a lot of C++ a lot of historic code that's very useful and lucrative to your business an example would be Google and you don't want to have to rewrite the world in order to a new language there's also interoperability with Python so if you are coming from the Python space you probably realize that there are a ton of data science libraries available that make life a great deal easier so examples would be numpy or sci-fi or matplotlib or you know as Seabourn or any of the other data visualization or data pre-processing or sort of machine learning capabilities for traditional tasks with Swift you import Python and you use it just as you would in a Python IDE alright so you can import numpy use it in an array get back exactly the the sort of values that you would expect and this is all implemented in pure Swift you can also create custom kernels directly in Swift with the jupiter notebook so here you see an example of one d average pooling i engineer on our team named Eugene also has implemented a custom CUDA kernel inside of a jupiter notebook and historically those would have been sort of very difficult or impossible even to include in your machine learning models and now they're they're just as simple as being able to implement a few lines so with differentiable programming you can also have custom derivatives user-defined types and it's flexible where you need it we have language integrated auto diff which I showed a little bit easier our a little bit earlier you prefix any function with that differentiable and you're off to the races and for performance it's just as good as C++ often just as good as C you can also have multiple threading so you're not at the mercy of the Gil Python is a single threaded and you also can have graphics trashin this is a picture that was taken from a recent a recent course that Jeremy Howard the author of fast a I taught with Chris Latner both of them are available on YouTube if you'd like to learn more they're about three hours long but they're really exciting watches and they kind of run through the concepts of machine learning implemented in Swift and in Python and then also why something like Swift is incredibly useful for machine learning tasks I also think that it's interesting Jeremy sort of famously was a very avid tensorflow user and then migrated to PI torch specifically for performance considerations the things that he needed to do and tensorflow just weren't as fast as the computations he was able to get him PI torch at least not at the time and then after after working with PI torch for a bit he transitioned back to tensorflow specifically for Swift for tensorflow so he's taken as fast a a library which is very popular and sort of a concise way of expressing machine learning concepts and he sported the entire thing - Swift and he's calling it Swift day I instead of fast AI because puns are awesome but it's also available and github if anybody wants to try it out we've also been working really closely with the alpha the alpha zero team and the alpha go zero team and this is a kind of a perfect partnership these folks are based out of deepmind and they're responsible for the the model that was able to beat a human it go they have their models a combination of three technologies so it's deep learning concepts Marni Carl Ertz research and then also they're running it on massive clusters of TPU pots and they were not able to actually implement all of this model in Python they had to revert to C++ in order to do it to do it effectively and we were able to port the model to Swift and saw precisely the same performance as the C++ instantiation but only a fraction of the lines of code and it was actually a much more efficient implementation and in terms of like the the way that it was configured we've also been working with a team called open schpeel they do reinforcement learning as well at deep mind and they've just recently open sourced open source the framework if you would like to play with it feel free to feel free to send pull requests or to use it and in your academic work or at your work and let us know how you like the the swift implementations we've also focused on developer tooling so we have all of our documentation available online is sort of auto-generated from from the Swift we also have sort of collab examples and notebooks available for you to use and we also have support directly with nvs code so not just Xcode right you're able to get those things that I mentioned for Symantec where autocomplete jump to definition and it's actually quite nice if you have experience using Python which doesn't necessarily have the same benefits so future Direction ad in the process of being up streamed we're working on integration with mobile and embedded devices you can see some of that work talked about in our open design meetings which happen every Friday morning at 9:00 a.m. Pacific time we're doing C++ Center up concurrency and ownership and if you'd like to get started everything is on github it's all open source we have a whole bunch of collabs available for you to try if you're new to machine learning or if you're a seasoned developer and you can stay informed by joining our Google group a swift at tensorflow org by going to tensorflow org slash lyft and by checking out our github repo so thank you for coming today really excited to tell you a little bit more and I think we have time for questions so if you have any feel free to ask I'm delighted to answer about Swift or about tensorflow in general or machine learning or what have you thank you [Applause] [Music] hi I've heard of Swift but I was curious what's the GC story and Swift garbage collection so Swift is not garbage collected that was one of the reasons why we decided we decided to pursue that as opposed to something like Kotlin witches garbage collected languages are not particularly well-suited for many machine learning tasks especially deep learning tasks so that was that was one of the reasons why we selected Swift excellent presentation thank you uh I'm actually not familiar with the concept of differential programming and can you I know it's sort of a general crest question but can you drill down into what it is and why it's so important right so so that's a great question and differentiable programming as is so it's still kind of married to the idea of machine learning and deep learning because it's all about change over time and understanding sort of the the gradients of the operations that of the operations that you're performing right so machine learning and deep well deep learning in particular is just a whole bunch of mammals and in gradient computations and if those are the only operations that you're performing so just matrix operations over and over and over and over again if you're able to if you're able to take those those very base Lego brick blocks and incorporate them directly into the language directly into the into sort of the compiler behavior it ends up giving you a lot of performance winds and it also dramatically reduces the complexity that you as a human compiler have to have as their architecting out your programs so so differentiable programming is so important to deep learning I guess because of its one of the most sort of foundational Lego bricks of deep learning as a concept and understanding change over time what actually happens when you put that at differentiable imitation right so so with tensorflow if you have experience using using that particular framework the same concept is the same concept happens when you apply something called gradient tape and in tensor flow it's you would have like with gradient tape and then you would start sort of collecting variables as you as you go through some sort of change operation so you would just be sort of collecting variables in an array and understanding the differences between them and that gets very difficult to wrap a person's brain around way or at least my brain it's difficult to understand sort of how to deal efficiently with those variables especially across clusters of machines or if you have lots of variables or higher order differentiation operating simultaneously so at differentiable does that but without you having to define a gradient tape and having to keep track of all the variables yourself it just does it for you so you know is this future festivities to actually build all these mathematical libraries natively or to depend on Python for instance numpy today you know if you actually wanted to write some deep learning or machine learning algorithms in Swift basically depend on numpy so you know running down the line or two years down the line is the roadmap for swift is to actually build all this natively that's a great question so the road map the road map for Swift itself some of those libraries have already been built out so this summer we had for google Summer of Code students two of which built out a library called Swift ml which is essentially a scikit-learn replacement one of which built out Swift plot which is replacing matplotlib and then we're in the process of building out something called penguin which is a replacement for pandas but implemented in pure Swift so the data science ecosystem and tooling in Swift is small but it is growing and you can use the the python implementations if you prefer but if you want to see the performance benefits of operating in a language like Swift and not have to do the single threading then you could just use the the Swift implementations yes all of that's in github yep hey I think we got time for one more hey my question is if you or anyone on your team has executed some standard training for spur training for some standard machine learning problems and compared performance for tens up compared performance with Python or C++ so so I'm not sure I'm not sure I heard the question correctly some do you have specific numbers how fanciful 4pf it's whip is how much better than TF with python for some stranger algorithms right so so I guess the question if I'm understanding correctly do you have a performance comparison of Swift versus Python for doing the same operation yeah yep so so I don't think that we have any listed on github though that would be that would be really interesting to see a couple of ad hoc examples that I can give is we recently we recently partnered with the team at Google I can't say which one because just sort of they haven't they haven't talked about the work that they've done publicly yet but we were able to take their Python model which took so about a minute to train on on kind of a constrained device setting and we were able to get it down to sub one second which is which is kind of awesome and this is partially due to like it was it was kind of a spline based model so it wasn't doing anything too awful complex and it was very sort of CPU focused as opposed to like mmm like massive cluster of GPUs but it we were able to see performance enhancements of over a hundred X which is kind of crazy another example would be Jeremy Howard and one of the lectures that I referenced he was able to implement a hyper performance data pipeline using C instead of using sort of the the typical Python tooling that you would see and he got a 10x perform an important improvement there so so those are those are just a couple of examples the the performance benefits aren't due to you know tensorflow being you know slow or an architect like it's not because of that it's due to if you're using a typed language or if you're using C++ like you you end up getting these these sort of massive wins without having to try to awful hard and whereas Python if you've I love Python Lee it was one of my first my first language was basic because like text adventures on Apple twos but like my second language was Python and even though it has a you know beautiful vibrating ecosystem of products it does have kind of very tangible performance limitations thank you [Applause] [Music] [Applause]