Scale By The Bay 2019: Erik Reppel, Hack Weekend: ML models on mobile
Recording: Scale By The Bay 2019: Erik Reppel, Hack Weekend: ML models on mobile
[Music] so a little bit about me I'm I'm an ml platform engineer at Quinn base Quinn mace is about like it's a team about for machine learning platform engineers for machine learning engineers we maintain and deploy about like 12 ish models of which we have I think a couple thousand different training runs on and we build like our teams actually managed by the same manager rather tightly knit we do everything from like model development to infrastructure to the whole thing most of our models are fraud-related on tabular data we have a couple texts a couple vision our platform is basically optimized for taking like cutting edge research and running it in production as quickly as possible so like most of my job is to think about how to develop machine learning models then how to use them in in production just a quick ball like out of the people in the room how many of you have built a machine learning model before just by show of hands very few ish how many of you have shipped an ml model to production before and had it like actually interact with user data looks like three ish people and then how many of you have deployed a machine learning model on mobile one guy this is the guy ok this is like I I had never done this I had never run a machine learning model on mobile I figure like it's a gap in my skill set I'm gonna try to learn how to do it like I've never actually even done it any iOS development so it was a long weekend I've didn't realize it was a long weekend until the day before it's like you know what let's try to do this thing let's try to like take a complex machine learning model and put it on an iPhone and like I'm gonna live-tweet my progress and like let's see how this goes so you know what here's here's like the outline of the talk we're gonna cover like why you would want to run in a machine learning model on mobile we're gonna talk about the progress I managed to make in two and a quarter days and then like what the takeaways I had and where I think this is kind of going like I said I have never done machine learning on mobile before this might be rambling garbage Brett Kuhn so I'm assuming that's that guy he's given a talk tomorrow thirty four machine let me on mobile if you want to know how to do it properly probably go to his talk but this is kind of my green fields from the trenches experience going into it knowing very little to nothing so why would you want to run a machine learning model on mobile if there's a bunch of reasons it's good for privacy it's good for because you don't need a host the models yourself it's ultimately I think the real case is for UX in that you don't have to go and hit your servers and round-trip which means you can actually have iterative machine learning as part of your product which is is interesting I think it'll only continue to increase UX for like web and iOS and mobile in general but there's there's a lot of trade-offs like running a machine learning model in the cloud as is kind of my day job is it's pretty well understood we've been doing it for a while there's a lot of flexibility if the models not behaving you just simply like swap it out with the one that's better you can like a be test and track and you can emit events and it's relatively secure it's it's pretty it's pretty good mobile is hard mobile you're giving someone your model it is on their phone they can do whatever they want with it and there's like we haven't been doing it as long core ml came out in 2017 tensorflow came out in 2015 and tensorflow was like it's one of the oldest or it's one of the newer widely used machine learning frameworks we've been using like like d lybien torch came out in 2002 for example like that's i was in middle school i was in elementary school and mobile is it is just less secure in the security realm there's there's what's called like black box versus white box pen testing black boxes were you kind of like you don't have access to the internals of a system you just kind of send it data and it you look at the output white box is when you can actually see what's happening in the system and it's the same for models the cloud is black box functionally you can have rate-limiting you can add off your endpoints you might not even have a direct public publicly accessible endpoint we don't have a coin base everything goes through like a service or a queue or some Kinesis or kafka stream whatever on mobile like and we have this happen with the Quinn news app all the time someone will decompile the app extract the model and then just be able to do whatever they want with it with the weights with the inference the whole thing and like breaking machine learning bottles even if you even if it's a black box if you can send enough to request through it it's a solved problem like we can you can I'm sure everyone's seen like those like adversarial noise attacks or the guy wearing a a guy wearing a like shirt with like some like crazy colors throws off all the AI that that's real it's it's well understood so here's my plan here's the plan I had on the weekend step one I want to make an iMessage app I want that iMessage app to like generate texts and like reply to my friends when they like send me an iMessage so step one is is make the I message app sub two is get G B D 2 which is a cutting-edge like text generation model in Python get that working in Python then convert that model to core ml and then load it onto an iPhone that's that's basically the plan oh and then you have to like actually reply to the messages an etc but this is this is it and for the purposes of this talk like we can basically like just think of the model as a function that takes in text and generates more text it's GPD to is like it's very cool and we're gonna jump into it in a little bit just to like give you the why it's cool but like for most of it when I say model like function that takes X return tax returns text is like enough of an understanding except for one piece where it's by show of hands how many of you have like written anything with neural networks before ok so the way that neural networks run these days is that you basically define a graph of operations on tensors and so the way that you do that intents are flows by ahead of time defining a static graph of operations and like what will happen on different inputs so like X W and B are like tensors that you would define and then you're specifying this graph of operations that they'll kind of flow through like tensor tensor flow that's that's why it's called tensor flow and in tensor flow 2 and pi 2 which they're dynamically created and evaluated it's yeah it looks more like the gift we're like it happens as you do it rather than just like statically passing through this is just keeping in mind it's not important to like fully grep but you know good to know so GBD to real quick gbt to is a model open source by open a aiya if you like heard the headlines of opening eyes says they can't open source a model because it's too dangerous this is the model and they're like not wrong because GBD to is really really good I'm not gonna read this all feel free to read it but there's there's one mistake in this generated text which makes it seem like it's not done by a human it this should be quiz bot instead of flashcards and that's like pretty much the only flaw from this outputted text being grammatically and like be perfect in every other way and like that's kind of crazy I you know I took NLP courses in school if you would have told me we could do this in a few years I wouldn't believe you like it understands that quiz bot is a noun that like acts on things like that understands like noun subject verb prefixing it understands that like memory and remembering are related it understands like extension of concepts like the input text Allah mentions like English vocabulary and science it like pulls out that Lego there's there are other things that it could learn as well and like it understands that quiz bot is made by a group of people like there's an entity behind it it also pulls out the like school and learning is like equivalent or like belong in the context of remembering and like learn and studying and that that's like that's the DVD tear and it's like very good it would be cool to reply to people this is the motivating like factor to the talk so step one make the iMessage app turns out it's it's kind of a soft problem like the only thing I found is that you can't if you're in an iMessage extension you can't read all the messages which is like great that's really good to know I thought I just kind of assumed you could read the full catalogue but no so I like pivoted make it a text field you have to like enter text into and then a generates text but you know it's about the model like don't really care step one done like I like Swift development iOS pretty well understood step 2 GB d2 in Python so open and I open sourced the model and like a smaller version of its weights but not the full weight set so I'd already written like a couple of abstractions around the repo to make it a little easier to work with I just like reuse that code it's just as simple as like GBD to download model jpg to create the class and then GBD to generate gvd2 and by thunder like I'm kind of cheating I'd done it before convert the model to CoroMill bread is like laughing now because he understands like this is where the this is what the complexity lies like this is here be dragons we I don't know how to do this part this is and like there's a lot of unknowns when you're going to convert a like Python world model into mobile because there's a lot of kind of assumptions we make that I don't know how they map into the like on the phone world one of which is like can you have dynamic text input size like when you're tokenizing a string for a model you're generating a vector where each item in the vector is a integer which corresponds to a like token match of a vocabulary and so like if you have okay cool that's two tokens vector length of to fire emoji might be one it might be zero if it's not in the vocabulary you have and then like are you on Netflix question mark is probably five but it depends on how the vocabulary encodes punctuation number two is like they're meta parameters and in machine learning models like especially in DVD - there's a lot of things that affect the output of the model so gvd2 you can specify the length that you want the output to be the like temperature and top K are basically like how entropic the response is going to be and like matters a lot to the quality of the response and under different circumstances you want to have different values for that can we can I affect those at runtime or do I have to kind of pick values and then just live with that live with those choices another one is like can I swap out core male models can I like can I find two and GPD two on like a single person's catalog of text and then have it reply as that person the example being like there's like a million hours of Joe Rogan's like podcast transcripts can I just like fine-tune this model and have it reply as Joe Rogan or Michael Scott or John snow because then you can have like a little app that like you can choose who you're responding as and that's that's pretty sick and then like the last thing is like will this actually run on an iPhone this is a real big beefy model it's like 500 megabytes on disk it has 117 million parameters in the model will it actually run like this one I was relatively confident in this one but you know it's still it's still enano and I don't know the answer this but like it's it's enough so it's time to do the thing we it's not gonna work or it's not well we'll find the answers to what knowns as we go and hear the notes I had during the time like this is my rough plan step one is like I have all these like tensorflow checkpoint files I need to convert those into a dot ml model file I then also have like this vocabulary that I talked about I need to convert that and the way that it parses input vectors into tensors I need to write rewrite then and Swift and then like the checkpoint 2 dot ml model file which is what core ml uses as a like file type is like non-trivial because checkpoint files represent an in-memory graph of a machine learning model session the quorum LTF core ml tooling expects a dot PB file which is a different serialization format than checkpoint we got to jump through a few hoops of conversion to do this and like its it might be hairy and spoiler it will be hairy but this is this is where I'm at I've got a model I've got all the like state which is these model dot checkpoint files I've got like an encoder dot JSON I've got a vocab dot BP those are basically like how it will like tokenize all the input texts then I've got like hyper parameters which are like we can effectively ignore so like I don't know enough to like jump into this directly let's do the core ml hello world so like the first thing is granted this is core ml 2.0 2.1 because I was doing this a few months ago three dardo might have solved this core mill at the time did not support python 37 which you know that's that's pretty solvable just you know use three six it'll happen one day three seven well we as an industry won't be a year behind maybe next port next is like let's make a hello world model like this is just M NIST image classifier in Kara's TF Kara's the key point here is that I'm saving the weights in a check point format the TF check point format because what I'm trying to do is I'm trying to replicate that same like state of directory that I had from the open AI model and so like now that worked pretty well it did the thing at these like checkpoint files that are the same now I need to convert to a dot PB file and I did it like if you look it took me probably like three hours to figure out how to do this properly because it's like what you need to do is you need to load a in memory graph representing the model you then need to from that those check point files you then need to like take that in memory graph run it in a session which actually forms the graph and then use the TF writer API to like write that graph definition and state as a binary which you know there's like five different ways to do this and intensive flow and I might be missing some like there's checkpoint there's PP there's TV text there's md5 depending on what you're using tensile blow has an answer for everything but it means like moving between the different formats as a pain in the ass okay so now we have the model it's the same as what our opening eye model is let's let's go to core ml and just core ml you need the in you need the input and output shape of the graph and for something like amnesties e for a more complex model especially that varies an input size or and output size like gvd2 that's that's a non-trivial and in in pi towards for example you can just print the input shape in the output shape because the graph is dynamically generated and tensorflow you can't do that you have to like in t f10 you have to insert CF dot print operations into the graph statically and then create a new session and then evaluate it which is fun or you need to attach like the TF specific debugger you can't just use PV or any use standard Python spooling that's also fun and it also turns out like here's the second problem I ran into TF care is is not supported by core ml tools which is like for those of you who don't use like tensorflow and care is a ton Cara's is like an API for defining models since it was like that's great let's hire this guy's creator and like pull it into tensorflow and use it as sensitive loads and tensor flows main abstraction high-level abstraction so they did that and now like all the tooling hasn't caught up to the point where it supports TF cares it only sports Cara's annoying but like not the end of the world redo the Model T in Cara's and you know it's good I finally have a model I can run on mobile I do the thing I run the model on mobile I make sure it works it's it's great we I did a Ford bass there we are I also solve only unknowns so I was worried about the like model cooking the processor of an iPhone I found that core ml's like quantization works excellently quantization for those of you who are unfamiliar is basically just representing a float using a fewer bits as precision so typically use a float 32 half precision has you represent that float as a float 16 quantization allows you to do like arbitrary representation of that float and when I like quantize this model originally is 200 252 kilobytes like with the mm this with a 8-bit quantization it would be 40 kilobytes so worst case scenario I could quantize gvd2 to like 6 bits and it like might not be as good but you know it'll run and it'll be like 20% the size of the original so like now I've done the whole world it's time to do the it's time to do the real thing but as I'm reading through more of the core ml Docs I find this which is that TF core mental converter has a following constraints TF graph must be cycle free this is a big issue because this is the code for gbd to function what it does is it for each input or sorry for each output of the that you want for text it just is a while loop and for that so if you want an output text length of like 10 it will while loop for 10 and just like produce the output indices for your cab you Larry and that's like in the that that means it's a cycle like a while loop is a cycle and so if you're going to do that like it means you can't use core ml for this model at this point and it's not just at first I thought like oh maybe it's just TF core ml maybe I can like write a custom serialize ER that'll that'll do it turns out it's all of core ml does not support cycles which kind of makes sense if you think about it really hard and we'll get to that later so it's it's theoretically possible to do gbt to on an iPhone in a weekend just not not by me in a weekend so I you know what you pivot right like oh sorry this is if I were to do it this is how I would do it if I had more time well you what you do is you have to unroll that while loop so instead of having a while loop you say I'm only going to support outputs for length 50 carat 50 words and then you just like unroll the while loop for 50 iterations and then now you have a static graph and now you can run it and then you just have to re-implement the text encoding and Swift and then you just have to glue it all together and it'll it'll do the thing but on the weekend like can't convert the model to Cornell that's that's a break we got a pivot I'm just gonna like have the extension call the server because that's really easy and I know how to do that and that's my day job less fun but you know when you live tweeting stuff you want to have a product to ship and this took like half an hour like running a model on a server is like something I do all the time it didn't take me very long to like build the prototype and then I just like you have the last little bits of the iMessage extension have it post the server display the text etcetera etc you know what it's 11 p.m. on a Monday night of a long weekend close enough right except then this happens which is which is interesting so like Jillian Shem odd is the CTO of hugging face if any of you use like pi torch transformers for like bird or gb2 or any like pi torch text model you might have used this code him and i had been like chatting about this because i'd like i have like tweeted at the hugging face guys before we were Twitter friends I'd say and he liked it several months afterwards like I don't know three months afterwards he like tweeted me this which is hugging face implemented exactly what I wanted to do they released gbd 2 for Swift in core ml on github and you should go read this code it's really cool you should check it out but it's brutal like the amount of work they had to do to make this work is really a lot like don't try to read all this code it's a lot just know the left side is like the prediction code it's the glue and all the other stuff the right side is the like encoding code and I truncated these files like there's like hundreds of lines of code I think you know the tokenizing alone is like 700 lines which ok fair enough stuff the tokenized stuff predict is not not terrible but this is the terrible part because ml multi arrayed which is the like type that is used to represent tensors that are passed into Cornell models don't support math operations they had to roll their own math operations so for example here they're trying to do an Arg max which was basically get the highest value in this array and they have to like do to do it efficiently they're doing like DSP pointers to like DSP operations on a raw pointer using the metal API and it's it's rough and like on the right there that's like they had to write like 600 plus lines of code in Swift to like do the basic math that like ml multi array is kind of missing and I said that's the worst part this is the worst part so in order to like actually get it to work they had to redefine all of GPD twos structural graph in Cornell's neural network builders meta graph option so what they let's let's trace that back gbto is a graph right its operations are graph structure they want to run that graph structure in core ml how do you do it you can't convert from anything in Python land to core ml land you need to write it in core ml and what that means is like you're doing several hundred while several thousand of these builder dot add expand dim operations to define like each node in this neural network and each operation between tensors and again it's like four hundred lines of just defining this meta graph so that you can copy over the state and like they even did it pretty efficiently like they did a while loop in a they looped over like how to expand for the while like they did a relatively good job I I did they did a great job as what I should say I I can't think of a way to do this that would be better but you know it sucks this is in the comments of the source it's like I hate this I don't disagree it's it's pretty brutal that you have to like do all this and this is where we get to the takeaways number one like Swift is a really great language I think that Swift and you just watch this wit for tensorflow talk I think what Paige and Chris and like the team are doing is great number two is like currently shipping an ml model for mobile is a large amount of discrete engineering work it is not at the point where you just get it for free you need to have people on it you need to like spend time and actually do it otherwise it's not going to work TF light might be a little easier it seems to have higher like op coverage core amela least is like a large amount of discrete engineering work the tooling is still immature and then the reason why it takes a lot of engineering work is that the tooling around ml for mobile is still immature the Interop isn't there and ml engineering I think is still immature as a profession like ml for each research has been around since the 80s we've only been in like unless you're google we've only been running machine learning models in production at scale for like 10 years at most and that's not enough time to like build all the engineering rigor and kind of best practices as an industry especially if the models are changing as rapidly as they do which which happens and this is the example right like this is the entire definition of ml multi array in core ml and you'll notice like there's there's like it counts and like shape and like get a pointer and like get the type there's no add there's no multiply there's no it's not numpy it's basically in a rave with like slightly better like this ability and like that's that's fine it's just like where we are right now and it's because like a lot of Cornell is built for this model it's built for a model with like a static input size and a static output size and you're going to do one forward pass and then you're going to do something with the output and that's for that it's great like it's super easy to take this model or not and in core ml but and like the reason for that makes sense if you think about what core ml and Tia flight actually do so core ml uses dot ml model CF laid its final extension is dot T of flight in reality ml model is a proto buff file and what they're doing and T of flight is a flap up her file what they're doing is they're interpreting the graph operations so like whatever's there there's an interpreter which says like I'm doing this operation here's the most efficient way to run it like I'm just going to like interpret this graph and do the thing it doesn't have any sense of like stopping conditions and that's why you can have while loops because you might just like be no opping through this iterator like forever it also means that like machine learning for mobile is always going to lag behind cutting-edge ml and research because like if you come up with a novel new activation function that gives you like 5% increase in performance and whatever like you you invent swish or like some other new activation function you need to go and implement it in core ml and add it to the like set of things that you can stick in this graph and that takes time like each operation is defined and you can actually look like that screenshot there I realize it's not very big is just like a preview of the operations in the core ml documentation that you can use and that's it's not a super complete list it's like it's always going to lag behind cutting edge and this is a problem because more complex models these days are their control flow with neural networks like alphago for example it's a Monte Carlo search tree which evaluates the position with a neural network at each point which basically means it's a tree with a neural network in it and like GPUs the same you saw the while loop it basically like outputs and then it has stopping you nisshin's and control flow and everything else you have in a traditional program and I think this is only gonna become more common it seems like you need a little bit of both we're not just going to have static a neural networks that just do everything anymore and so this is the like this is the this is why Swiffer tencel it could be absolutely massive is because it's the only like Alex quorum LT of light they're all great for serializing static models but at a certain point you need a compiler and tftf for swift for tensor flow is the only thing I've seen with the potential for write once run on any device like here's the here's an example for you like if you wrote alphago zero in Swift for tensor flow theoretically and so it for tensile is not here but this is where it could be in a couple years you write it you write it in in swiffer tensor flow you train it on the million GPU server farms that Google has and then you just cross compile at forearm and run it on a phone and ship a quantized version of the weights and that means that with effectively two more CLI commands you can have that model running on running on mobile I don't know if any of you write code but like the thing that I think stands out to me most about go is how good the compiler UX is and that you just define the architecture and like the OS you want to target and it just will cross compile for you and like Swift is relatively similar similar because it's a low V impact that that's a world we could live in we're like cutting edge research hits consumer product within a couple days obviously we're not there yet but that that's what it could be in general ml compilers are going to be huge Swift for tensile might not be the winner in all this but there's it's gonna something is going to and that's because people keep putting specialized neural engines in consumer chips this slide is from the open AI they're sorry it's from the Apple iPhone Pro event and it for their ASIC a13 ship they have like a specific region dedicated to running neural network operations it's only gonna become more common soon we're probably gonna see laptops which have discrete processing units that aren't GPUs that are meant for forward passes of neural networks and it turns out like at the limit we've done a lot of work to figure out how to define neural networks and machine learning models at a high level and we've done work to make them fast on CUDA and like x86 but we haven't done a good job of making it like we're in the java stage we're like something's gonna come around and let you Brown it on everything and it's gonna be big because it's turning into a compiler problem everyone is like everyone's starting to come out with around a 6 or TP use or like alternatives to CUDA and we need to be able to run there and there's a lot of projects gloaty vmx elite julia that are like attempting to solve this and it's it's great it's good that this ecosystem is evolving and this is like mobile is just a proxy for this is that like swift you just you just heard the thing they're gonna have differentiable programming in the language that's going to be a first that is like a systems issue is going to have differentiability built into it and that will unlock a whole like domain of programming tasks that we couldn't otherwise do and here's here's maybe they like the motivating piece this is a paper released by like Jeff Dean and Google in 2018 and what this paper showed is that if you think of B trees and hash and bitmap indices you know database as learned indices or models and you just replace it with ml models you can outperform cache optimize B trees by as much as 70 percent and use an order of magnitude less memory and like ml for systems is on the rise like it's an entire workshop at nerfs this year for the first time ever it's got like a massive slate of like interesting talks Jeff Dean's gonna do the keynote and I think it's where we're headed I think that having machine learning models as part of systems is only going to become more common and it's gonna make our programs better and our UX better and that that's what I learned that's that's my time thanks you can hit me up on all these things that's that's it we can do Q&A if anybody has anything