Scale By The Bay 2018: Chris Fregly, Continuous ML Applications in Production
Recording: Scale By The Bay 2018: Chris Fregly, Continuous ML Applications in Production
you so I've got kind of a fun different way to do this talk it's accomplishing a couple goals so we're going through the fundraising process right now and all that's been on my mind is these stupid PowerPoint slides and so I figured I'm gonna combine the two because we actually have a pitch right after this so I'm gonna explain this explain this this concept and what pipeline does in terms of our pitch deck so you guys get to see that we started using this tool because it's called beautiful AI because one of our potential investors actually that's one of their portfolio companies so it's an interesting tool they they claim to actually do AI for your slides and but really all they do is kind of color matching and things like that so I guess that's that's considered a high but so what we're going to talk about today and if you don't mind I'm gonna update these slides Esper talking here is this concept of sort of the AI runtime and actually there's a talk after lunch today by my buddies from hydrosphere that's gonna cover some of these things as well and you know this is really bringing life to these machine learning models most of the machine learning companies that you talk to or or hear about our you know kind of rooted in spark rooted in in batch type processes and you know this is very boring for us we've been doing batch you know like coming from Netflix coming from data bricks actually I used to work there as well you know batch is kind of old-school so we'll be talking about training models off of kafka streams continuously training these things and ways to get these models into production so let me put this in presentation mode and see if that helps it all here get rid of my bookmarks so we're very much runtime focused right so basically yes everything that happens like yes after the initial model training and I point out the initial model training because we are continuously training these things and I'll show you guys a bit more on that later during the demo and also when you kind of own the runtime which is one of our goals with with pipeline AI is to own the actual prediction runtime and if you're familiar you know some of you that are 30 and older I'd say 40 and older probably remember be EA systems back in the day right very very fun company I used to work there and we we owned the you know Java transaction space and so the ability to start to add value into these you know sort of transactions these like prediction transactions is very valuable you could start to add things like model explanations you could start to route traffic in smart ways you can start to combine model predictions into one prediction ensemble so there's quite a bit of engineering that comes after the models been trained and all the companies up till now really focus on offline hyper parameter tuning is the term and it's funny these VCS actually know this term they don't are sorry they know hyper parameter optimization they don't know hyper parameter tuning like to them that's you know totally something different so you have to when you speak to these VCS it's very interesting and I apologize if there's VCS in the like the audience but they kind of lock into certain you know Triplets of words right and or you know like tuples of words and if you don't snap into that they have no idea what you're talking about so we do online hyper parameter tuning so this is as these models are predicting right we're collecting data about these things and we could start to then make recommendations and say make some changes on the training side and that that will then improve either the prediction performance or the actual system performance of these predictions online so to contrast this you know there's folks like data Britax h2o Domino data labs all these folks focus on offline stuff batch boring there's folks like Amazon sage maker as your ml Google Cloud ml these are you know very much building blocks they they each you know have their cloud lock-in of course whereas where multi-cloud we write to kubernetes you know this is a very technical audience so I'm so I'll highlight these things so just as long as we can get to you know basically docker and then we can build a kubernetes cluster we can deploy pipeline so the the sort of entry point to pipeline is the trained model and you aren't just giving us the actual trained binary or the the trained asset the Train graph and trained weights you're also giving us what was used to train that model which hyper parameters were used so yes this is how we actually combine online metrics with offline metrics we use something called ml flow now this this just came out of data bricks a few months ago and it's essentially an experiment tracking project so if you go and very interestingly enough data bricks chose to not make this an Apache project and as you know like a lot of you can imagine these Apache you know projects there's quite a bit of control that you lose when you donate these projects down to using you know Nabal for your user lists and stuff if you try to use anything else you get your hand slapped one thing to note too is that tensorflow itself is not an Apache project and this lets folks like Google who run this project start to insert things into the open source that are a little bit questionable right so they start adding things like oh hey by the way your model will look really good on a teepee you you know here's a free coupon go go get $300 in credit you know from the actual open-source tooling and stuff which is a little bit you know crossing the line but that's one big difference I've noticed going from data bricks into this this world of tensorflow and kind of non Apache world but yeah back to this so like we use ml flow for the experiment tracking not just offline experiments but online experiments so we're constantly feeding ml flow you know which is basically just a rest endpoint we started to build this towards a project called model DB that was popular you know a few years ago that projects kind of gone dormant it hasn't caught up or stayed current with versions of psyche there's a lot of rough edges around the tensorflow support so versus ml flow which is very spark friendly obviously coming from data bricks and is very tensorflow friendly scikit-learn and friendly and are friendly as well there's a project called coop flow also we have some opinions on on coop flow there's a lot of Google's support going into it and you know Amazon is kind of a second or third thought there so coop flow is essentially a docker registry full of docker images that all kind of work with each other but and we pull in pieces that we need from coop flow but like coop flow is a much much broader thing that you have to buy into okay so yeah this is one of the things beautiful AI does is it recommends ways to lay things out and then yeah I don't know if I agree with the layout but here's have been on traction we have a community version I'm trying not to make this a sales pitch but at the same time you know these these pitch decks end up being this but we do have a strong github support we're just over a third you know like close to 3200 stars and the persona this is how VC people speak right the persona is the app developer which we're like very proud of because these these kind of projects tend to lean towards data scientists or the ml engineer and we're actually seeing all three of these personas using pipeline open source and community and of course the Enterprise version let's see so here's the problem you know said differently there's there's all these models being trained there's tens of these or you know hundreds of these training jobs happening here at Twitter here everywhere and then there's this kind of magical handoff and that's where you you know hand over your our model or Sai kit model and then someone rewrites it in Java C++ whatever but the data scientist kind of loses touch with how this model does online they know how it's doing offline because they've trained they validated but they don't really know how this model does online so there's a lot of like metrics just just being simply ignored and this whole disconnect thing so we're keeping things continuous the real world is continuous you know frauds happening all the time the second that you put these models out right like these models can get defrauded and start to decay right away and there's no visibility into this right now so if you use pipeline for example yeah even like the open source stuff you can push models out side by side and actually like send traffic in a shadow mode to different variations of your model right now we find it takes about one to three months to test any new model variant or just like any new model we also have been pulling people and we find that less than 5% of all models trained offline end up in production and it's either because they're like typically too slow for applications because these things are optimized for accuracy and not optimized for right like prediction throughput and that kind of thing right now most people either spin up a flask app in fact we registered no flask capcom and it goes right to pipeline that's pretty much like the main like competitor is these bespoke One flask apps that call into a psychotic all an AR model and you know like these things take rather upwards of 5 seconds per prediction and that's not the best way to go so for full transparency this this world has been like completely busted open by these more modern machine learning frameworks like tensorflow MX net these things have been designed with with online performance in mind and there's lots of systems-level things that that happen between the research lab and production right so even with like tensorflow it's just the tense well actually let's you you rightly do these things where you can fuse together layers you can change the model from a 32-bit model down to a 16-bit and make this like matrix multiplications much much faster there's also movement on the hardware side too there's things like the Google TPU there's also a company called Cerebrus right down the street down in Los Altos company out of London called Groff core that are creating 16-bit native right like TPU like things these are like deep learning specific chips and they're like very much targeting tensorflow this is how you know tensorflow is really going to be around for a while it's not just Google on the software side its these hardware manufacturers that are paying very very close attention one interesting thing too about these like hardware folks is that they're paying close attention to kubernetes as well if you've ever tried to get docker plus a GPU relic that alone will take up a couple spare weekends if you try to get kubernetes plus dr. plus a GPU that took us all of 2017 and so these guys are trying to make it easier to make these TP use and these deep learning specific chips work well with kubernetes which is very interesting ok yeah I can't have a pitch tag without a value prop so this so these actually come from some polls that we've been doing some of you probably got blasted by us like over the last couple months begging you to fill out these surveys because we use these to actually show the VCS that people are paying attention to this project but yes at a high level we're like able to actually generate models so when when you you have a single model for example bottom right you can't really see it but you'll have one model that you've trained and then we can actually generate multiple variations and these can be GPU specific variations TPU specific but these are all meant to serve predictions online right and there's a bunch of switches you can try that are like unlocking all these systems level things that I mentioned right like fusing these layers together dropping down to 16-bit dropping down to 8-bit in some cases people drop down to one bit neural network right like which is very interesting it shows you that if you have a neuron a whole bunch of neurons that can only be 0 or 1 it's still relevant the deeper you go right like you could still train something with a very deep neural network at very very low neuron precision right like numerical precision the benefit of that by the way from us some standpoint is each neuron that participates in these matrix multiplies the activations and all the fancy stuff happening there can now stay closer to the ALU right to the arithmetic logic unit so they're not getting if you have a 32-bit float being multiplied by another 32-bit float they could potentially end up back in the main memory and be flushed through all the caches if you have one bit by one bit it's like Tetris right they could stay super close to the ALU and then memory movements is very very fast oh and then also I try mention across all three clouds so we're actually right like pulling each model and spitting out and so some of our Met are a key part of the metrics that are being generated our cost per prediction and so because we know the instance type that these like models end up on you know these are just docker containers running within kubernetes we can estimate the cost per prediction if we lose our spot instances throughout the day from Amazon now the next cheapest cloud might be like Azure which actually it's almost never Azure because yeah is most expensive but will start to send traffic over to Google or to Azure then once we get those bought instances back we can then shift traffic back to Amazon so we can kind of do cross cloud multi-cloud cost savings which we do have people using all right this is all go to market blah blah blah let's see let me pull up the demo so if you go to community pipeline at AI you could try this this actually uses our runtime from the open source so this is like a nice way to if you go to a quick start pipeline ai that'll kind of navigate you through the different options here you can start with the open source and then go bounce into community we have notebooks and all the usual stuff we have streams here now too so you can create a stream you can send data in you can hook up and do streaming predictions right that's kind of the most obvious continuous I like predictions but you can also wire it up so that you're doing and training these models right off of these streams as well too so yes I have this ring in another tab so there's a bunch of notebooks in here that you could play with how much time okay five minutes so there's a whole ton of samples here yeah anything that's in white we have the sample right now all the other ones we're working on the samples we support the runs I'm we just don't have a coherent sample that matches the rest we've chosen em NIST which is the classic hand written digit the hello world of deep learning mainly because it's been implemented in a lot of different languages and ml frameworks so here's M&S that's been written one through three was at seven different ways so just straight up Python in spark so this is showing JVM runtime you know Python runtime XG boost Karos tensorflow so really like you can if you think of this at this point as kind of a you know private Kaggle right like specific kaggle specific to your enterprise is just kind of one way to think about it but it's kind of an extreme sample where there's seven different implementations of the same model written across all different frameworks and then here we go and deploy them I think I was able to get five of them deployed before running out of time just right before this we've routed traffic to these so 20% going to each we could also run them in shadow mode so I can click this and that'll put it in shadow mode it'll still take live traffic it but that traffic won't be returned to the user right the other four actually returned to the user one of the other four but we're still sending traffic to anything that's in shadow mode and yeah this comes from right like Netflix background and built on some good Netflix like open source here like you can scale it up you know one we have it limited to ten containers for community edition you could also set auto scale if you want to create your own function to define for for your scale metrics you can click new and that'll let you define your own function this is all server Louis we don't talk about that that's just implementation stuff but you can create your own function to do pretty much anything even the auto routing so here I could drop this down and say across all of these model variants or across all these different clouds right Li try to minimize cost and so that's really going to trigger off of that cost per prediction or we just want to find the most blazing-fast like the highest throughput version and make decisions on where you're going to route traffic to minutes or you can create your own so here we can do Oh No okay yeah there we go so here this is just kind of a sample input box this is a JSON representation of an image I can click predict and see the different yes looks like one of them still starting up the different implementations back here and then yeah here's a nice little view so we could see these things lighting up and telling us you know throughput numbers and response here you can pick any language grab the snippet of code that will make the call into this service and then this is something cool so this is actually showing side by side all you know was five six versions of this model and they're all predicting the same value which is good these are all the potential values here this is the same input this is just some hash you'll have to worry about actually what that is it's a hash of the contents of the image so that we can group all the different model variants and and what they're the like physical prediction coming out of each of these variants for the same image if there was any outliers here we could go click on it and figure out which variants are giving us one thing we didn't get a chance to talk about was some of the ensemble aware we could actually combine variants or even go across projects and say I want you know M NIST and yeah this is like a little like github star thing that we have when you start the repo it sends it to a slack channel okay 30 seconds so yeah we could actually ensemble across all these different variations you can upload a model you look at all the metrics in the world streams and was everything I was going to show oh the explained ability so like I said at the very beginning you can inject value add now to these predictions so we and so this actually has two different like algorithms for explain ability this is showing the input this is showing what the actual prediction was and which pixels led to the prediction and which ones didn't and if you have more algorithms that that show up right like day two after you've deployed this you would just plop it in and now you have a third algorithm that's actually sort of intercepting all of these predictions so that's it hopefully that that makes sense [Applause] yeah yeah I mean that's a business decision right we're so like we're the platform that will let you figure out or that will let you implement the strategy that you want right it's it's a lay common question with the online training I think is what you're getting at and how sensitive are you to that because that could potentially skew the model towards certain trends so yeah we actually encourage a discrete continuous kind of thing so all of our continuous stuff flows through this and it shows up as a new variant and it could be shadowed so we're not actually making changes to the model live that's something we're like very much against because we saw this caused problems at you know like larger scale like Netflix if you're making configuration changes or any kind of change live in production you don't want that right there's always going to be a problem so the hyper parameter tuning part where we're combining online metrics which which would be right like relative performance between these models and we're combining it with the on with the offline metrics as well to to give a 360 yeah that's the key that way we could say when you made this change offline this is what happened online and like this doesn't exist today yeah so yeah yeah each one yeah exactly yeah and that if you go to the QuickStart and follow the docker thing like you'll see exactly what's happening the short answer is there's kind of a base docker image the development version which is the open source version has all of the runtimes in there so it has tensorflow it has it's got Python it has side kid it you know has like everything there so the development version we extend that base image and burn in the model and then it's accessible via Kafka stream so you can make predictions so that that's same so you have to implement the invoke or the you know predict method and that's the contract that's the only contract between the framework and your model just ask to have like that method called predict yeah yeah yeah yeah so yes everything has to have yeah like there's certain admin endpoints like health checks and things like that that are just rest endpoints the most basic is a like rest endpoint that's listening at slash invoke correct yeah yeah yeah that's that's one implementation there's also G RPC n points there's also Kafka streaming consumer as well and the developer version has all of them in there so you can kind of mix and match the the like real version the like product version has separate optimize runtimes for each kind and then it's just a different way that we do the sub classing of these docker images yeah good question yeah awesome thank you