SBTB 2019: Chris Fregly, End-to-End ML Pipelines with KubeFlow and TensorFlow Extended (TFX)
[Music] mmm thanks for coming yeah always good to be here see some old friends people in town [Music] South African guy over there Nick speaking in in an hour so yeah check out his talk okay so as Mike pointed out I actually have one small update to my current job situation we just joined the Amazon folks so Amazon Web Services doing currently doing developer relations and taking the pipeline AI project and integrating it with sage maker so if you want to talk about that after we could talk about it but yeah pretty cool stuff so the Amazon guys have done a really good job with sage maker over the last couple years in the beginning it was a little shaky but things are are picking up so basically lots of the pipeline customers we're asking about sage maker and wanting to integrate so decided to join forces I have about a hundred and six slides that I need to cover in half the time that I thought I had a classic problem that I always seem to have so we're gonna cover lots of technologies and tools and I'm gonna give you guys my recommendations for what you should use or you know depending on where your you're at in your machine learning the like complexity of your pipelines and also these slides are all posted if you go to pipeline AI you can click there's a resources button and you can get to all the stuff that's going on there okay that's so quick background Netflix data bricks paid my dues they're moved to the Bay Area about ten years ago also have this meetup in this area which some of you might have found out about this conference because of the meetup it's called the advanced come flow meetup it used to be the advance and tensorflow meetup is now advanced coop flow and I'll explain that here in a bit so here's the meetup got about twelve thousand members we've been around for about four and a half years hundreds of videos the globe lots of different talks you know four and a half years in technology terms as a lot of technologies and a lot of shift there's a free upcoming workshop actually for you guys thanks to Amazon bill that was one of the reasons also I wanted to join forces was I get access now to lots of compute and lots of eks clusters elastica kubernetes clusters and we'll be installing come flow and getting all this running so basically it's this eight-hour version of what I'll be showing you guys today in 30 minutes so sign up for that go to workshop that pipeline dot AI it's free it's my little holiday gift to you all okay just I curiosity who is okay with command-line Python docker most say okay so I'm alternate yeah hopefully if you're a command line you're also docker yeah how about Jupiter notebooks okay statistics oh yeah yeah all right start off with a little joke here that's another joke it's actually real life while going for funding over the last couple years we run into this quite a bit where these VCS would ask how many github stars does your project have I would say you know X number and they would say ok we can give you Y in seed funding and all the numbers ended up being about $1,500 per github star just always seem to work that way it didn't seem to be you know from our perspective that's what it was so please start this repo and then maybe when I'm done with my Amazon tenure we can go back and get some funding again okay don't look at this just yet because I'll be doing that for the demo but I'm going to show you that link at the very end so the context here and as I've gotten older and wiser and and hair on my ear yeah I don't know if you guys start to get perón weird places but I've started to pay more attention to what I'm actually doing and what's the business value of what I'm working on I used to do and focus on tools and I would geek out and I'd spend all the weekends getting GPUs running and I would end up Monday morning and I would have basically nothing from like any business value so I do want to talk about some of the kind of higher-level problems that we're trying to solve here and this is kind of a depressing slide but just know that when you push your model into production it is already out of date it most likely has bias number two based on how you've collected your data right depending on how you've you've collected your data how you've managed your data how you've sampled your data during model training your model is likely broken or could very likely be broken you will notice it more when it's broke and then when it's not broken so it always seems like it is this could be blowing the heap this could be we have a provision like enough resources and your lambda functions or your eks containers or docker containers your model will instantly start getting hacked the second that you can pass an input and look at the output you can reverse engineer and learn that model this is made worse because we're all using the same models right we've we've all been told over the last couple years don't like reinvent the wheel grab somebody else's model take a model from Google take birth to take Riley imagenet the problem with that is these models are public and all the weights are known the architecture is known and very easy to back into so we'll have a couple slides at the very end three or four slides and a notebook at the very end talking about data privacy and specifically with respect to machine learning and how to train your models with something called differential privacy which sounds as scary as it is if you start looking at the math also your models are probably slow or they could always be faster you're probably not paying attention to the hardware or you like haven't gotten around to paying attention to the hardware that you're deploying on there's lots of different hardware specific optimizations so the concept here is called hyper parameter tuning you can actually type a parameter tune on your models after write like they've been trained there's certain like one-time characteristics that you could try out that are specific to the hardware that you're deploying to if it's GPUs is the most obvious of its small devices so we have a couple slides and a like notebook or two on post training optimizations to make your models faster alright so a lot of these slides are actually from the workshops or just kind of ignore some of the setup stuff but at a high level this is what we'll be doing we have a curve an ADIZ cluster so during the workshop everyone gets their own we're gonna use tf-x tensorflow extended is actually what google had in mind with tensorflow but they did they didn't really tell us this that they've been using something called tf-x internally for many many years within Google they've just been slowly sort of open sourcing pieces so the first piece of course was tensorflow core the second piece was actually tensorflow serving which is part of the larger ecosystem the ability to take your train model and deploy it then about two years ago or about a year and a half ago they released these models called tensor flow data validation tensor flow model validation and like these were just kind of Python libraries there was this project called facets that could visualize and show your models after they've been trained you can kind of slice and dice all of this was really part of tf-x but they were just kind of slow to to show us the whole picture so this has become and so this is literally pip install like tf-x okay we'll also talk about I'll show an air flow pipeline I'm sure quite a lot of us are using go flow well if you're in Scotland probably not but if you're in Python land you probably are will also show come flow will show how to do hyper parameter tuning with coop flow will also talk about ml flow which i think is on the next slide but this number 500 point is probably my favorite which is taking a notebook and deploying it onto a cluster so my time at data breaks I didn't like really get why this was such a big deal but it's a huge deal the ability to take for example the data break spark notebook and have that run as a job on a cluster is the whole reason data bricks exists right and that future took about probably two years to stabilize right they had to build their own notebook they had to build their own scheduler it's actually not using Mesa it's not using kubernetes and yeah lots goes on so fast forward you know three or four years we're finally just now starting to see the ability with coop flow and with things like ml flow to actually deploy a notebook onto a cluster and then run a job and then we'll also cover an oil paper mill yeah who here has heard a paper mill yeah so paper mill is pretty cool you could pass in a jupiter notebook as a parameter or as an input pass in parameters and then run that notebook as a cron job okay so like Netflix uses paper mill pretty heavily I mentioned tensor flow privacy which is a first class project off of tensor flow at work like github this is using differential privacy we'll also hopefully have time for the model serving and some quick a/b testing I got it set up so I could just run through and just kind of show you guys and then if you want to offline you could do it on your own and then model optimization this is taking a model quantizing it down making it smaller so that the actual influence the actual predictions are run much quicker alright let's get through these white EFX where I can flow so let me set up a little bit more context here I've specified six seven different systems here zero index of course these systems are really the whole pipeline right and this is one way to look at it you know and I sort of Demark demarcate de mark mark separate out each system sort of like it's basically by the hard way that that is like needed for each system system zero not a lot of people think about this system it's called feature store there's more advanced shops are actually building features store there's a couple startups up in San Francisco or across San Francisco to the West that are focused on feature store it's really when you start getting beyond just converting string or you know floats into insert like doing small-time feature engineering but actually creating word embeddings and complex features and the ability to share these across different teams so the sort of common machine learning user probably doesn't even think about this or they're just storing them on disk somewhere and then you know sending a slack message to some other team to to point to that same directory but feature store is a much larger like project much larger system so keep that in mind there's not a whole lot out there there's a couple meetups that have covered it there's a project out of Pinterest that's focused on it I believe I think like Airbnb folks have a project that they might open source also system one is you know this sort of broader data ingest you know this is trying to get CSVs and then converting them in to like park' something like that Avro there's the the second one so so there's data ingest and then there's analyze your data and that's really where tf-x comes in is that you could take a look at all of your data as it's coming in you could stream through all the data the details they use Apache beam which is a big mystery to all of us except for people at Google and can actually extract out the schema I can look for any anomalies you can tell you if there's certain certain features that are just missing in this current batch of data that you're pulling in this happens all the time when you're scraping application logs you're very dependent on upstream what's happening with these application teams you then do data transformation this is all classic ETL and then tf-x also helps with data validation so the DV so data analysis which is kind of looking at the raw features transforming and then doing data validation system 6 is kind of a cross-cutting I didn't want to forget it it is a pretty major system that's really where stage maker and a lot of these folks come in system 2 is more ad hoc so this is your Jupiter notebook this is weird right like a lot of us tend to end system 3 is really you know training the model validating the model getting a little bit more complex if you think of like a concentric you know Russian doll thing is what I always say think of it as getting broader and broader larger and larger Hardware system for this is where you're doing the full distributed training on all the machines that you have doing hyperplane your tuning system 5 we're pushing the model out we're doing a/b testing and that's the full pipeline and of course we're like learning from the model in production and the flywheel continues so another idea is that we can detect bad features coming in so if you think of it like a funnel you know classic funnel where we are always trying to stop this pipeline for some reason the you know funnel gets more and more narrow there's there's more and more reasons or yes we should get you know like way more confident as we get through these systems so we might kill the pipeline because the data is bad or we're not expecting it or this it's you know violating some schema we may you know have bad hyper parameters things aren't converging so there's like lots of different reasons before we're actually going to put the model into production and so tf-x helps with this coup flow helps you automate it yeah this will help you stop early and yeah lots of other stuff on the left there composability so cool flow really comes or most of them yes most enterprises right now seem to be very excited about Kuhn flow because they either just have sage maker they just have Azure ml they just have Google Cloud ml they don't want lock-in they want the safety to build pipelines and then be able to port those pipelines from one cloud provider to another to on brim and so coop flow opens that up which is why I'm excited to work on ku flow at Amazon so here's T FX if you think of T FX as a set of libraries you're not too far off it's just Python libraries it's also kind of a reference set of components that you could use to build your pipelines so there's a little you know there's kind of a mapping between these two but yeah so let's just focus on this one here but there's feature load feature analyze feature transform model train model evaluate and the model deploy so it's all pretty much the same as the previous one the names here those things that are in orange are physically the out of box tf-x components that you can use to build this pipeline those tabs up on top show that you could actually use T FX with air flow it's just Python libraries you could use T FX with code flow and I've even seen people use T FX and build these ml pipelines with Jenkins write some very large machine learning shops that really all they needed was to have you know something like some component taking action there's inputs outputs and then some like future downstream component be able to pick up the output from the previous one so yeah Jenkins works ok people are moving to come for because it is a set of standards also note that you don't need to use tf-x with chrome flow you could completely use T FX on your own you would kind of have to coordinate things on your own you could also use ku flow without T FX I would recommend starting with the simplest thing you can and just add things in as you need them most shops don't really need at least straight out of a box they don't really need to be doing you know the constant skew detection they probably should be doing more bias detection but if you're just starting off just trying to get a model trained don't start here but like just to give you guys the sort of lay of the land so the example Jen that term Jen is a it's a Python thing generator so the example Jen when you talk about machine learning that term example does not mean what you think it means it means like a row of training data sample right like example I've had multiple conversations with people or I'm talking about a row of training data and they're talking about a sample piece of code and it's not the same thing and somehow we ended up talking for like five or ten minutes so there's statistics Jen this is actually capturing all of the the statistics standard deviation norms all that stuff and by taking one big pass right like using Apache beam for all the features in your dataset we're then validating those features where the schema generation is super important we can use that schema later on so it's very important to get that right if you're familiar with spark spark actually has or has very good schema generation also can take CSB s that are all over the place and then try to make sense of it so training is probably the simplest part of this whole pipeline and then we're going to like validate the model which you can sort out of the box just like just have it pass through or just do some simple cursory checks is this model you know 1% better in in terms of f1 score than the previous then then pass it along and then push it into production I've seen people actually use slack before the model validator where they will suggest that a model is good or not and post a message to slack and have the team actually click through and then validate the model manually and then push it into production with /yes or slash deploy something like that slash commands all right so this is airflow implementing those same components and this is super easy to get working out the box yeah we have the Lake Community Edition for pipeline just for you so you can go out there check that out and then here's a similar pipeline with slightly different names this is how you do it in coop flow this is what it looks like I'll show you guys the code here in a sec the other one I haven't talked about this too much but this is ml flow if if I were you and starting completely from scratch and I do not have any like data bricks options or anything like that so I have no interest in this project succeeding or not succeeding in any way like financially it's a really good project it's very very simple it does exactly what you want it to do which is track and experiment train your model like capture those hypo parameters and then capture the metrics and post them to some single tracking server so very powerful has really good visualizations out of the box very simple Python API supports kalos sports spark Java and our sports tensorflow scikit-learn Karos all the big ones PI torch all that stuff so check out yeah I'm out flow I have a couple notebooks on it that will skim through here it a bit hyper parameter tuning this is actually a slightly older UI from a slightly older version of coop flow it's been updated a little bit but this was pulled from a project called model DB that came out of MIT yeah so that project got a little crusty got a little long in the tooth we've got some hair on its ear too as well I think and so here what we're doing is showing this nice UI or like somewhat nice UI that will let me actually generate the yamo that's needed to run on my kubernetes cluster so I jumped ahead a little bit like just know that you can with one click or one change in your configuration change your hyper parameter tuning strategy and supports all the like big ones hyper band Bayesian optimization and a random search grid search so this is all part of the kupo umbrella this project was pulled out a model DB basically copied and pasted into this project called khatib which is a first-class project off of coop low github root for the traffic routing we're using something called Sto it's the kubernetes conference next week so this will be I think third or fourth year for this conference and this project gets more and more talks as each and every time I go it's it like uses something called envoy which is by the lyft folks so although you know yeah there should be lots of credit going to the lifts people it's a way to route traffic so from a machine learning standpoint or like micro service standpoint you want to deploy your model and point five percent traffic or mirror traffic to a write like canary version of your model so it does traffic routing takes care of all that so with just one little change to your configuration you can point em percent 15 percent you could like trigger off of some some header that's in the HTTP header and send to a completely different cluster yes all that stuff's right now the state of coop flow and sto and kubernetes you're really doing a lot of this sort of manual editing yourself of these like configuration files there's tooling that's been coming along under coop flow that that helps out all right I think I'm gonna skip the internals of tf-x I do want to highlight this this is data validation this is what's actually looking if there's a high number of zeros if there are certain features that whose scale is much much larger than other features then those features need to be are like standardized and analyzed so that they don't dominate during the training process let's see we're also looking at distributions here okay yeah TFTs is pretty basic stuff all of these like tents are these like tf-x components you are like pretty much always like going in and customizing some some function you know the sort of template pattern kind of thing I'm gonna skip ahead here yeah how much time do I have six minutes okay just how you getting ready to give me the five minute here's the model analysis so this is very important so you do need T FX for this you would use air flow and coop flow to rather automate the process here but you you have to use T FX and specifically model analysis library that plugs into your jupiter notebook and shows you the distribution across different slices of your data so this is how you would sort of visually detect by us so you'd see certain parts of your dataset are more represented than others if this was gender and age you know this is bad if it's not even this is actually the chicago taxi data set trip data set so there's certain times of day where this model is more confident more accurate these are things we have to keep an eye on another topic is is metadata so in like the upper right as we show all those those like components all of these things have to store data to be picked up by the next lay component this is all stuff we built probably probably multiple times over the last you know 15 20 years whatever but yeah just know that all this stuff comes around again can flows trying to tackle this problem as well NT FX has some opinions on how to store that data all right let me jump into some hands-on stuff here so this is what can flow looks like we have five minutes right okay okay ten minutes yeah so lucky you guys yeah okay Jupiter notebooks we have airflow pipelines control pipelines the ml flow pipelines so we've made some modifications because some of these projects like air flow and like ml flow are actually not part of coop flow but they're but they do have kubernetes versions and because cupola is all kubernetes native very easy to deploy as tensorflow distributed so yeah let me just kind of go through here this is our actual kubernetes cluster right now this is what it looks like we have air flow reading the central dashboard you know these are pods that are running inside of this cluster these are basically docker containers okay that's like what the 1/1 is it means one docker container is ready out of one possible container kubernetes is you know handling all of this resource allocation and the action of notebook is this it's this community - zero so and there's a lot of other stuff here's a pipeline that I just ran and I think somebody else is probably running right now this is the hyper parameter of stuff that's backing all those different strategies like random search grid search and here I'm gonna describe this pod so this is a you know kubernetes command line thing here the most interst thing you should do when you describe something in in kubernetes land is search for the word image because kubernetes or because can flow is kubernetes native that also means it's docker native and it is there's a docker image that's running somewhere that's powering this Jupiter notebook and so here it is we've actually built one with a bunch of that like tf-x libraries pre-installed everything first thing you should do so if you're if you're handed a you know kubernetes cluster I'm just out of nowhere that's the first thing you should do the second thing you could do is do a QC TL logs if you're familiar with docker which a couple of you were you can actually go in and some of this should look familiar this is what the jupiter notebook is logging when you run it locally except now it's running it inside this cluster and it follows the same like commands like docker logs yeah Coop's ETL logs all right so that's the cluster here's some of the tf-x stuff I'm think I'm gonna skip over this just get more of the visually interesting stuff alright and here is airflow so this is live this is actually this is the like airflow that these are each of the components here we can go look at the code click on code this is one big Python file now so it doesn't have to be but just kind of how it ended up it's using tf-x libraries so if you're familiar with airflow it like basically has its own library that you could set up these graphs of execution and they end up looking like this here I click on the graph so that that one single Python file has built this and now we can go through and just like keep running this we could fail if any of these fails we could fail the pipeline we can notify people as kubernetes support s3 support Google Cloud Storage all that all right let me show you guys the equivalent on the cupola side it was this first one so here's a run so this is live this was if I click into each of these here I should be able so each of these components like I said so Kuhn flow is kubernetes native it's its biggest strength and its biggest like weakness in my opinion right like depending on who you are it's for though for the ops folks in the room it's a weakness for the machine learning people in the room because we always have to think about docker images right like versus just Python snippets of code so I'm clicking on this this this one node or like component in this queue flow pipeline is a dock or is a Koopa Polident and therefore a docker image was created here's the inputs here's the logs is actually if you did coop CTL logs on this pod this is what you would see and the other cool thing is that you could build these like custom artifacts or like custom write like viewers of like different artifacts I keep pulling out the wrong one here so this is a training node I'm actually training this deep neural network I could actually spin up but yes I could spin up tensor board here and then view it directly within the UI here which is kind of cool I could also take a look at here's the ROC curve here's confusion matrix supports only binary right now and lots of other cool things so this is kind of the groundwork for like more and more visualizations the other thing is I could actually compare these runs so right before the talk I ran two runs and here I could see these metrics now coop flow right now to me is not the best tool to rather run these jobs and then compare these metrics it's a little bit weak it's getting better I actually like the ml flow stuff so this is really the best I could do here I could kind of highlight two runs and then sort of you know I could sort I think over here okay maybe not cannot sort by that column so that's not even good but I can just sort of side by side look at these like parameters being passed in let me show you guys ml flow I mean here's the hyper parameter stuff here so talk about that in a sec five-time ml flu so the code here it's super lightweight let me show you yes all it is is an import for the ml flow so you do pip and then install ml flow this is a scikit-learn model and yeah ml flow has deed of support for scikit-learn I define my tracking server because this is on a cluster with a shared file system I'm actually taking advantage of that all these pods including my notebook could access the same shared file system I'm not going to go over the wire or anything like that I'm just going to dump it in the file system create the experiments that's just a logical grouping of these runs and then just do my normal thing and then here I'm using a Python context manager with something so all the statements that are indented will now run within that context of the ml flow start run here I'm like logging the hyper parameters that are used for this run I'm logging the metrics that are used and they end up showing up nice and clean over here and I could compare I could also do parallel coordinate plot which are some of my favorite so if I'm trying to optimize if I'm trying to find the best type of parameters and trying to optimize root mean squared error or something like that I could plot all these and then back into the best type of parameters okay I think here's papermill so here's where I'm actually passing in a jupiter notebook as an input passing parameters I'm saying take that notebook and run it and slam in the values 0.6 and 0.14 these two parameters it will actually render that like notebook and then save it out if you do any visualizations if you're doing matplotlib or Seabourn those notebooks actually get rendered and you can inspect them later so let's take a look at one of the ones that was rendered two minutes okay and yeah so these values were actually slammed in place and take over it's a super simple project it's used heavily by Netflix it's you know pretty powerful ok I do want to end I think on the privacy stuff I think personally is probably the most interesting so let me get to that so tense phil has this thing called or it's a library called tensorflow privacy it's using differential privacy beneath the covers so right like what's happening here is we're trying to we are adding noise right like during the actual training process to to mask each individual row that's kind of the high level so what we're doing is if you you know have the patterns you're you're still able to capture the patterns of your data but not able to get back to specific rows in that data set so of course this is very important for you know polls and voting and if you release a model and now you can pass in someone's a person's name and see their vote if you have super high confidence if it's 95 percent confident that that that vote is yes or Trump or you know Hillary whatever that's not good because that means that the model has likely seen that data okay so it's a little bit counter intuitive because right like we always want high confidence with our predictions but we don't want too high confidence because then it's very easy to hack that model and back into so think of it like bloom the input data yeah and so there's obviously trade-offs sorry yeah this came out of there was the tensorflow world I think one or two weeks ago and I was taking screenshots as as fast as I could so I can capture this but there are trade-offs of course with accuracy but and so like differential privacy has this concept called a privacy budget okay it's time you can you choose the budget that you're willing to allocate to privacy the trade-off of course is accuracy you could also use privacy to help with explain ability also because you're kind of this like rising tide of confidence okay so check out TF privacy and project called KF serving like coup flow serving also like lots of good integration with core fauna Prometheus and then there's a model optimization so I'll end it there's the link for community edition so that's actually what I'm running on right now if you guys want to go check it out and then yeah thank you I hope to see you guys December 20th online [Applause] [Music]