Scale By The Bay 2020: Catherine Nelson & Hannes Hapke, Automate the boring ML stuff with pipelines
[Music] hi everyone and welcome with apologies to al swiger to wrote the classic book automate the boring stuff with python today we're going to talk about automating the boring things in machine learning production with pipelines i'm as nicholas mentioned i'm a senior data scientist at conco labs i work in a small team that drives innovation at sap concur which is sap's business travel and expense tool we evaluate evaluate new technologies and then we recommend them to the rest of the country company i'm also a co-organizer of seattle pi ladies supporting ladies who code in python o'reilly author and google developer expert in ml hi i'm hannes i work i have a great chance to work with catherine at concur labs as nicholas mentioned i'm a ge for machine learning and i've had the chance to co-author those two machine learning publications talking about the machine learning publications this talk is about how to build machine learnings machine learning pipelines and we recently published this wonderful publication with alrighty which is now out in in print and also available as a digital version if you want to find more information um take a look at building machine learningpipelines.com so a place where we really want to think about automation is when we have some successful machine learning model and we're integrating it into a real-world system and this is a different situation to experimenting with model architectures in production we want to automate as much as possible and standardize it and we want as much as possible to be reproducible if we're looking at a production machine learning system a lot of the time everyone just talks about the model code but you might have seen this picture before the machine learning code is just the tip of the iceberg there's a lot more code that we need to turn the machine learning model into a production system we need to validate the data going in monitor the models we need serving infrastructure and so on next slide we believe that we believe there's a few really key things to consider when you're moving from an experiment to a production system some things that we think you should focus on are really integrating that model into the real world scenario we believe you should have aim to have a machine learning system that is reliable scales and is easy to retrain on new data and this helps to reduce the burden on the data scientist or whoever is managing that system and helps with reproducibility but right now what happens to most to trained models in most situations most models don't get deployed this is an old tweet from over two years ago but it's still true this is partly because many teams will run a lot of experiments and it's experiment expected that not everything works out but even when the experiment is is a success there's still a lot of barriers to getting the model into production just getting deployed is not the end of the battle there's other problems that we detail here data drift training serving skew complex retraining problems with latency so there's all kinds of boring things involved in production machine learning systems that we would like to automate maintaining glue code is boring having to manually retrain your model is not what you want to spend your time doing and dealing with training serving skew when your feature engineering doesn't match what your model expects is really dull i'll hand over thomas now and he's going to start to explain a solution to these problems before we start talking about the implementation of how we can express our machine learning pipelines let's take a look at the machine learning machine learning model life cycle and with that we mean sort of a recurring cycle where we take data we validate the data we pre-process it in our feature engineering train the machine learning model and then take the trained model and analyze it and validate it and then deploy that machine learning model before we capture more data to re restart that cycle as you can imagine those machine learning systems are heavily entangled so for example when we ingest our data we can generate a schema that schema would be used by let's say the training component or the validation component and this is where tensorflow extended comes in and helps us with tensorflow extended that is a it's a tensorflow library or it's a set of libraries and it allows us to abstract our pipelines in a python pythonic way so we don't have to configure our yaml files or other bash scripts or things like this we can stay in our python ecosystem it provides us a wide range of components and we will talk about this in a second here but it's also highly flexible so we can define our pipelines we reuse existing components but if we want to extend them we can easily do that as well and then once we have defined our machine learning pipelines we can orchestrate them on a variety of platforms including quickville pipelines apache beam or battery airflow so how does the how does those pipelines or pipeline components tie together so in this case for example we have our blue boxes which is representing our machine learning life cycle and we can represent those boxes with machine learning components or pipeline components and then those components are being executed on our orchestration platforms and they also integrate with a metadata store called the metadata or mlnd if we take a closer look into tensorflow extended we can see that tensorflow extended is not just providing us components for the individual steps of those pipeline processes but it also provides a standalone libraries for example this tensorflow data validation library which we can use with non-tensorflow models for example with a pytorch model or a scigirlarm project as i said before once we have defined our pipelines we can then automate our orchestration so for example if you define a pipeline and you want to re-update a model or update a model every time your data arrives or maybe at a given time frame you can orchestrate those pipelines through tools like apache airflow or kubeflow pipelines on top of it there's also the option to run those pipelines in jupyter notebooks but in this case you as a human as a data scientist you're the orchestrator and you execute the individual components so now let's sorry yeah so let's get into the details of some examples of how to automate machine learning life cycles with tensorflow extended the first step in our pipeline is to ingest the data the data can come from a variety of sources from a simple csv file to a cloud-based storage system and at this point all we want to do is split the data into the training and evaluation sets and that's what these lines of code define here we have a three to one split between training and evaluation we have some external input data source and we pass in the path to that data and we've already defined our outputs above the next step in the pipeline is data validation this is a super important step because at this point the pipeline stops if our inputs have changed outside what we're expecting so the pipeline doesn't continue to the time consuming training step the first part of this validation step is to generate some summary statistics on their data set and this can be the distribution between different categories or the mean of the numeric feature the pipeline component also generates a schema which defines the acceptable values for these statistics it's generated automatically based on the training set and then future retraining sets are checked against it we can also edit this schema manually if we want to enforce things like the proportion of missing data that's acceptable so if a new batch of training data reaches this point and it has more missing data than we specify in the schema the pipeline stops here and the model doesn't get trained and this means that a model trained on incomplete data would never get deployed one of the best things about tfx pipelines is the visualization tools that plug into them directly so we can visualize here the statistics of the data that's going into our pipeline we can dig into each individual feature and we can check that it's what we expect now let's take a look how we want to do the feature engineering we can perform the feature engineering with tensorflow extended through a library called tensorflow transform and in this case we can consistently express our pre-processing steps um we create powerful pre-processing steps which we can then also execute on um distributed platforms like apache beam apache flink or google cloud's data flow and then we can output not just the transform data sets but also a consistent pre-processing graph so in this case for example we define our pre-processing steps as tensorflow operations and then we can execute those on our given data set which we have ingested in the previous components once we have performed the feature engineering we can then train our machine learning model the training is very similar to what you have seen previously and with like training keras models or tensorflow models but we have the opportunity here to ingest the pre-processing steps we can learn we can use the the transformations and use them as a starting point for our model we can then use the similar concept as we normally do in our keras training steps for example here we use a mirrored strategy if we want to distribute our model training across different models sorry across different gpus we can then compile our model which we have here abstracted in the function get model and then when we define our model in this function we use the transform steps or the output of the transform process as the input to the model and then we train our model as we know it with like model.fit and save it as usual with model.safe and this is all defined in that com in that component object called trainer after the model is trained and saved the next step in our pipeline is to analyze the model's performance and the evaluator component in tfx lets us define our own criteria for what is acceptable performance and we can also compare the new model with previous runs we pass in the details of the new model to the evaluator component and we define the metrics that we're interested in we can choose any metrics that exist in tensorflow maybe we choose precision here and we set the criteria that would mean that our model moves forwards and gets gets deployed so for example we could say that for the model to get deployed we want the precision to increase by two percent but if it doesn't increase by this much we don't want to change the model if our model does meet this criteria it automatically gets deployed we can also slice our metrics so this means that we can divide the metrics by different categories within a single feature so for example if our data has a country feature we can get the accuracy for each different country and we pass all these things the evaluation set the model a previous model that we'd like to compare and the metrics and slicing details to the evaluator component and if everything passes it moves on to the next step but this is a point where actually you shouldn't automate all the things automation can mean that harmful impacts get missed a high profile example of an unfair model in the real world was highlighted by the gender shades project in 2018. they found that the accuracy of commercial gender prediction algorithms was really different across different groups and the worst performance was for darker skinned females the companies did improve these models after this study highlighted the problems with them but we should aim to spot these problems before we deploy our models tensorflow model analysis is another visualization tool that plugs into a tfx pipeline and this gives us a visualization from the evaluator component you can see that the metrics the accuracy and number of false negatives are sliced by the different product categories in this feature and being easily able to do this to slice our analyses on these different categories is absolutely crucial for checking whether our model's predictions are fair the final component in our pipeline is the pusher component if everything has passed our analysis checks and we're satisfied there aren't any harmful impacts this component moves our saved model to any location we like and this includes the preprocessing steps along with it and then it can get picked up by a choice of deployment option so once we have defined our component pipeline steps we can then execute the pipelines through the orchestration platforms which we mentioned earlier so you can define the pipelines by simply defining a list of your components then you define the configuration of your pipeline itself so for example you want to point to a given metadata store where you want to store the information from each component and so that follow-up downstream components can use the the meta information and then you can execute the the kubeflow pipelines or the the runners of those particular pipelines uh you want to execute runners for airflow and b will execute immediately and runners for kubeflow pipelines will convert this entire python definition into an argo file which we can then run with kubeflow pipelines behind the scenes is something magical happening all the components are talking to a metadata store and the metadata of our machine learning pipelines will change everything as an example we can then power applications like this like google pipelines we can visualize our um our graphs of our pipelines and see the progress across the different components furthermore we can schedule model updates so for example as i mentioned earlier we can then say every monday morning we want to update the data or based on a given trigger saying a new data set arrives in a cloud storage bucket but we can also go back and investigate the models after they are deployed let's say three months down the line somebody complains about a prediction we generated with a model which we try to train you can go back through the model lineage and track all the artifacts which had an input to the model to the trade model which we deployed so you can see who signed off on the model you would be able to see what was the raw data set what were the transformation steps you can even pull out the old information about the model validation or the data validation steps where you can see the schema and the data statistics so why do we want to automate things with tfx pipelines there's a number of advantages you get the metadata in one place you can deploy your model continuously you get this amazing audit trail you can get a human review as part of the loop you get automatically um conversion of your models to tensorflow lite and tensorflow.js out of the pipeline you can include model tuning in the pipeline and you can plan you can branch your pipelines to produce these tf tf light models as well if this was also fast we highly recommend the publication which we published with o'reilly where we dedicate one chapter for each component and then also explain to you how you can extend those pipelines with your custom components we touch on further topics like machine learning privacy or how to create feedback loops and to give you a holistic overview of like how do you how you can bring your machine learning models into production if you have questions feel free to reach out to us via twitter thank you thank you