Scale By The Bay 2018: Aaron Davidson, MLflow: An open platform to simplify...
Recording: Scale By The Bay 2018: Aaron Davidson, MLflow: An open platform to simplify...
and also for braving the weather to get here I know we're all on so that was very brave of everyone so yep today I'll be talking about accelerating production machine learning within although so as we know machine learning development is complicated to give a sense for this a typical machine learning pipeline looks like something like this you take a raw data you want to eat yell it or feature eyes it this is data prep then you want to do some sort of training to produce a model and you want to play that model to production you know score it produce the rest api serving layer run it through spark that kind of thing and then from and then you might as you get new data i don't want to reiterate this process and do it again there's a lot of different frameworks for each of these components so for the raw data you might have any sort of storage system as three hub back whatever kafka data prep there's a lot of tools out there as well and so both for like ETL as well as for the creating application of feature engineering training wise there's also a ton of different tools so from you know scikit-learn pi torch tensorflow all these are things at your fingertips and then deployment you might go for kubernetes or spark or azure ml or sage maker these kind of things you might deploy to any of these and so one thing the note kind of interesting in many parts of like engineering we want to pick like one tool and then like kind of deal with the problems with that and then just make it work but in fact for a lot of machine learning what workloads your goal is to try to just maximize to get the best kind of results so that might be the highest accuracy or it might mean some mix of accuracy and explained ability or some mix of accuracy and in performance and so in order to do this you actually want to go through and just try out as many different kinds of models as many different frameworks as possible both in the features Asian data prep space as well as the training space okay so we actually want to try and leverage as many of you as possible in addition to sort of trying different ones like tuning which which kind of framework we use they always have different hyper parameters for example for the training side or different features that you can engineer and build off of for the data prep side so we have to kind of iterate and do a lot of work in order to optimize that and then as we kind of do this multiple times we need to keep track of how things are changing and we need to sort of scale up every component and after we do all of that and we have a nice successful scalable model we have to build to share it with our co-workers and then you know over time be able to manage this model so that's the kind of the notion of the exchange so this is all you know the very typical machine learning development workflow when you want a production either a real model now as a data scientist or m/l engineer what you really want to spend your time doing is like the feature engineering potentially or maybe just the training and the moving to production stage you don't want to spend a lot of time in terms of the data prep or like the raw data the deployment or like the scaling or all these other things so what's kind of the state of the world today so the state of the world today is that there exist these systems that big companies have produced that are doing a lot of machine learning so Facebook Google for example LinkedIn as well I'm sure Twitter has one too that that have they're very customized to their internal cooling and so these are great because they they standardize the entire data prep training and deployment cycle and so if you work within this company within this platform you can move very quickly you can share your results and that's awesome but the problem is to two problems one the oftentimes though they'll be specific to a small number of algorithms or frameworks that this team wants to support and has the capability of supporting and they'll also be tied of course to one company's infrastructure so it's not broadly usable so kind of the problem statement of ml flow is can we provide similar benefits that these these tools get and at these companies in an open manner that everyone can leverage so ml flow is an open source machine learning platform one of the goals is to work with any existing machine learning library and language and to be able to deploy it and run it anywhere so on any cloud locally whatever and to build a scale up to look to big data so we actually this is a relatively recent product we launched this this June open source state already we have 57 contributors throughout numerous companies and there's a lot of new features that have been introduced since its inception so in this talk I want to talk about kind of the architecture of an L flow I want to give a brief demo and then kind of talked about announcements in future direction of NL flow and that's it okay so to start with how does mo if it work how do we kind of take this to this complex pipeline that we want to build and break it down to different components so there's three major sub components within ml flow that we kind of booked this problem down into so the first is tracking projects and then models so for the tracking piece this is how as you're iterating and running experiments how you capture the code and the data and the results that you that you've done so you can compare them now and later in the future projects is a way to package your code in a way that can be ran in and dependencies so that not only can you run it but other people on your team can also rerun these needed workflow steps and finally models is an interface between the creation of amount of the training step and the deployment stuff and so I'll talk I'll walk through kind of how all of these work so to give a sense of kind of why we're talking about this let's first talk about how we might do model development without mo flow so this is today so I might have this is a very simple very you know obvious problem I have some text I extract it into engrams of some number of n I train some model on it so that's kind of abstracted away and that model maybe has a parameter such as a learning rate and then I compute the accuracy and in this case I just printed and then pickle it out to a dko file which I could then load back in to do inference somewhere else so this works great I can just try out a bunch of different end a bunch of different learning rate and pick the best accuracy and this is kind of like what a lot of people do and it makes a lot of sense it's a good it's a good starting place but as we kind of talk about change over time it doesn't really age very well this piece of code so for example if I expand my input data how do I then compare the new results to the old results or if I want to tune some other parameter now how do I the same with the same question arises or if I upgrade the library and I want to see how that changes things or if I just want to change other parts of the code and want to compare two entirely different sets of code now that this N and L R and accuracy may not be sufficient to do that additionally so that that's kind of the development side once I finished developing my model I have a great model I wanted to play it so typically what happens is that a data scientist will create the model and that will produce some amount of codes amount of models and they'll ask the production engineer please put this in a rest server or please do batch inference on our production data and they kind of want to hand it off that way so in today's world this looks a lot like the data scientist might produce a secular model please deploy us or a spark model or R or tensorflow or just a random archive idea that they've implemented and so the production engine has to do something special for each one of these and that's quite quite a headache and it increases the burden and the inefficiency between these two parts of the organization so we can fix this with it with ml flow so we're going to replace the print and pickling statements with this notion of ml flow woged / m and mo flow log metric and this notion of logger model so once I'm just log the parameters that were input these are the inputs to my program the metrics that were output like the score and then the actual model itself because mo flow has a scikit-learn package as well as it does for tensor flow and all these other things I was going to log that out as well and so now I can just open up locally this ml flow UI and I can do I can track these parameters metrics and output files and the code version and in fact of this code itself and just search through it so you can see here like this is an example of the UI I can you know sort by various components I can find the best accuracy I can click into one look at the actual like plots and results that came out I can say ok this was the best one and just put a note and so this is a way I can sort of collaborate and view runs across you know my little experiment here either now or in the future additionally I can compare runs so I just want to try to find how different runs are different and so this is just one one example representation where you can just do a scatter plot against two different axes and just kind of see you know for example the dependence of these two metrics in this case so that kind of gives an overview of how mo flow provides tracking utilities as you're trying to train and compare your models additionally so the next problem is is projects so I might have code and and dependencies and config you know kind of packages Python packages jars whatever and I know how to run this locally but I want to have someone else on some other team also be able to rent locally or I want someone to be able to run on some remote cluster and so projects provide a way for you to specify and operate an environment where you provide your code dependencies and config and then can use it in these other environments finally the last component of ml flow is models so models allow you to so rather than having this n by n problem saying you've produced a tensor flow model or you produce a scikit-learn model then you want to deploy that scikit-learn model to spark or that tensorflow model to Azure ml instead of having this problem for each training train model in each deployment tool instead you can say how can I create an ml flow model from my training application and then as long as your inference code can take in an ml flow model then you can't you have to couple these two so you only need to write one connector for spark or one connector for as your ml as opposed to n for each individual training program and so what kind of a kind of show an example of that during the demo once you have this kind of model now instead of saying please just deploy this like it learn or are etc model you just say please PLEASE deploy this ml flow model and then if you can just get it into the rest server or spark or wherever okay and then you can you know update over time for example cool okay so now just to kind of show a demo of this cool so this time I'm going to be showing using a notebook this is actually part of the open source examples though so you can actually just download this code locally from the ml file repository and you can just play with it and you know run the entire UI entire thing locally so there's nothing specific to notebook environment or in this case the database environment so the data were working with today is gonna be this wine data set so there's some very serious researchers at a university who collected for 4,000 different bottles of wine they measured some of the physical properties of them like the acidity and alcohol and pH this kind of thing and then they they had three people taste tested and then they took the median score of those three so this is very serious research it's very important to the profession of society and so the goal here is can we take these physical properties these chemical properties and predict the the quality that someone would would would estimate from it so the so that's our data so we're gonna use scikit-learn for this and I'm just gonna import some stuff here you see we're reading the data we're gonna split it into let's say a twenty five seventy five training and test split and we're just going to say okay Oliver that the features we're gonna use is everything except the quality so all the features you see above and then the the thing we're trying to train on is quality and then we see here we're going to using an elastic net model so elastic net if you aren't familiar with is a kind of linear regression so for every one of these features we're going to try to find some coefficient and matches it and additionally we have this extra regularization term based on alpha which can be C D could decrease the tendency to over fit on this model so it won't be quite as good of a fit on the training model on the training data but hopefully it's a better fit on our test data and additionally there's an extra parameter called l1 ratio which is related to which which regularization term are using and how much weight to give it anyway at the end of the day there exists these two hyper parameters that have some effect on how well our model generalizes and how well it does so we're just going to use scikit-learn as a last net to Train it we're going to predict the the results and then evaluate kind of how it did and so we can kind of just run this and we see here that we get like you know a root mean squared error of 0.8 1 in this case and I can just kind of rerun and change this around a little bit so 0.8 so obviously I'm trying get a lower error I mean to 0.77 okay so I got a little bit lower but now that I've kind of run this a bunch of times you know I might have a hard time looking back and saying okay which was the best one overall or as I change parameters over time kind of what was the best so what I'm just gonna do is I'm gonna add in first gonna import an alpha 1 so I've taken it I'm just going to take this code and wrap it with some useful logging metrics so I'm gonna say start run mint and in addition to these prints I'm just gonna have actually I'll just replace the prince with ml float log cram so we'll start by logging the alpha value in the l1 ratio and then we'll kind of we'll just log the metrics as well so we're gonna log the root mean squared error as what means great error and the metric mean absolute error doesn't mean absolute error and Y are two and additionally one last thing we in the log is the actual model itself so this LR is actually a linear regression model that we can then read back in so I'm just gonna call SK learn log model and so that will run cool and so I'm gonna run this a few more times just with different parameters 3 maybe just try something cool and then now I've run this I can now go to the ml full UI refresh and you can see the run that I've generated right here and let's say I'm I've taken the root mean squared error and I'm kind of sorting well descending I probably want a sort ascending to get the smallest and we can see here that for example the Alpha point one and an L 1 ratio point two was actually the best among these these different things it's like an actually click in and kind of see more information here for example I can show you the ml model file type and you can see that this for example is a scikit-learn model and it's also loadable directly into python so that's the notion of an ml model one other thing I kind of want to showcase here is that in addition to these parameters and the new metrics we also have the code here so for example let's say I if you just changed my code to say okay what if I just try a training on more data maybe that will help so I'm going to change the test size to 0.1 I'm going to rerun that and then let's say I just I'm going to pick the same parameters that were the best so point 1 and point 2 let's say gonna rerun that alright i refresh the page okay actually in this case didn't wait maybe I didn't do something correctly that looked good okay in this case changing the test site did not help at all so they're identical but I yeah okay anyway mm-hm nevertheless I can see look at the actual revision since anyone's like it so if I click on the actual source I can see the different revisions of code so for example on this one I can see that the test size here was point 1 and that the test size here was point 2 5 and they're just kind of like associated with each run is not only the metrics and parameters but also the actual code that executed here so I can go back in time and kind of restore to this and run it for example oh I guess there was a slight reduction in the mean absolute error and an increase in it r-squared cool okay so once I've completed my model training now I could as a maybe a different engineer and different like environment entirely I want to do inference using this model so I have a good model and I have new day that came in and I want to kind of check out and check out how well it's doing so I'm gonna plug in my run ID and so you can see the code that we're doing here it to start with this one is very specific to scikit-learn it says load back that scikit-learn model and then just predict based on this kind of very specific wine feature set and you can see that it predicted for this one quality of five point five seven I can also do this in spark of just loading it like saying mo flow Python spark UDF and then this this is actually PI func so it didn't actually require anything about ml flow itself sorry about scikit-learn itself and using the same run ID I can then take my original data data frame which is this guy up here and I'll supplement it with the actual predicted values and so we can see here that we have these quality metrics along with the actual predictions per for each quality okay and so basically I was kind of an example how to do batch prediction using the model that had trained like somewhere else and there's a notebook in this case and predicted here so that's that's just an exam that's a demo of how we can use ml flow to take an existing workload and supplement it in a way that we can kind of decouple both the depots decouple the deployment or say video training and model inference parts as well as kind of help compare and evaluate the metrics over time and go back and look at how things went so just to kind of give a bit on kind of how it how this project is going so since the release in June we've added a lot of model packaging formats so for example we support ml live h2o tension flow piped or just carrots and we're adding any new ones that as people have them there's a you know storage back-end first these artifacts and that supports basically any world in three major qualities as well as f/2 SFTP and in addition to the Python API we have Java in our api's as first-class citizens of course it's also API in general where API first so the REST API is like always good and up-to-date but these are just kind of convenient wrappers in these languages and additionally we have new examples like so it's easier to get started with there's a lot of documentation on that and improve UI features for this help of like comparing and visualizing results so in conclusion workflow platforms can greatly simplify annal development improving usability for both data scientists and engineers and ml flow attempts to do this for you know all machine learning more loads so get started a ml flow Network cool thanks yeah so questions yes so it works with Scala but the the API the current client is I could just its Java first so I mean we implemented Java so it's more natural to use it from existing Java we're closed as well as call our floats no no so yes the the UI is completely open source so you can just download it and call mo flow UI locally and actually launches a server and you can either run out server locally you can run the server remotely and just authenticate it to it that way yeah so actually I did actually show that so it maybe was a bit subtle the so the deployment could mean multiple things in this case I was deploying it to spark and so this is actually running this could run in parallel any number of machines either in a batch format or in a streaming format and so I was actually voting as a UDF so now this is just a UDF and so I'm calling with column where I'm adding a new column called prediction that actually takes in from the data frame so just to kind of show that again so the input DF was all of the features and the output I added prediction and so that is actually running across a cluster know so yes so good question so the question is like what does deployment mean so one thing deployment can mean is you want to do batch inference across many machines in this case mo flow comes with like the ability to convert your model into a spark UDF it could also mean though for example just running out one machine and so we also have a way of just loading it back into like Python itself for example and then just serving it we also have a way of serving as a REST API endpoint so then you can just deploy it to kubernetes or sage maker as rimmel or whatever you like just as a REST API endpoint exactly yes so the tracking site is about logging in version of your models and the serving and the deployment side is both about serving as well as other kinds of a batch inference or streaming inference it's a good question so actually not necessarily so git is actually a pretty good way of defining so for example just to show the so this examples actually pulled from this one right here and you can see here in our trained app i it's very similar code almost identical the advantage of git is that you can kind of check in it all at once and kind of like point to the actual gate repository and if I can actually run it from a get repository in which case it will include the commit history like the actual commit ID so what you can see here is in this UI this this source is a notebook and actually points to the revision but if I if I ran it locally from like a git repository it would actually be the gettin hash instead one advantage of that approach is that if you if you run like this Python file it's like all or nothing whereas some one weird thing about a notebook is that I could have actually run things in a different order and it's not necessarily obvious which order you ran it in so it's a kind of subtle slight weakness of notebooks as we have them today one thing to also note here actually I didn't show this because I was right in an opal instead this is the notion of a project so it's very simple you can see it just says so you know we're not specific to Python or Java or Scala or whatever but it is called Python trained a PI but I guess one thing I didn't mention I know that the scholar conversation came up most data scientists do use Python and python-based libraries the one the the main exception to that is spark for NL lib almost nothing else is in anything other than Python or C++ as far as you know we found so but that said this is not specific to Python this particular part you can find parameters you can define a kondeh gamma file so this includes this is like Conda is a way to package Python dependencies so you can select specific versions is now this is reproducible so if I just come into this directory call ml flow run it will activate that it will download these packages in a virtual environment it will then run it hi there so similarly you could you do dr. container we actually don't have a way of doing jars right now because like - that's a little bit more complicated how you like you read you have to like in line SBT or in line maven or something in this but this is the notion of a project which allows you to reproduce this across multiple machines another question oh well thanks