Scale By The Bay 2021 : Milecia McGregor, Tuning Hyperparameters with DVC Experiments

Scale By The Bay 2021 : Milecia McGregor, Tuning Hyperparameters with DVC Experiments

Recording: Scale By The Bay 2021 : Milecia McGregor, Tuning Hyperparameters with DVC Experiments

all right thanks for the intro so hey everybody my name is melisha and i'm a developer advocate at iterative ai where i mainly focus on our open source tool dvc so if you do have any questions during the talker after the talk feel free to reach out to me directly on twitter at flippedcoding or you can reach out to the whole team on twitter at dvc.org but today i want to talk to you about just a problem that machine learning engineers run into every day and we'll talk about using reproducible experiments to make better models and what that really means so let's just jump straight into it there's some common issues that come up with every machine learning project no matter how well defined it is how much data you have all that good stuff usually you're trying to find the best combination of hyper parameters your algorithm and your data set so that you get this really good model that you can finally push to production but over the course of even just a week you make hundreds of experiments so you're going through you're changing algorithms to see which one performs better you're updating hyper parameters to figure out which combination gives you the better results you're maybe even changing data sets because business requirements have changed or you just have more information but right now you have to keep track of all of those changes yourself so i've seen teams that keep track of their metrics and the associated hyper parameters in spreadsheets i've seen teams try to associate a zip file with every git commit that they make and it just gets really messy because over time it's hard to follow all of those changes so maybe you have some really elaborate spreadsheet system for tracking your experiments but somebody else on your team actually just makes a bunch of different zip files to try to keep all of the code the configs and the data just bundled together so over time as you're doing all of these different experiments keeping track of every little change you make gets super tedious because you get to this point where you're like oh my gosh i remember there was that one experiment and it gave me this ridiculous accuracy of like 97 what did i do to get that model what hyper parameters did i use was i using this algorithm when i did that or was i using a different one was that the data set that they sent a month ago or was that the data set from last week so you see how over time it gets hard to keep track of all of those other experiments you've run and that's why we're going to talk about fixing that problem today because nobody wants to have to keep track of all of these really tiny changes that they make maybe every few minutes so we definitely need to fix that problem and the way that we do that is by using tools that help us bundle everything together so when you have the code and data associated with each of your experiments it's really easy to go back to any experiment you run so you're able to basically bundle together your data your configs and your model and you do that for every experiment you do so this sounds like maybe kind of magic or fancy or something but there's plenty of tools that handle this for you but of course today we're talking about hyper parameter tuning so now that you know our issue is with keeping track of all of these tiny changes the metrics that are associated with them the code the data the configs let's go through a real a real example of what this is kind of like in practice so we're going to start with just some basic hyper parameter tuning and i'm sure a lot of you know what hyperparameter tuning is already but just in case you don't hyperparameters are the values that define your model so for example if you're using something like a neural net maybe you need to specify the number of layers that you have in that net or maybe you're working with a random forest classifier and you need to specify the max depth for that so these are the parameters that define those parts of your model maybe you have some kind of learning rate that you need to play around with or you have a momentum that you want to try different values for so these are the hyper parameters that i'm talking about and a couple of common methods used for hyperparameter tuning are grid search and random search so with your grid search let's say you have three hyperparameter values you have your learning rate your momentum and i don't know maybe the number of layers in your network so you have these three hyper parameters right and each of these hyper parameters has 10 values each so with a grid search you're going to go through every possible combination of those parameters and if you think about it you have three hyper parameters 10 values each that's a lot of different experiments and i'm not sure about you but i don't want to have to keep track of every little change that i'm making when i'm going through a grid search because that makes it take even longer like we're already going through all these values and now i have to stop every time and be like oh well this was the loss associated with this experiment when my learning rate was 0.5 instead of 0.1 so just imagine doing that for every little thing every grid search combination that is what we're going to try to not worry about and then random search is very similar to grid search in just the way that it operates so you still have let's take our example of our three hyper parameter values and instead of taking 10 discrete values or you could keep those 10 discrete values what random search will do is just well randomly hop around different combinations of hyperparameter values and that's it it doesn't exhaustively go through every possible combination it just jumps around and usually if you are doing a random search for a similar amount of time that you would a grid search you will come out with more optimum values and it's just because the random search gives you a broader range of combinations a lot faster than grid search does but now that we've you know talked about our problem we're doing this hyper parameter tuning with grid search and random search and we have all of these little incremental experiments we need to run and we don't really want to have to write down everything as we go right that's where a tool like dvc comes in so when you're thinking about dvc remember it's open source so you can come look at the code tweak it however you want it works on top of git so it's not something that's like its own standalone application or service it's just a little command line tool that you add to your project and it tracks your code and your data together by taking advantage of git and it's not opinionated at all so there aren't any libraries you have to add to your code you don't have to call any apis you don't even have to use any particular program in language you can use python javascript which yes people do use javascript for machine learning and i am definitely people you can use it with r julia whatever and language your project is in dvc doesn't care it's just the cli tool that helps you manage your experiments it helps you do data versioning so when you do have those updated requirements from the business and you need to expand your model but you already have these existing data sets and you don't want to worry about well which data set am i working with dvc tracks and versions all of that for you and kind of the way that it works is that it has metadata it associates with each experiment that it pushes to a custom get rest so that's how it's tracking all of these experiments in the background for you it's just adding things to the dvc cache when you decide an experiment is good enough you commit it and then it's yours forever you can share it with people on your team you can go back and redo other experiments like if you run a hundred experiments and you see number 71 has the best loss value or something you can go back to that experiment and reproduce the exact conditions that you ran it in so that's that's a few reasons why dvc is such a cool thing but what it does whenever you're ready to run an experiment you'll execute this dvc exp run command and what happens is it executes whatever training script you have if there are other stages in your pipeline like maybe you train your model and then you have some kind of evaluation step where you print out your metrics to some kind of json file or maybe a csv whatever it is you use to store your metrics but what happens is you run your experiment dbc takes whatever the current code is whatever the current data version is and whatever your configs are and it just neatly packages them together and associates all of that with this model that you get at the end so when you run an experiment you don't have to pay attention as closely anymore you don't have to worry about well what data set am i using did did i change this code this time or did i change that code last time was this the algorithm i was working with now you don't have to ask yourself those questions and lose sleep at night over missing out on that really good model that you had you're able to run your experiment dvc bundles everything together so that whenever you're ready to come back and maybe compare different experiments or make changes to things and try again you already have this neat little package whenever so you're going through this hyper parameter tuning and you have all of these different experiments so each of those experiments with dvc looks just like this you have your data your params your model and the code that makes up the model so you have everything that you need to go back to any previous experiment and pick up training from there maybe maybe you see that right now you don't have a best model but you have a general direction of where to go so instead of having to go through your spreadsheet or you know bug people about going through zip files and figuring out what metrics are associated with what now you have just one tool where you can look at everything and then you can start pulling out those different patterns for where you need to go with your model next so let's say you've run an experiment with dvc just one experiment like you see here so this is an example of the table you'll get after you run an experiment and you see you can take a look at your metrics you can take a look at the hyper parameter values you were using so in our case our hyperparams are this trend.nest and this min split value and this is just one experiment so it doesn't look like it's doing a lot right now but we're gonna go through this hyper parameter tuning and what that means is that you already have some set of values right you already know your hyperparameters so what we're going to do is use dvc to set up a queue of experiments so basically it's like writing a little script to do a for loop and you run you make all of the combinations that you want to use for your hyperparameter tuning so we set up a queue with these different hyperparameter values and we get a table that looks something like this so what's happening here is that we've cued up a bunch of experiments with a grid search so we have two hyper parameter values and each one has i think maybe five each just to keep things simple but this is just a small snippet of what this whole metrics table looks like so for example if you are doing some kind of training experiments and you're on a team because of course you're probably on a team you might want to look at the experiments you have in your queue just to make sure that you're not repeating something that's already done that's the cool thing about dvc is you're able to share these experiments since they're all neatly bundled up it means anybody on your team can take your experiment and resume training from that particular set of code hyper params and data so when you have a bunch of people running a grid search or a random search there's bound to be some overlap in those values right so maybe you use this table to look through and be like oh well they've already run experiments with this combination of hyperparams i'm just going to remove those from the queue and run these new ones that's the kind of flexibility that you get when you're using something like dvc so we've looked at our cute experiment we've decided that you know this combination of hyper parameters it's fine so we'll run all of the experiments now and the cool thing with dvc is that you have all of these experiments in your queue maybe you have hundreds just in your queue you can run them all at the same time in series or you can run them in parallel so if you're offloading things to the cloud maybe running experiments in parallel will give you i don't know maybe it'll save you some time on your overall research and it can also save you some resources if you have two or three machine learning engineers running hundreds of experiments in parallel in parallel you're doing a lot of work really fast but let's say you've gotten all of the experiments in your queue they've run and now you're ready to look at the results from those you can just go back to your terminal run that dvc exp show command and you'll see all of the experiments that were run from your queue so you can look at your average precision in which set of hyper parameter values are associated with that and you see we have kind of a decent range of average precisions here which is probably what we would expect given how widely our hyper parameter values swing but it looks like if we have a larger nest and a larger min split that we get a better average precision so even though maybe this experiment or this set of experiments didn't give you your you know golden model you at least know which direction to go so you might want to compare these different metrics just with the plot because sometimes sometimes a picture is just easier to understand than the table so we would use this plots diff command to compare two or more experiments so maybe you just want to get a quick visual or you want to check with your other machine learning or data science friends and see if your results actually make sense so you'll run that command we just have these two experiments because they were the furthest apart but if we look at these plots this is definitely something you would show to you know a colleague another data scientist ml engineer just so maybe a sanity check because you might be looking at this table of experiments and you're like well i think these two might set like some bounds for what our hyper parameter values should be going forward and then you pass this plot or these plots on to somebody else to review and you see what they think about it and then you all decide on which experiment to go from here maybe you pull another one from the table to compare but you have this flexibility to just jump around different experiments and quickly compare them so can you imagine trying to make a plot with just these two experiments from that giant table of experiments if you didn't have some kind of like tracking slash versioning system it it takes a while you might need to make some separate tool just to do that so we don't want to do extra work because why it's hard enough to train the model let alone make a whole system to keep track of everything for you but this is just one of the ways you can take those metrics and visualize them so let's say you know you've talked to your friends and they're like ah maybe you should run some more experiments so we go back we set up a queue with some more values for our hyper parameters and we run this and we we're trying a random search this time just because we're like well it looks like larger values are better but maybe we didn't try a good enough range of smaller values because we jumped from like 26 to 100 and something pretty quick but we're still pretty sure that our nest and mint split should be larger but we just really want to make sure before we rule out those small values so we have three new experiments in our queue we check with everyone to make sure nobody's tried this set of values and we decide to run them so again we just open up the terminal wherever your terminal is it could be in your ide it could be just a plain old terminal it could be something in your cloud wherever it is you can open that up and run all of your experiments and then again we'll run this exp show command to look at our table and would you look at that it does look like the larger values are probably going to be better in general right maybe not as large as you know 3 400 9 400 but definitely above 92 that you know maybe above 100 we're just trying to figure out what is the right approach so you keep running these experiments and you keep setting up these cues so again you set up another queue of experiments because those didn't give you the results you were looking for and now you're running through more experiments from a random search and you've isolated the nest values so we're trying to figure out which hyperparameter maybe has a larger impact on our model's performance so we're going to keep the nest values under 100 and we're going to keep the men split values over 350. so we've come up with this idea that we just want to test out so we'll run some experiments again and we'll look at our table again so if we look at this maybe we should definitely factor in the nest because it looks like having that as a lower value is doing something we don't know but again we're like well colleagues friends what do you think when you see these results and you might not need them to see your entire table of experiments mainly because that'll be a lot for them to parse through but there's probably a few that you just have you're like well ah i'm not sure which one to choose i'm not sure what these results mean which way should i go so we grab a couple more experiments some from the one from the newer ones one from the older ones and we look at the difference between those metrics so again we have this plot that we're thinking about sharing with all of our colleagues so that we can get some feedback maybe they'll see something in this experiment data that they're like oh my gosh i remember i was looking at this problem and my results were way off maybe we should just start with your experiment and now everybody's pulling down your code that data set associated with it and your configs so everybody can jump in on this great new direction that your experiment is taking them so you run all of these different experiments and you end up with a table that looks even bigger than this and this has quite a few different experiments on it so you have this giant table but this is probably a lot better than worrying about having a spreadsheet that has links to zip files that might also have a link to some git commit and then things get jumbled if somebody edits the spreadsheet in the wrong way and the rose gets shifted it it's so much easier to just have a tool that's tracking everything for you it's making sure that if you did want to go back to any of these previous experiments you could so now that you see how dvc works you have a tool that does all of your code and data tracking for you it adds reproducibility to your project because you have these little bundles that take you back to any experiment you run and you can share all of these things with anybody on your team so there's no more well i did this experiment on my computer i don't know why you're getting different results on your computer now you have a way to keep everything uh what's the word consistent so with dvc you have a tool that keeps all of your experiments consistent regardless of what environment you're running in which means when it's time to go to prod you know exactly what it's going to take to get the exact results you've already been seeing so now that you've seen what it's like to solve the nightmare of keeping track of all of these different experiments let's wrap up with the few key takeaways the first thing i hope that you see adding reproducibility to your experiments is important so when you are doing a grid search and you have these hundreds of experiments and you do find maybe a handful that you want to test a little more now you have the ability to do that regardless of when you ran that experiment that little bundle with your code hyperparams and data will always be there and then dvc is a great tool that helps you track every part of your experiment i think it's the only one that bundles your code and your data together but i do say don't be afraid to try new tools so dvc is open source is just a command line tool and you don't have to add anything to your code but there are other tools out there like ml flow or tensorboard that might fit your needs better so just try out different stuff to see what fits you the best but if you do have any questions i'll be around in the spatial chat and you can still reach out to me on twitter at flippedcoding but really if you have any deep questions about dvc or how it compares to tools like ml flow or tensorboard you should definitely join our discord so the link is down here in the corner but this is where we talk about all things mlaps so thank you for your time and i hope that you were able to learn something from this talk you