sfspark.org: Sven Hafeneger, Hyperparameter Optimization -- when scikit-learn meets PySpark
you understand I just came from Germany so I'm like but I'm very happy to hip to be here to speak about a specific topic mmm I member of the da six deaf team where I work as a kind of software engineer back in the engineer and a data scientist so I can do both which is quite fun and through my work I was on asked - epic sparks big data engine this is useful when I have small data set and the answer was kind of yes and I had a little bit which is basically the foundation of this small talk here and hope you learn something which might help during your daily work or private hex so today I would like to talk about hyper parameter optimization and where it can be interface for second learn and PI spark leads so you all most likely have familiar with the data sense workflow where you have a business problem a very interesting one you saw an understand data you enter the modeling phase and evaluate it and deploy it to have a solution for your problem so modeling is one of the most fun parts of the cycle but also they are lot of steps which are very tedious and boring and during this modeling phase you have to tune the models and the models have a lot of knobs and this task is actually embarrassingly paralyzed above in many cases not working and not working so you can imagine those models as boxes where they have on the surface different knobs and different knobs may actually tune them the right way it could improve the robustness of the model or the clip influence and complexity or help with the class and balance see and those models can have a lot of those knobs okay so in terms of hyperbolas ation what is this kind of a problem is the problem of choosing the set of parameters for learning algorithm with respect to a certain quality measure where you set it to say how good is the model there are different approaches a grid search random search or based on optimization but for the sake of a short talk I will concentrate on the grid search today and I'd like to start with a short question who are you knows what this is yeah but this is specific the living room of Big Bang Theory exactly okay so I hope everybody else you know spec Bang Theory and the character Sheldon okay perfect so who have you knows what she'll what hitter a picky right and he has in this room a specific spot where he likes lucid so since have a property authorization is very like abstract and some people think is boring I don't want to like to give you a livid analogy so we have to think about this throughout the talk okay so in terms of machine learning and hyper cramped emulation imagine that this living room is the box where Sheldon is a learner and there are two parameters for the x and y coordinates of the room and the goal of the learner Sheldon is to find the optimal spot for this living room and we all know shared is very picky okay and you can imagine that way and the days throughout the year are the training instances or the test data and the training data for the learner which is shown so imagine that and for that one way which is embarrassing paralyzed ability with search so she'll just go through all out all the spots of this living room and the two parameters of the Sheldon learner X&Y in this box and when he spends the days without the e it was in a year he learned about the optimality of his spot and he has like several features like like weather conditions lighting conditions weather television wind temperature and so on and so on because she's very picky right so you could say okay the different techniques to do hypergravity optimization first a micro random search well shared with just spawn randomly a few times and with the probability of the few iterations she could reach around 90% for optimal spot but Sheldon is yeah connection perfect in a sec so that will not work there's another approach for Bayesian optimization but Sheldon in this box would know the underlying distribution of his optimality and the the feature set and that he can omit certain spaces but since share the paranoid the dual zone of work okay so there's lovely thoughts to trail the good search and for us as well and the goal is the child in time to the final spot where they say K here at CF an optimal space here I will be and the same is for the machine learning algorithm in terms of hyper parent optimization so but after we know what is now how perhaps optimization we have to answer the question what is cross-validation because the content in hand just shot in true so assuming shell would just learn one year and you take those blue bar blue lines as training data enter and the green line as a validation set you could get a score for your optimality for example accuracy the thing is if you split it a certain way it's not stable so shells would learn for example during the winter months and predict for those summer months and then Oh note that the spot is not perfect in the summer only in the winter so that's why Sheldon has to train on different month of sort of a year to do so we have also to split the data a different way and activate about around the accuracy so we have a more stable result for the parameter set not working okay who are you useless your scikit-learn and Python cool a lot so for this could be for you but specifically interesting because it gives you a mean if you even have small data to start using PI Spock ok so here I just generated with a psychic learn some training and test data splitted the randomly and build default random forest classifier with all those default parameters ok I fit it to the training data I predict for the training data well I get a very good accuracy score then I predict the test data and well yes it's a lot fitted because it sucks to overcome this we need to tune the parameters so back to the knobs so when inference has a lot of different knobs and each of those could improve the test score of our algorithm so for simplicity I just built a grid now with number of estimators and the max test of the trees in the random forest and now this is our search space like in the living room ok and share against our learner and render for us the children so here you see the psychic learn code to perform a grid search for this random forest classifier and you set the cross-validation to 3 so we have multiple results and out of this grid there around 528 unique jobs and with one core you need around 13 minutes ok which is a lot so if you increase the data a little bit or bids a bit bigger it becomes more and more even two hours or half a day or a day maybe four very simple problem actually so second but it it's worth it so after 30 minutes you get a best estimator after the grid search where the market s is 15 and the number of estimators is 200 and if you now perform the test prediction you get accuracy of 0.67 around so then big improvement so definitely pays off to do the parameter optimization so now assume you would have doing this hideous death star cluster available in this part of this notebook I have a spot contact available you see in the first line and then you can import a package called spark at colorin and basically the code does not change this much except that you pass for the first argument to spark context which is entries to your cluster and then you say still cross-validation three you want to tune the parameters which is a great we define beforehand and it still will be 2500 jobs around the only differences for my experiment I took a enterprise cluster on these eggs and yeah it took only 28 seconds and depending on how much money we are willing to spend and how big your castle is you make actually a very tedious task for which you have to wait a very interactive one where you just had to wait a few minutes instead of two hours it can just proceed because this is a no brainer it don't have to spend many a lot of time on it so the goal is actually to save time because this time you're safe waiting you can invest in actually solving a business problem or two more fun part which is teach engineering or try other models right so the ultimate goal is to have faster cycles especially if you're working in a Jupiter notebook you don't want to wait for your parameters like 8 hours the 30 minutes is acceptable but otherwise you have to go to sleep and the next money to check your parameters and assuming you don't have one single classifier but maybe 10 in your ensemble then you have to a lot of crits you have to train and a few bucks and a cluster you could do it very fast so you also see on Kangol that those parameters you find also of high value so they are like n+ discussion boards about when on what when or if to share parameters for models because it takes a lot of time to train them for example one guy here took the whole weekend to Tunis parameter so he can come up to the top scorers in the leaderboard and other say ok you should not share those parameters like three weeks before because it takes weeks to tune them so a lot of people have the same approach they're very similar and if you know the parameter you can just take the approach on solve it with your own one and you are better off than alone so those parameters of a value so there's a package whose pockets can learn as I mentioned you can install it via tip then it's working I guess I used basics as my platform for experiments where generated arbitrary data set students I could learn from the regression and classification problems actually it's working quite well and you just have to pip install it on the a6 we have a star conduct available then you can just do your own experiments it's quite fun so I hope you will like it and yeah yes actually what you're talking about is a depend equation so this integration package you don't transform anything sure you have a you start with zone scikit-learn SEC right you define your data is the grid and what you do is it just in course this package sparkers color on you don't have to convert anything and give them the collective fire ii learnt classifier the spot context and the grid and that's all one line of code change nothing next more actually two lines for the import no the greatest just a dictionary like a key - value key is a parameter type or the string for the parameter and the value is the list of values you want to test you don't have to transform anything no you do not know this is a just pan of data frame yeah but you don't do it explicitly so you just use the API so therefore you have this wrapper package to be imported and you only change the call for the grid search nothing else yes no okay no no you don't have to do it through this approach because it for that you have the package imported okay yes I stayed in the beginning so people are okay when I don't have that much data can I still use spark in the further north moment I suspect and this is the answer so if you don't have like terabyte of data and don't need like a cluster for a computation then you step would normally start with a normal python snake with your models but at this point where you have embarrassingly paralyzes the problem where you need a class or you can jump in a new spot because if you use spark and you don't have set that much data then there's a so assuming you have an XY coordinate and you increase the data size and you measure the time for computing the results for machine learning algorithms there would be a turnover at a certain point where it takes off to use spark before that point it will not necessarily be faster using spark understand so depending on your size of your data if you have big data problem you would start with ties back right away but say a lot of people out there who start with a small sample or a small data set and will reach this step of the parameter optimization where they have to wait and this package where you distribute the hyper parameter optimization the good search only on spark could help you to reduce the time for waiting effort that's all any other questions yes you might a choice of algorithms this very tough questions there's no standard choice as there's no best algorithm so I mean first of all if you get the data upset from your customer no no no there's no three best algorithms I will not tell you that but I will tell you an approach we determine those best three algorithms so for example when customer gets me a data set normally it comes to seducers question with this I would make more money or I want reduce a cost right so simple as that so you want to maximize the car weight maximize food revenue and minimize the cost normally then you get a lot of data which is numerical or a categorical and then you have to specify the target so normally in on kegger competitions you get a nice target feature evaluation that you'd normally don't get from customer customer just gives you dirty data and wants to save money or make more money so from the data you have first to mathematical model the problem correctly so for example if you want to predict the revenue it's a regression problem and what if you want to clarify like fraud with known fraud maybe a then a classification problem or anomaly detection when you don't have labels it's clustering so I think those first category of algorithms I can give you to determine problem and then I would say there are a lot of different blocks I don't have here an iPad for example if the cool linearity of the data is too high you cannot use logistic regression is the number of columns is too big you cannot use random forests so you have to try to from the data to derive those constraints on a paper and try to map or to to exclude all the algorithms we should note for tools or constraints and then you look at a set of those three algorithms thus this help for you okay oh yes yes yes so there's an awesome free book called elements of holistic learning from TV honey in HD which is a lot of mass this one it's for free and then it's a less mass version called introduction to statistical learning with our code which is also for free so take those two books written do the code samples and then you get an intuition which I've written to take how the data must be how you have to set up your cross validation pipeline evaluation yeah so that would be good approach I can share them later with you any other questions okay thanks attention [Applause] you [Music]