Time-series Feature Engineering Done Rig...

data.bythebay.io: Parag Namjoshi, Time-series Feature Engineering Done Right

Recording: data.bythebay.io: Parag Namjoshi, Time-series Feature Engineering Done Right

thank you for the introduction Michael and welcome so I'm going to talk about time series which engineering does right and just to add a little bit of color to my background I started working at acting after graduating from grad school and I worked on vehicle health prognostics and driver behavior analytic and we are working on streaming data from the cars a few years back then after spending a year working on NLP problems I moved on to identified where I was working on certain problems related to record linkage and since then identified was acquired by workday and I have been working on predictive analytics for work day or since and in particular I have we have put two applications in production which are retention risk prediction and retention is prediction and the retention is prediction and the customer collections so we'll we'll talk about few lessons that we learned over the course of last few years while we as we proceed so we are going to talk about a particular class of predictive classification problems and it's a sub class of problems that we encountered while at work day and retention risk prediction is a interesting representative of this class and we are not going to talk about retention risk by itself but we are going to use it to illustrate the ideas that underlie these kind of problems we'll talk about how we formulated this problem and a validation methodology we had to use and finally we'll talk about feature engineering and then I'll conclude so as we are going to talk of use the retention of risk as our illustrative examples let's talk a little bit about retention risk so we want to quit who is going to quit in next three months for a given company and so taking a step back workday is a cloud-based provider of HR and Finance software and we have over thousand customers who use our services and work with the same application tracks transactions these transactions are related to employee hirings and terminations performance reviews pay raises stock grants promotions etc question is can this transaction history be leveraged to deliver more value to our customers and retention is caused our first attempt in that direction and a little bit of terminology so when I say worker actually mean an employee active means employees is still employed with a with that company and terminated means that employee quit all right so let's let's see what kind of problem this is so is this a historical classification problem are we trying to classify people who you know we have some history of transactions we have Roger quarry and some transactions for them are we trying to predict or I will try to separate who quit and who did not quit that's a legitimate question to ask and it could be used to answer things identify things like what are the drivers that are causing people to stay or quit but that's not our goal our goal is to predict who will leave that is Roger will leave and predicting that is our goal all right so this is the forecast like a time series forecast and you think you can look at this time series of data the blue data is historical data and the red dashed line is our prediction and gravis is the confidence interval surrounded prediction so what we want is a really a forecast now so let's talk let's go through a simulated time line to frame the problem that we want to solve and well we are trying to solve a predictive classification problem we are looking at a Jim and Jim is a great employee and his employer wants to know if Jim is going to quit in next three months and we chose to formulate this problem as a supervised classification problem in a predictive context there are other ways to formulate this problem for example several models but after some initial exploration we settled on the supervised learning as the appropriate way for us to go proceed so let's go through a similar timeline just to see how just a set of the four problem so here is Jen year back she received a raise of about five thousand dollars then she was transferred and she received another raise of seven thousand dollars here is Mary she was promoted around some time last year and now Mary starts reporter Bob the rumor is that Bob's is a terrible manager and lot of people who report to him quit and sure enough Mary left soon after now Jane is reporting to Bob is general quit soon that's the question that we want to answer so again it's a forecast based on what we have seen till now we want to predict what will happen so for workers who are active today we want to predict who will quit in next three months so we have this plus three that's our prediction horizon and we based on what has happened till now all the transactions that we have seen we want to predict who will quit or who will not that is we want to fit a function f based on data X that will allow you to predict who is going to quit and who is not all right it's almost a Markov like assumption we are our hypothesis is that we can you know feature matrix X we can capture everything that has happened to date that Lala was to predict what will happen in next few months all right so here is the timeline again and what are we predicting whether Jane will quit or not so let's go what are our hypothesis let's imagine that we sit in a time we go a take a time machine and we go back three months we have forgotten everything that has happened in the last three months and we only know what has happened till that point two three months back and our hypothesis is that what has happened till this point is to a large degree going to determine who is going to quit and who is going to not who is going to not quit that is the fact that Jane is act going to stay active and Mary's going to leave was determined by what has happened in the in the in the past all right and so how do we again we said we are going to frame this is a supervised learning problem so where our training labels going to come from we are almost setting it up as a parallel version of today sitting here today we want to predict three months in future similarly we are going to step back three months and we are going to decide based on who quit and who who remained active in last three months our data labels all right so that's our Y and where is our X the feature matrix gonna come from it's based on what has happened up to three months back okay and we are going to go with a single feature for existed purposes the feature is do you report to a bad manager or not and for Mary that feature takes value 1 since she reports a bob and 4j n that feature text values zero and we fit a model based on these X&Y the feature matrix and the prediction all right so now we need to make a forecast and we need to make a forecast for Jen Mary has quit and we no longer need a forecast for her but wait a minute didn't we train for Jane and we are again predicting on her I'd argue that we didn't train on the same Jane that we are making a prediction on we train on the Jen as of February he was a happy camper Jane of today reports a bob almost different person all right and by the way this is very similar to auto regressive time series models all right so and when we extract features for Jen sure enough now the feature text value one for Jen because now reports a bob and hopefully our model is going to predict that Jane is at risk all right so how do we fit a model we held back everything that happened in past three months all the transactions and we build our feature matrix X bar based on that our labels came from people who quit or did not quit in the last three months a little bit of terminology I'm using the or bar to indicate that we are actually using this data to fit a model alright so now we make a forecast the feature matrix that we plug in into the model X came from all the transactions that we have seen up to this point now remember that when we compute X we are only looking at people who are active today all the people who terminated we don't need to make a prediction for so and we plug this in into our model and we are going to get a forecast so how do we valid this model how do time series models get validated so you know from a restrictive example we see that we are trying to make a prediction here one year in future and here we are looking at we hold back everything that happened in last one year that is this window and we train our model based on the data we have seen before we make a prediction and we check it against what happened in the last one year fairly standard time series stuff we need to we are going to our validation strategy is going to be based it's going to be inspired by the same methodology so we hold back everything that we that has happened in last three months that's our ground truth and that's what we are going to hold ourself against we train our model based on the people who quit or did not quit between six months back and months back so that's where our viola bills are coming from and our feature matrix for training x-bar is based on everything that has happened before six months now we make a forecast we had a model that we trained and we are going to for all the people who are active at three months back we are going to create a feature matrix based on all the transactions we have seen we are going to plug them into the model and we are going to get prediction Y hat and we are going to compare that against the ground truth and that's how we are going to compute our matrix and just the quality of our model so there we go so since I like pictures so much I'm going to take another part time line so here we we have our old friend Mary at the bottom but at top we have Joe Joe had a very similar time line so here arrays a year back but he started reporting to Bob soon after and he left pretty soon so as we said we are going to hold back everything that happened in past three months and we are going to train our model based on what happened between the period of - six months - - three months and that's where our levels are going to come from so follow the label is one because you quit and for Mary it is active and whatever features so feature matrix X is based on everything that has happened in past six months and for Mary the for Mary the feature to quell is zero and for Joe the feature took value one that's how that's how and we fit our model and now we make a prediction for Mary because you asked wait so we don't need a prediction for Joe but for Mary we need to make a prediction based on we have seen till three months back alright so the feature will take value 1 for Mary in the well in the in the validation period and our model will say maid is at risk and sure enough Mary left and we get to count that as a false positive or as a true positive all right so here is a not not such a happy case here is Jack Jack started to report to Bob in our ground truth period right and he left pretty quickly after that and when we generated features from Mary the feature had value 1 for Mary but 0 for Jack our model is going to say Jack is safe and Mary's at risk and sure enough jack is a false negative for us and here we have an important point based on what we knew 3 months back we had no reason to believe Jack was at risk but stuff happens and that's why predicting feature is harder like Yogi Berra said so going back we call this methodology temporal validation we train based on which generate up our training which Annette our feature matrix based on what has happened till 6 months back our labels Phi came from what happened between 3 to 6 months back we build our model we made our forecast based on the feature matrix X that we extracted based on who was I based on transactions in 6 months back we had we predict made a prediction and we validated that against the ground truth so that's what methodology so why did we have to invent this methodology why could we not go with standard k-fold what is the fundamental assumption behind careful validation that your labels why are independent is a iid assumption there and in our case the labels are not independent so I have an example here Jack and Jill are friends and a fact that is not captured by our features but when Jill quits Jack's retention spikes so the labels themselves are not independent of each other there is a dependence structure here and the data itself is not stationary do you remark it forces and other external factors the attrition rates change as a function of time so the temporal validation methodology which prescribes picking in the future allows you to do reliable predictions that k-fold will not allow you to do all right you need more data you want to capture seasonal effects sure we do what we always do which is slide windows backwards so we we got our first data set by looking at six months back like we did the x-bar came from x1 bar came from Looking Backward six months y1 bar came by looking at between 6 to 3 months we got more data by sliding back by three months and now y2 bar which other labels came based on what happened between nine months and six months back now remember that people who terminated between this period will not survive for the next window right and then we take a union of these X eyes and my eyes and we train our model based on that right and we've forecast in the usual fashion by looking at what has happened till three months back and we validate again in the same fashion all right this is similar to non-overlapping moving block bootstrap methodology that is used quite commonly in timeseriesforecasting alright so there's one more thing that we cannot forget there are non stationary effects as I said the retention risk change at different times is different and so we have a heuristic for dealing with that we generate our matrix for last several months same methodology we simply slide the window one month each time and we generate our quality estimates here they here I am showing a you see ROC as our quality estimate with some simulated data and then we looked at 75th percentile that is this little red circle mark Maret a little red red circle and we use that as our our estimate for our quality of our models and now that we have done all this we have a solid foundation to do feature engineering so this is the done right part of my talk till now till we had set up the right methodology for validating our models we could not proceed in a scientific fashion without which engineering because when we started we actually try to fit a model using k-fold and we got really great results but those results were misleading and till we actually fixed our methodology to be right all our feature engineering was essentially worthless because we k-fold methodology was picking into future and we were getting results that were artificially inflated we could never get those it actually those kind of results in the and that's why it is crucial that we that we set our violation methodology the right way now we can start talking about features any questions still now yes sir any tree based model what the drawbacks if like random for GBM so the the there's no drawbacks per se the the linear models actually gave us performance that was comparable to tree based models after our feature engineering and it's just slightly more efficient and easy to fit linear models so but there isn't till there is no commitment till this point to how we fit the model you could use for random forest methods you could use neural networks you could use senior models you could use whatever the methodology doesn't change how you fit the model that function f of X as I was talking about can change but the what we are talking about is the methodology itself for validating and going forward with feature engineering if you would use other model non x rays so the so there are other kind of models we like survival models we would still have to validate them in a fairly similar fashion the feature engineering for them would proceed in a very different fashion but it's still a prediction right it's still a forecast you cannot validate it by not validating against what actually happens based on feature you cannot allow information about what happened when we turn the model we cannot allow information from future so to speak leak in otherwise your results is going to be inflated yes sir was it important that the window be non-overlapping or could have you you have used a sliding window to get a little bit more training data oh absolutely so in one of our applications we actually do use overlapping windows it just was beyond my power point skills to actually write out but it's a perfectly legitimate way to do you get more data yes sir so so imagine if pop had started in October would you've used any of the data before October to Train I'm sorry I suppose Bob right started in October uh-huh right would you have used any data before that to train so I'm not sure I understand your question correctly so are you saying that from let's say today from two let's say that we are making a prediction today while you are saying Bob started in October of 2016 I'm just referring to the timeline you had right he had May 2015 to May 2016 so suppose Bob started in October would you've used any of the data before then to train right for Bob well we are basically turning water based on what everyone a single modify everyone okay so the thing I guess I'm trying to ask is the reason why everybody's leaving is because of Bob so I'm wondering if you are gonna be using any data before Bob joined to Train so I think that was a more clear let's take an example but if Bob joined and you didn't know that he was a terrible manager then obviously you cannot build a feature that Bob is a terrible manager I thanks for sharing your methodology and stopping us fall in some of these traps did I notice the first kind of data at the beginning of your timeline is included like the most often and it's sort of as you go forward it's include less and less do you find that you models have sort of over trained for the earlier data even though it's the furthest away from what you're trying to predict and how do you control for that so one way to do that is we assign different weights and historical data gates those weights for getting the old historical data right cade so that allows us to give our subs less points for fitting historical data it's also actually sometimes important that if there is a data has seasonal patterns to slide back in a similar similar time of the year last year so but yeah the feature is used for the model was it specific to Bob or was it just indicating a change in manager in general for the people so as I listen to example we use that Bob is a bad manager and so in that case yes it was a feature sort of that was based on Bob but we'll actually see what features actually went you know Prussian model pretty soon right I think let me talk about what features we used going forward so all right so all right it's working so we started with a few simple features one of the features decided with this ten year how long employee has been around another feature was time since promotion we had a few other feature features based on number of promotions how long it took you to between promotions number of pay raises was this employee hired requisition the employees performance ratings the managers performance ratings and so on these simple features did not give us a lot of lift and but so the intuition is that a lot of features that are useful turn out to be time-dependent features so let's take example of hired requisition so this feature was included because our customers were very certain that whether employees acquired or not has a very strong impact on attrition we put it in our model and it would show up any it didn't show up with any kind of weight right what went wrong so we started thinking a little bit about it and what we realized was that we had to slice this feature into features like this so times this equation between 0 to 12 months between 12 to 15 months 15 to 24 months and 24 to 27 months any guesses which features were strongly correlated positively correlated with retention or attrition the that which was actually strongly correlated with people staying the feature that are strongly correlated with people living our time since acquisition is between 12 to 15 months and 24 87 months that's where the people stocks best and peep the times acquisition between 0 to 12 months and 15 to 24 months actually are strongly correlated with people stay because you know they have stocks twist another feature that is that turn out to be useful was will you will you or to your window since acquisition happen during what prediction window because there are people who apparently live on the day their stocks best so other kinds of features that are also time-dependent have you taken lots of one day PT O's or time day of days off recently I guess you have been interviewing alright and and then we have interactions and external effects so we have few feature features like parent penetration which essentially measures how well are you come how well are you paid compared to people who are and for the same job title so and historic electrician returned a manager historic electrician return award so we'll talk a little bit more about this feature in particular and so this feature by itself wasn't very powerful but when we started looking at cumulative attrition rates in one year and last two years those started to be more useful and org is a department you can think of it as a department in addition to that a almost derivative of these two features turned out to be very useful as well the attrition rate between zero and one years and one to two years that was pretty useful also useful was a derivative essentially of these two features is the attrition rate in your org increasing or decreasing or staying constant and all of these features turned out to be very useful so in the end we we put together a data pipeline for doing the doing these things right at the beginning we partition our data then we do feature engineering more training and validation getting this pipeline the insuring correctness of this pipeline was a venture and it's the subject for another talk in summary we initially formulated the retention areas forecasting problem as a classification problem using K for validation and the prediction quality did not match our expectations in customer trials and we realized that the independence assumption was broken and we extended the time cease validation methods to temporal validation and now we have two apps in production based on temporal validation methodology for over 100 customers and lesson we have learned is temporal validation results consistently match what we see in reality and also the lesson that we have learned is based features that we include in our models are often time dependent so takeaway is that if you have a forecasting problem of forecasting classification problem be very careful about it methodology in some cases k4 is going to be work just fine for you but at there are times when you really need to look at the temporal validation methodology and so I think this slide is required by law in bay area we're hiring if you're interested please come and talk to me and any questions I have time for questions thank you yes sir did you find that you were able to train a single model that worked for a variety of companies or did you have to train use new training data for each individual customer so the way our legal contacts are structured we cannot use data from one customer with our customer our legal language is ironclad so we have to train different models for different customers yes yes sir hi so you mentioned that one of the other approaches that you were looking at was survival curve analysis but ultimately chose to treat it as a classification problem can you talk about sort of why you made that decision what didn't work about survival curve analysis that and how did you validate that the classification what approach was the better approach relative to survival curve so we basically though let me take the second part first we violated it based on our predictions we made predictions using both methods and this method or not we use my intuition is that the this kind of data this kind of problem is so event dependent that you are quitting because it's two years since your acquisition and so on so including all this time dependent factors into survival models makes them really hard to scale and work but it's a really longer discussion that we can have later why these are all models didn't really work okay I guess I'm out of time sorry and I guess I have one more takeaway from