The Lego Model for Machine Learning

data.bythebay.io: Vitaly Gordon, The Lego Model for Machine Learning

Recording: data.bythebay.io: Vitaly Gordon, The Lego Model for Machine Learning

thank you oh sorry hey guys uh by the way uh thanks for our wonderful uh recruiting teams that gave away socks we love nothing more than people taking picture wearing our socks and not mentioning self for IQ so please do so okay so this is what I'm going to talk about uh today I'll introduce myself and we'll talk about something that is I think doesn't get enough love in the kind of the machine learning Space by the way thanks for everyone U who's here and kind of miss some great uh streaming talks because you know you will not get another chance to watch another streaming talk at this conference and so I will talk about feature engineering how people think about it and what it actually is and kind of more as a buffer of time if we will have I have a lot of things to go through so I will also touch on some Advanced topics and eventually just kind of present a summary of what we talk about so a little bit about me less fancy than uh kind of the intro I just received but kind of a more formal one um I'm a data scientist serial data scientist I've didn't it uh you know long before it was actually called um data science then moved to the valley and I guess now I'm a data scientist and I kind of tweet at big big data and just kind of a uh something I have to say even though um kind of I have a slightly fancy title today um the stuff I'm going to talk about is actually code that I still the second top contributor too so believe me you know I'm I know what I'm talking about at least for that code so so let's start with a little bit context of what the stock is about so when I joined sell force and um I'm part of what's called sulur IQ and sulur IQ is the kind of a a division of sulfur that is in charge of uh creating um machine learning platforms for self uh selfers internally and it's its customers and I do data science and engineering there and kind of the first thing I came to sell for and they kind of told me about is hey we want you to build Mal for every one of our hundreds of thousands of customers and because two issues one is the privacy issues you cannot really cross-pollinate those data sets and second because of uh also performance issues what we kind of found out later so actually the usage of the selfers platform um kind of varies so much and just to give an example from my previous life at LinkedIn I at long I was working on the job recommendation algorithm and for the job recommendation algorithm so people who work at Google or people who work you know a bank tellers at Wells Fargo actually use the platform pretty much the same way so you can build one job recommendation algorithm because you you know you have the same features you have kind of uh the same predictors and you just go and build a single job recommendation uh algorithm this is not quite true itself for because the platform is so customizable that every single customer uses it very very differently because it's uh adjusted to their sales processes in the case of our sales solution and we have service solution and we have marketing solution we have a bunch of those Solutions and the point is that customers use them very very differently so what we found out is that it doesn't really make sense to have a one Global model one size fits all and we actually need H to generate all those models but then even if I kind of you know we hire every single data scientist in the valley we will still not have enough to do uh that work um and create a customized M so we had to build a framework that basically we kind of refer to it as metad data science we had to teach a machine how to do data science and how to automate most of the day-to-day tasks that data Sciences actually do and this is what the talk H is about today so again one kind of takeaway that hopefully you will take is feature engineering is not an atomic operation it's not the one thing that a lot of people say hey I will just do feature engineering it will actually I'll go over a list of 10 different things that fall under that bucket and I will touch on every single one of them most of the steps can be automated and this is kind of the uh the second takeaway so the first is like let's say if you're an engineer kind of never really practiced a lot of machine learning you will you will hopefully understand a little bit more about this you know craft that is called um um kind of machine learning but but if you are a very seasoned uh scientist hopefully what you will take away uh from uh this presentation is how to automate a lot of the steps that we keep repeating again and again and again and the first thing that we'll also something we just found out is obiously we took skull and Spark U because well they're awesome but it's also we learn a lot of the along the way that what we actually did would we would be extremely more uh difficult if we would have chosen of kind of U untyped um you know dynamically type language uh like python or something else and actually the kind of strong type and a lot of the functional characteristic and you know Malcolm GPS covered a lot of it very well in the talk before I did is it's actually great for machine learning pipelines so let's begin what is Automated machine learning and here I'll take an actual example um for a very kind of common data set I hope you can see it but it's it's the Titanic data set it's basically a list of people that were on the Titanic with some uh features on them and eventually the important part here as you can see is kind of one of the things people like to predict about who actually survived uh uh the Titanic and whether we can kind of uh build a model and the reason why I take um kind of um the Titanic example and I will talk about for 40 minutes about the feature engineering that goes into that data set the the point I'm trying to make this is probably the most trivial data set that you will ever encounter you will never encounter such a trivial data set in real life and still I will give now like a 40 minute presentation on how actually you do data science on that trivial data set and just to give kind of the other um extremal Spectrum so when I was uh working on the job recomendation LinkedIn were two people and only to understand senority like for example to learn from the LinkedIn data set that a VP is kind of higher than director was a higher than a manager in the tech industry it was a feature engineering work of two people for about a year so and basically what I'm trying kind of will try to show here that is a lot of the step can be automated here and I will show kind of how it's done and it will be a very very partial talk but again we only have the time that we have so hopefully we learn from it so the platform we created is called Optimus Prime internally and we call it Optimus Prime because it's just a bunch of Transformers get it thank you and basically those are the kind of 10 steps and I kind of took it from a tutorial that someone written very Ally it was first written an r and then someone adapted it to psychic learn about basically taking the same data set and doing a tutorial with those two tools and around how to kind of classify the survivors uh with that data set and I will go through the same uh thing how it works uh in Optimus Prime and I will talk why actually we need Optimus Prime differently the one thing uh I will kind of not talk about is actually kind of the rantom environment and how it works internally because there is a talk given by Li McGuire from my team um that she kind of went uh on in qon very deeply into it there is no point to rehash it if you don't remember that link uh well first I will post the slide the second is just you can L look selfers machine learning qcon selfers machine learning infoq and you will just find it will be the top results in Google so you can watch this talk because it will give the outer aspect of kind of more the runtime environment I will talk more about the design the design decisions that um led us and so one thing to understand about Optimus Prime it's a tool that makes data scientists far more effective and the way we decided to do it is by instead of a data scientists to start writing code from scratch we basically build build them a code gen tool that takes a data set and basically generates a lot of code for them to start as a very very good starting point for the DAT data exploration when they can go and then do more and more feature engineering and all the things that kind of data science go to and by the way for most use cases we actually not even do that extra feature engineering and we just deploy it but kind of the point of how it works this is op stands for Optimus Prime and we you know took the SBT whatever gen idea we give it Titanic Titanic CSV and then D- label there are a bunch of other um bunch of other feature but basically that is how you kind of generate that code generat code I will go into slide because unfortunately like I I tried in a previous um kind of while working on the stock saying hey you know I'll show you the code in live but it was like it was a massive amount and I think it was actually was missing the point of uh there's a lot of stuff in there that the data scientists work with this kind of day-to-day understand and I think it's it wasn't as good uh for the point I was trying to make so I just took some of the parts and put them into slides but basically kind of that is um how it's done and the rest is Magic okay thank you no I'm kidding okay let's go into it so the first step is feature classification and what do I mean by feature classification if you look at this data set again you will see that you know the features are actually of different types we have some numeric we have some categorical we have something in between and this is the first thing that and obviously the treatment that we would like as kind of a data scientist to do for every single feature would would like to treat them differently and a lot of time is like actually going explicitly saying that you know this feature is that and this feature is that that and this is kind of a part of um kind of this is how the generated code looks like and you can read it is and again I took it just to you know to fit into it but you can see that it generated so now we have kind of a real Val that we can actually it's now is something of type feature um in this case is categorical and again this is not code that was written this is a code that was generated that's important point that that is the point of uh would be at all and here you can see that the passenger class was again sorry the age was numeric the class was categorical and survive is actually a binary feature that was generated and you see there is also part of the Builder pattern is as response because we deal with those um the feature that is actually the predictor sorry the response we deal internally differently so this is kind of now um into that feature there's a kind of bunch of Step it's it's a trade that behaves in a different way but this is kind of not the you know not amazing yet but it's still kind of what's great about it is now you can actually address all those feature instead of like parsing parsing string separating by comma doing all this you know uh crappy stuff like no one likes to do is now you actually have strongly typed features that you can do a lot of pretty cool stuff with and I'll show some of them the other thing is actually we have the r features which is the feature that we um we kind of don't really uh see here immediately by the way something I forgot to mention the reason why we have here this Titanic is kind of a legacy unfortunately we still kind of haven't fixed it is because we first implemented for AO or case classes and then we kind of in order to make it work for CSV we first convert the CSV to kind of case classes and then apply it but basically that's the same so dve features here you can see there is this um home destination column and the home destination is actually is not just a string right it's actually there is a little bit more logic to it and this is something that Optimus Prime actually captures as well so first of all it just this is the same feature that you could see as well it's just categorical but in some cases and this is kind of a statistical method that it goes through in this case is bunch of rexes with some prority and it tries to classify every feature and you know if it sees that over certain percentage of the features matches to some pattern that we recognize then he actually goes and creates those features as well again not something that is mind-blowing by itself but it's very very useful because people don't need to do it themselves and here you can see that Optimus Prime identified that we have a CD and you know it extracted it with a reg extractor and it recognize that there is a state and it extracted it with the state and the reg looks something like you know there's the if there is you know string comma and then one of the 50 acronyms for state or their full names then it kind of captures those two variables again not mindblowing something that you know you requires a PhD in machine learning to come up with but it's actually very very useful because on kind of larger data sets you have more and more of those types and because now this is actually even though it's like categorical it actually builds a a feature of type c feature of top state is we will kind of use that information because those features actually have additional functionality that just a categorical feature does not have which is kind of cool missing values this is something that again data science data scientists deal with u every day and the problem with missing values there are multiple of them one of the problems with missing values is that like algorithms just you know if you take a just spark ml logistic regression it just does you know it expects to get a dense Vector it just doesn't have like a vector of options like you need to get it like a vector of numbers because that's how it works or data frames or uh whatever some algorithms just don't deal well with missing value so there is the Practical problem the second problem is actually when you have many missing values like you can see on this it just m it might be that um in that case um the you will get better performance if you actually fill in the missing values it's just a performance so one of them is just practical consideration the second is a performance consideration you will get a more accurate model if you actually do something and not just have uh Missing vales because again uh decision trees Ren Force there are some machine learning algorithms that deal with missing values but even them if you uh do some uh stuff around missing Val is they kind of do it more accurately and this is some of the code that is generated for it so you can see that we have the age value that was actually defined before and this is kind now like a different class that I'm uh I'm showing and what it does is just we created the you know it has a method that called field mean which create a new value which is Field age so the second line here is just a synthetic sugar you can achieve the same which is basically what it does is age. transform and then we have a Transformer that to the Optimus Prime thing is you know like it has a pretty long name mean numerical imputation Transformers we have statisticians on staff that are very proud of their craft and they like very verbos naming we need to work on it but uh the point here here is you can see that for example in the age and this is by the way configuration so this whole thing this whole Cod generating has a pretty big configuration file and for example one of the things you you say in the config configuration is that the appropriate way to fill in missing values for numeric Fields is use the mean Transformer the proper way to fill um the categorical fi which is that home destination is to use the mode transformer for those of you who are not statistician mode is the most common term in a data set so now we have and again basically there are more of those along and kind of in order to keep the code a little bit shorter we actually generate those lines only for um only for the features that actually have missing values so if a feature doesn't have missing values we'll not just you know call this function to do nothing okay next so now we come to the kind of transformation and this is where probably most of the uh kind of uh work uh happens in coming with way different ways to do kind of more and more uh Transformations so for example some of the transformation that you can uh do are taking the AG column and normalizing it uh that is uh one thing so why would you want to normalize a variable so for example for algorithms like logistic regression um that are basically kind of this weighted average across multiple features in order to make sense of the weights after you apply the algorithm uh you want um uh you want all your uh um kind of feature or all of your values to be normalized let me repeat it again because I don't think clear enough so like for example um if we use um if we use two features and both are weights but one is in kilograms and the other one is pound so the the weight before the pound because pounds are just you know higher just bigger numbers that represent exactly the same thing so the weight there H would be bigger than the the weight before the kilograms because it's just you know linear operation you multiply one number by the other so in terms of being both of them equal you will need to multiply the kilograms by the high by a bigger number than the pounds however uh because people like to understand the meaning of those numbers uh a lot of people when they look at a linear regression equation they want to see which U which variable which feature has the higher weight because it also kind of means that oh this feature is kind of a again there's some hand waving please don't K like statistician in the audience don't kill me but it kind of implies that this variable is more important so in order to actually get this added benefit of understanding which feature in your data set is more important than not you need to have all your numeric variables in your data set actually be normalized to the same thing and I try to explain it two ways if um if it's still not clear please blame me for not explaining it right it's um yeah I need to be better at it but going to it is we actually have several normalization uh sorry several normalization techniques so one of them is just normalize the other one we use is actually bueti which is also uh very common H we use it both for categorical variables and we also use bucketization for kind of numeric variables in this case is fair so the question is why would you want uh to bueti a numeric field so and the thing is again the example here gives like for example if you take um your weight as a function and then you try to predict uh you know uh how uh far you can throw a javelin then you will see that it's actually not a linear function because obviously you know grown men probably throw Javelin farther away than kids so in that case it's kind of linear but at some point where you know let's say you become morbidly obese it doesn't mean that you kind of throw javalin farther away than a person that weighs four times less than you so I'm saying the in most cases in life actually the linear um relationship between kind of data sets are kind of does not hold forever and this you probably would want to make kind of bueti even kind of the weight which is a continuous variable want to bucke ize it for you know the weight weights of children and then kind of the weights of adults and then the weights of you know adults that are maybe kind of um over the weight where it becomes kind of optimal because there is probably some optimal mass of Javelin fls or or something like that and so in what bucket I does is basically takes a variable whether it's categorical or um or sorry continuous and basically breaks it into n um into kind of n binary variable so basically once we deci side of the bucket then that uh feature will now represent whether um whether the kind of U you that record is within that bucket so for example if our in my example the weight is you know between Z to 100 then it will be a binary saying hey whether that person's weight is between 0 and 100 and so on and so forth and the thing to see here is again yet another configuration thing that actually with we found out that uh the way to bucket as continuous variable because like the mean elements and max depth like we shouldn't go too much into it because we can spend an hour only talking about like proper bucketization techniques but the point here is that it's kind of based our implementation on decision trees and that's kind of parameter that what we found that kind of works around finding kind of good bucketization for the features that we use and again that is all generated and people can just go and change that number or or change the configuration if they find that they need so okay augmented features which is this is actually I would say that is we still haven't implemented that but it's just I saw this cool demo about a company called spark Beyond and I said oh my God that's like so cool we like we need to have it as part of our but this is kind of so this is the part that is uh does not really exist but I I want to say that this is definitely a part of what data a lot of data scientists do that once we identify that we have our home destination is city and state we can go and pull a bunch of stuff like for example and this is again something data scientists do very often they uh take and you know you can look at the weather so maybe the weather from the city that this person from had some play in around how well they deal with cold you know like uh the cold water where the Titanic sink and maybe that's a predictor I'm saying the median income might be again another variable you can take their zip code and transform it to the city go to Wikipedia find the median income or some other data source for that City and you can kind of create bunch of these features once you have better annotations around your uh variables and you know like much uh richer types which is kind of what we have than just string or categorical you can actually do all that transformation about going to uh different data sources and just pulling almost kind of everything that you can think about um from that U from those data sets and this is kind of a fairly common technique that data scientists do but it's actually an extremely tedious manual process and this is something I would love for us also to do in the in the platform automatically the other part that actually does happen and quite often is there are interaction features so what are interaction features is sometimes a feature does not uh um kind of M Mak Sense on its own but if you combine um combine two features together uh they start making uh kind of uh more sense and one of my favorite examples was uh from a PhD dissertation of a LinkedIn colleague of uh of mine um she did some bird migration patterns and she had this like beautiful visualization to show that if you look at kind of bird coordinates over the course of a year um it just looks randomly it just spread all over the map and if you look at weather it's also you know spread all over the map like you know every weather exists at some point of the year at every place when you take those two uh variables together you actually see a disting pattern where you know birds are actually kind of in the warmer places kind of all the time and you by combining those two variables you actually see a pattern that without those two variables I tried to find an example I couldn't find something that's very good and hopefully we find out but in this Titanic use case is actually because again we run it the gender um and age so this is basically we're taking a categorical variable we kind of multiplying it and this is not the you know the algebra normal multiplication operations like I said our features have those kind of added functionality one of them is a multiplication so we can actually multiply features together so when we multiply age by fair is just a simple numeric multiplication when we take a category variable and multiply it by a numerical variable it actually creates another categorical variable which is kind of the concatenation of those two variables and kind of gender and age we found that actually um kind of works pretty well it does add additional information that you don't get by using only gender or only only age the problem with that and by the way the augmented features examples um I showed before there is kind of um this huge problem that if you create every single feature combination that you can think of and this is by the way I think there was like nine features here so the combination you can um kind of do the math I think it's a little bit more than nine but it's like several hundreds of those uh features and probably can be you know several hundreds or thousands more from the augmented if you pull anything anything you can think about a city about a state about an age about whatever you get to just a variable explosion so besides the com the computationally um hard nature of that problem there is another problem that you will start seeing things that are do not really exist and this is one of my favorite examples is there is the website that you know just finds correlation uh between random things in here case you can see that the diverse rate in main correlates with per cap conception of margarine in the US and the correlation is .99 which is like one is the maximum so basically it means it's perfect correlation and believe me if you throw and the whole point what this websites does is just takes you know as many um you know as many data sets as possible and then basically you have you know the the cross product between all those data sets and you'll find one that is you know has a correlation of um kind of U 99 and this is something that you have to be extremely smart into avoiding and I I'll I'll talk about how we deal with it but before is this is another very common problem that we have to deal with as well and this is what's called label leakage as you can see even in this data set we actually have two additional columns and the column one of them is boat which probably you know if you were in a boat you probably survived but even more there is a body which represents some identification of your body so you know if you have a buddy number you probably did not survive and that's a perfect example of label leakage but if you think that this is a contrived example believe me it's not here's another example where you know after you buy something you get a thank you email for buying something and there is a lot of people that say Hey you know let's C you know let's look at your email activity for e-commerce things and then you see oh yeah you know you create this thing everyone who got this email that has this war thank you in it suddenly um you know is very likely to buy but this is a perfect example of label leakage basically um relying on data that will not be there in time of scoring and there are many ways it's like it's actually an important problem for in in inside cell force one problem that we have that for example uh when uh customers sorry when clients close records saying hey we closed that deal then they go and add bunch of information about about that deal and then I say well you know if the deal has zip code it's very likely to close but no it actually happens after and because it's like you know the crud database actually don't have the time stamps and now I feel you know I create there is an not like perfect label leakage there are more and more examples so we kind of deal with that U automatically so this is kind of the code that actually eventually generates the the model and you can see here there is something that's called sanity check true and what the sanity Checker does is this like deals with this multic linearity wow that's a mouthful issue which is basically trying to find perfect correlation between any two variables and removing them uh from the data set another cool thing that I will touch uh very shortly is please take notice of why the positive response ratio is an array because I'll get to it in a sec so this is kind of goes part of the resampling and this is one of the problems if you think about like for example fraud fraud detection problem the in fraud detection like the likelihood of actually transaction be to be fraudulent is extremely low so let's say you know it's like you know 0.01% of your actual transactions the problem with that is if you have a model that says all your transaction actually nonf fraudulent this model will be 99.99% accurate which is great for any real problem but it's actually obviously it's completely useless model right and this is one way one problem that we have to deal because around our hundreds of thousands U hundreds of thousands of uh customers we sometimes deal with customers that whatever the event we're trying to predict like 0.1% of likely and and sometimes it's 10% likely we actually need to build our training set accordingly and the way we do it we actually try to go through uh several of of those and we actually compare so in this case it's pretty kind of dumb it's like research we just try all combination see what what works best but I will go even further because you will see we don't only look at that parameter we go to in model selection which is this is a kind of psychic Laur they kind of classifi the model but you can see there are multiple models that we can work and what we found across all many many customers is actually sometimes a linear model works better sometimes the forest model works better Sometimes some other model and we actually don't really know and the thing is we actually say hey we don't have to know what we can do and this is kind of a wrap wrapper of sparkml Pipelines is you know why not try multiple things why not try logistical regression with this regularization peram and this elastic net perams and why not try random forest in you know with bunch of these pams and basically it's a cross product of all of these pams and then once we actually try it this is when kind of we have this function called try models you know we give a training set test set models and this is where we set the metric and the metric a or C um doesn't really matter what it is um kind of goes and create all those all those models we actually smarter than you know just going brute force uh through those and there is bunch of uh bunch of packages that kind of do it for you including U I think sperman and U bigml and some some others that kind of do a a more a smarter search around that space that kind of converges much faster than just trying all options so I definitely you know would recommend to do it what's my timing just 5 minutes awesome uh so I'll skip the advanced topics because I really uh would much prefer um for you guys to ask me any questions you have so just kind of to uh wrap things up um uh this is what kind of I want to say so machine learning is kind of very complex and very complex I'm not saying you know like no one can do it I'm saying just it's not one thing there's like a lot of things that some of them are kind of domain expertise some of them just you know stats knowledge some of them just like experience of you know knowing what a label leakage is and what are the various form it can manifest in and to be prepared for it most of the steps can be automated like I said all the code you've seen is code that automatically generated it actually achieves very good results even without uh touching it so we have a benchmark where we have like I don't remember a couple of dozens of data sets and the way we iterate over platform is you know we have this uh uh kind of regression test that every time we improve improve the platform we go and kind of run it across all of these and see what what improve and again while I didn't touch it and it explicitly believe me everything I showed is possible because a a you know spark lets us deal with scale and Par with paralyzation and Scala give us this wonderful uh kind of T type safe static uh system and really that that make made our life easier so kind of here is my shake if if you're interested if you're curious to know more about how Heroku prediction IO the top schola project on GI Hub and metamind fit together please come talk to me to any of our recruiters any guys you see in the audience wearing the IQ t-shirt we are hiring and thank you very [Applause] much so the person who asked the first question question gets another pair of socks here sorry sorry about that so want to ask you there sorry there are new socks so we we haven't used them about the training do you do the training as well here or do you do the training uh outside of this plat sorry you you're asking whether we do the training the training of the model yeah like like doing the exploration of the model no so definitely like uh what I showed here is if where is the clicker it will probably take forever but as you see Val best model the best that Val actually holds the best model eventually so it's when when you run it that is the best model and then you can decide like I I don't show the code here but you can either write it and you know score with it do a bunch of stuff but it's actually holds what you know after we train all the mo all the combination of the MS that will have the best mall in it Yeah so basically that thing goes into production what the one thing I and this is you can see at Leah stock and having touch with it is actually we are you need to remember that in sulur context we actually that thing will then get applied and will generate a different m per customer so it means we'll have if you know in the linear sense we'll have a different weights for every single customer so that thing will be then applied many many many times kind of in parallel fashion yes there very nice presentation first of all it's a different uh thought process Al together my question here is um can I safely assume the overall model is um The Factory uh design pattern plus the old reusable modeler program to generate a code uh to do most of the things in automated fashion than building something scratch so sorry just to make sure so you're saying could could you assume that kind of all that code is a kind of the reusable fashion to generate all those models right plus they're using the factory design pattern models to to compartmentalize um those yeah so this is basically kind of the thought behind it and you know while not to be too philosophical is kind of bringing the same engineering practices that exist for you know decades on engineering to data Sciences for for example all the features that are gener at are actually part of this object that you can then extend and kind of create more and more because Exel for is one of the good things we have at least uh we know the data space that we're dealing with and then we can uh uh we can already create bunch of very sophisticated features on based on this and then you as a data scientist you can extend that get all that wonderful functionality and then kind of apply uh again it's really the you know objectoriented uh um model where you just extend the class and you get all that functionality plus you can and you can remove from it and you can add your own features yes there is a question here yes uh so in this we are looking at a very small data set uh pretty straightforward and simple data set um but when you want to do uh in production and at scale what are the some of the things uh What uh you have experienced or um seen to make sure the scalability is managed for these machine learning models so the reason why I run data science and engineering is unfortunately here's the data science part my larger team is actually engineering to kind of make sure that this thing runs and scale so first of all uh the thing I don't show here all these Transformers actually and again I'm going to be go geeky here they're actually all implemented as monoids and in order to kind of Ensure parallelism across and we do it in spark it's so today it's the Legacy we use rdd I can't wait for spark to point zero to kind of try the new uh data sets but the point is it's everything that is generated is a spark code and then on top of it we also have a very smart because we run thousands of those models in production very and they run on Amazon on different cluster so there is also a lot of very a lot of sophistication that the engineering team build and again if anyone interested please come to talk to us is around um how to take the data and the size of data because they vary tremendously we have you know customers that are free or magnets larger than other customers is to take that data and realize what is the size of the cluster how we tune the spark jobs and a lot of that goes into and that also all fully automated yes please so I'll repeat the question the question was whether there was a feature reduction space so definitely so this is if you remember there was the sanity Checker equals true so the problem with the feature reductions again uh in get you in the right mindset the feature that will be relevant for every single customer will be different so that is something that in the runtime environment happens is we can of that is the template and then the sanity Checker equals true is basically reduces bunch of features some of them just based on pure correlation with a label that is too good to be true and there is like label leakage or multicolinearity if we have two features that basically are extremely correlated with with each other and some of them are just because you we want want to um have the jobs run in a more performant manner so we just reduce to the in the like the model that starts is probably has like thousands of features the model that eventually gets it's probably dozens of features so there's like a very large reduction space reduction okay I'm sorry so um apparently I'm out of time but please come see me outside I'll be happy to answer any more questions you have thank you [Applause] ital a