scale.bythebay.io: Shiva Chaitanya, Scala DSL for ML Training Set Stratification
Recording: scale.bythebay.io: Shiva Chaitanya, Scala DSL for ML Training Set Stratification
I'm Shiva I work for the personalization infrastructure team and Netflix I'll be talking about a domain-specific language that we built using Scala for machine learning training set stratification as all of you are probably aware the Netflix homepage is a highly personalized experience for each user the ordering of videos from left to right in each row is personalized for you and so is the ordering of rows which appear in a vertical manner recently we also released a feature that personalizes the thumbnails that we show for each user to different people being self is the same video can have completely different thumbnails depending on their interests that Netflix has learnt over a period of time over 80% of what our members watch comes from all these recommendations and these recommendations are underpinned by state-of-the-art machine learning algorithms that we periodically train we continuously iterate on the machine learning algorithms and the data that's being fed into them and feature engineering and so on and what helps us in doing that efficiently is we use historical data that's captured by our snapshots infrastructure which I'll allude to a little later which enables us to rewind back to any arbitrary point in time and rerun those algorithms and probably a better algorithm to see if we could have done better with a slightly different tweaked algorithm and that enables us to identify the efficiency of an algorithm which all without actually going to production or even doing a/b tests which could be a long-running process and after we have promising results from an offline experiment we are then ready to go to a B test and we run a B test for a while Netflix is a extremely data-driven company and based on the results of these a/b tests then we go to the general production so here's a simplified view of how a pipeline looks for a machine learning engineer you start with the hypothesis you run an offline experiment based on the hypothesis you gather all the data for it you model the experiment then you analyze the results and typically this tends to be a iterative process which goes on for a while and depending on how the offline matrix look like then we go to an a/b test and if the beta succeeds then we productize the algorithm when X experimenting offline during the iteration process we want to do it quite quickly and when we go to production we would like to use the same type of code and we like to have great code throughout the pipeline and for that Scala is a wonderful fit for us it is in fact our defect of choice for for our machine learning code and Scala is our sweet spot really because it matches a lot of the criteria that we look for its type safe which means you know a developer has bound to make introduce fewer bugs it's clean functional immutable data structures allows for higher productivity unlike some interpretive languages it's compiled into byte codes and it's fast which leads to better performance and quite crucially for us it runs in a JVM which means it's compatible with our online services and most of our online services or Java based and last but not the least spar Scala is the first is the first class citizen in SPARC and we are heavy users of spark for all our offline machine learning jobs so earlier I showed a simplified view of the pipeline in a linear fashion it's a little more complicated than that so let's zoom into that a little bit we have a snapshot infrastructure which is managed by our team which runs a sequence of jobs once a day we stratify what's called a set of user segments or user cohorts we collect a bunch of these different user sets and then we go ahead and capture raw data for them often referred to as facts and when there is a new experiment that needs to be worked on by the machine learning engineer typically the cycle it's an iterative process it starts with the design of the experiment and then the experiment needs its own label data set and from the label data set you stratify an appropriate training set that's valid for the model and then the facts that we captured from the snapshot infrastructure are fed and as raw data and we generate features out of them then you train a model then you compute validation matrix on it then you test the model and if the offline matrix can be further improved you keep doing this in an iterative fashion until them until the model is deemed enough to go to a B test at which point we allocated an a/b test and hopefully at the end of it they be test wins and we go to production most as you can see here the snapshot infrastructure and the offline experiment is all mostly running in spark so in this talk I'll be focusing on the highlighted boxes here the stratification pieces which sees its application both in the snapshot infrastructure as well as when stratifying the training set for our machine learning models so why do we need stratification it's for the snapshots infrastructure it's required to place constraints on what type of users we snapshot while at the same time we have to ensure that we get the maximum yield when we join it with the label data set to produce a training set we do not snapshot all the users at this point so stratifying it intelligently based on user attributes is a important requirement for us stratification is also required just before the model is trained to select the training data appropriately if we were to randomly select training data then we some of the important demographics that we care about may be under represented under represented you know we are often interested in small countries emerging markets and new users and we would like our model to adapt faster and better to behavior from those select demographics so the certification library that we build is a domain-specific language which is meant to be used in the scholar and spark ecosystem and it places quite a bit emphasis on the type safety nature of it it contains an expressive API with which our developers can build arbitrary user sets on demand typically most of our use cases have been we have been able to translate them into queries using our library we use a lot we use a bunch of user attributes that are present in the query that people can stratify by you know you can stratify based on the user's country based on the number of plays we have seen for the particular user let's say in the last week number of searches number of devices that are registered for the user the mem the membership tenure and we support a lot more so as I mentioned we have a domain-specific language so I'll be talking about I'll be showing some examples of the building blocks that are used to build a craze that are part of the DSO so country is a case class and within that there is a specific object called country dot u.s. so that represents all the a part of the universe of all our users who are from us so in a Venn diagram you can imagine this the circle which are all US uses 10 euro dot m1 refers to all users who are in the first 1 month of membership and that's you know it overlaps with the user so in the Venn diagram space you can be sure that I said like that and you can use these building box to build further you can call out more user cohorts for example you can combine 10 euro dot m1 with an odd condition on place 1 comma 10 that indicates all users who have had number of place between 1 to 10 in the last day so you can mix and match like this with then we support all the boolean operations for it here's another example all users so not from us but I've had at and also have had at least 2 devices you stood on them so really using this DSL a developer can construct whatever use a cohort they have they they want and then they can build a training site out of it so using using the building blocks you can you can form queries where for for each of the for each of the building blocks you can specify a certain target count that you want in the results stratified set so in this example I'm showing that using our API you can say that you know starting from the universe of all the users give me a subset where the number of us uses is about five thousand the number of users were within the first membership first month of membership is about thousand I'm showing purely hypothetical numbers here none of this is what we really do not at a liberty to talk about our exact stratification scheme so take this synthetic data so you can build queries like this so it takes in a map where the key is is the building block and the value is in this case as a target there's a target count and we apply these rules on our universal data set and at the end of it it produces a spot data frame that contains all the all the users that match the criteria so often target count doesn't satisfy all our use cases so we typically tend to use percentage so you can specify it as a percentage and at this point what the developer is really looking for is give me the maximum use give me the maximum result data frame that matches this percentage criteria right on top of it they might down sample it to a smaller size and the DSL our implementation figures out the runs runs a constraint solver that tries to match all these rules and tries to figure out the biggest possible result data frame that matches all these rules so we use target percentage in win percentage and max percentage and you can you can specify some of the criteria that way this is an example where the dsl can be a bit more convenient where the user can specify for example country dot each as one of the building blocks and what the implementation does internally is it enumerates all the possible countries and expands the second rule into exploits the second rule into all possible combinations of it so country dot each and place one will be automatically expanded to each each of those countries and place in place one and we will try to satisfy the rule that all of these all of these individual building blocks here have a max percent of point two so all of this has automatically taken care by the stratification library so what tends to happen is as the rules become more and more strict and you're asking for a specific percentage for each of those rules sometimes you may not get the actual number of of users in the result data frame because as the rules become more and more stringent you are trying to fit a different distribution to a slightly altered distribution so you may not get that yield that you are looking for and we have an extra parameter called allowed error margin that allows you to stray away from that goal by a certain percentage and what that tends to do is increase the yield for us and we sometimes play around with this parameter so we also have a convenient way to refer to a distribution that exists in this in a different data set it's it's not sometimes the developer is not clear about the exact percentage that they are looking for each rule so they might refer to a distribution that exists in a slightly different data set so so we have a parameter called reference where you can pass in a different data frame and then the stratification library will calculate statistics on it and then replace all the target person dot refer by the corresponding percentage that exists in the and we have several other examples and the DSL is continuing to expand so that in general the country in the constraint solver especially when percentages are involved and if the rules or overlap it's a non-trivial optimization problem and there's no closed-form expression by which we can solve it so we formulated it as a linear programming problem it's actually an integer programming problem but you know in our experience we found that mapping it to linear programming gives us an optimal so close to an optimal solution and we used a third-party scholar library called optimist and I just want to provide a shout out to them so the API in the third party back package was pretty idiomatic and with just a few lines of code we were able to formulate all the equations that we had in paper and we were able to translate that into code so this is code snippet of how the linear programming problem was formulated so the domain-specific language is talking about is currently being used for our user sets which are essentially all the you know we store it as a spot data frame where each row corresponds to a user and a specific set of attributes for the user but it's possible to extend the country constraint solver that we have to any generalized spot data frame for example if you have a data frame that has three columns column foo bar and cukes you can extend spa data frame using an implicit method and provide all the rules that I previously showed in this manner for example you can select an arbitrary column and you can apply very similar rules that you know I showed you earlier and this is quite generalizable and we are thinking of open sourcing this so that you know everyone else can start using this and the DSL we are also porting the DSL to start to be built on top of this generalized framework so we can you know support it better and identify bugs much earlier okay so let me quickly switch over to [Music] Zeppelin notebook it's a font okay so yeah I just want to say here that this data is completely synthetic once again don't read much into it in fact it's fake and I randomly generated it so here's an input data frame and assume that it has four different columns and so what I'm doing here is this is an example of how a general sampling rules on a generalized data frame would look like I'm using a pretty strongly type non idea Matic way of expressing the sampling rules but we are we are we also have an idiomatic version of it which we are improving and we'll open source open source as soon so for example the couple of rules here say that say that from the input data frame make sure that all the all the users from country us in the result data frame is about 30 percent of it people who are not in the country u.s. or 70 percent of it so the input data frame might have a different distribution and when this query is run it invokes the linear programming library that I was talking about and the logo that you see is from them and I haven't been able to suppress it so it's actually promotion for them I guess and as you can see the constraints are being solved we started off with about 1.6 million rows and after stratification we ended up with 1.2 6 million we started off with a different percentage distribution of the two Venn diagrams we asked for a target percentage of 70/30 and we mostly met it with a very small error margin and I have a bunch of other queries in this notebook which I can probably show you off line that runs more complicated queries with you know different and or and not conditions so cool so just going back so that's the end of my presentation Netflix is hiring my team is looking for candidates I have my manager sitting there if you want to talk to him so I reached out to us on Twitter understood any questions and I'm not at a liberty to talk about all the metadata that we use but some of the examples that I showed is very close to what we use in some ways there's that you might use things okay so if it's possible to construct the more complex metadata out of the individual building blocks then our API supports it but I'm not sure about the weighted vector stuff that you're talking about maybe you can take that offline yeah it's quite interactive as you can see this query took about 27 seconds and if the it really depends on the data size and how much resources you have given to the cluster so yeah it is it's runs easily in a notebook and you can tweak the queries queries yeah yes yeah the open-source contribution that we are thinking of currently will be based off of completely denormalized data frame at least initial version of thank you [Applause]