data.bythebay.io: Justin Basilico, Recommendations for Building Machine Learning Software
Recording: data.bythebay.io: Justin Basilico, Recommendations for Building Machine Learning Software
thanks everyone for coming to my talk today so I work on one of the teams that report that provides the recommendations for the Netflix service and so working recommendations I often get a lot of people asking me about recommendations for what they should watch on Netflix you know what what are the new great things to see or what's you know the best movies that are there and well I can kind of you know give some guidance they're very biased by the types of you know movies and TV shows that I like so I just feel a little guilty when I do that but I think I'm happy here to talk about such subject I hope I can provide good recommendations about which is how to build machine learning software for real production applications and this is something that we've done a lot of work on at Netflix because we use machine learning deeply and also something that even my previous job at the labs I did a lot of work and I'm hoping that what you can do is really learn the from the lessons and the problems we've encountered and you don't have to run and see the same problems too and kind of can get things going on in the right direction and some of these might kind of be pretty straightforward but they're I think they're very useful to kind of help get to help you kind of build your software in a good way so first I'm just gonna give kind of a background around you know kind of where Netflix is that right now and particularly about the recommendations because that's obviously where we have a lot of experience with our machine learning software but I think a lot of the the ideas translate to other machine learning applications as well so when I talk about you know machine learning and Netflix the first thing usually comes to people's mind is a Netflix prize and the Netflix prize was launched in 2006 to help to basically have people try to predict what people are gonna rate movies on a five-star scale now back in 2006 Netflix is primarily known for shipping little red envelopes in the mail in the US and so it's a very different product then we fast forward to 2016 a whole ten years later Netflix is now a global instant video streaming service we've gone from you know just buying and shipping DVDs to even producing our own Emmy winning content and you can watch Netflix on pretty much any device that has a screen or can be connected to a screen so as you can imagine the chain we've really changed a lot we do in terms of machine learning and it's much beyond just simple rating prediction so what kind of scale does netflix operate at now we have over 80 million members we're in over a hundred and ninety countries which is basically entire world except for China we're over a thousand different types of devices we people stream you know billions of hours of Netflix a month we eat up over a third of peak downstream Internet traffic in the US and if you want to make sure that you're really getting the most out your internet service provider this week we launched a little tool called fast comm you can go there and it'll tell you just how much of your bandwidth your ISP is really providing you by hitting the Netflix servers to see that how fast the contents coming down so in this new world new scale you know the the gold recommendations is really changed from rating prediction to a more kind of holistic goal of helping members find the content that they're going to want to watch and that content that they're going to enjoy to increase their satisfaction with the product and with the content and then ultimately then if they are happy they will retain with Netflix and because we're a subscription company we'll be happy because they'll pay for their subscription and then we can go again buy more content build better algorithms you know how to make a better service so what's our general approach to recommendations nowadays really everything you see on your Netflix device across all different platforms as a recommendation so this includes both the set of rows you see things like dark comedies frittering a strong female lead or because you watch house of cards how we picked those rows and order them on the page is done in a personalized way in addition how we rank all the videos on each row in the page and across the site in various ways is also done as a recommendation problem a personalized way so over 80% of what people watch on Netflix comes from the recommendations and we've gotten to this point not just because we think you know machine learning and recommendations is cool but we found time and time again that it actually improves the user experience by improving how much time people spend watching Netflix and how many hoff - how long people stay customers and then the reason I'm talking about this here of course is that the recommendations are totally driven by machine learning algorithms so right think about machine learning in the context of you know solving problems that Netflix we really try to break it down into you know five different aspects and when we work on them we can really think about improving any of these aspects to make the algorithm better and the first one is obviously the problem statement what is exactly you're trying to predict the old world we you know might have focused on trying to predict ratings but that's not the actually maybe the best problems to go after there's a lot of different sub problems we can tackle and if we pick the right one that might be more useful or tweak how with the definition the problem we're going after we can find that that's actually probably the most important thing you need to just do when you when you try to build a machine learning software the second thing is once you figure out your problem you want to go and make sure you find relevant data right so figuring out what features you're gonna put in you hire in and collect your labels and put them together that's again super important to make sure you're gonna have an effective system from there deciding what model you're gonna train you know what algorithm you're gonna use to train it and then once you've picked those based on the data you set you've collected and then computed metrics see how well it does and then from there you know iterate on all of these things and and improve them over time so machine learning for us is a very iterative process it involves improving things on all of these different dimensions not just say the models and algorithms that you know a lot of research and papers are written about and we find that a lot of benefits can be chained it can be found by changing the things kind of on the top part of the diagram versus the bottom of course though everyone wants to know about oh yeah but what are you guys using what's that magic bullet that you found that solves machine that not that we solved recommendation but you know that we use and the reality is that there is no magic bullet the real reason we take basically use everything in the textbook and the reason that these things are in textbooks is they're all very good algorithms for a specific subproblems so in different setups you have so simple things like linear regressions with just take regression I have they're they're great use cases things that we use a lot of matrix factorizations and extensions like SPD's SED plus plus we use other extensions too like factorization machines in the neural network space you know our bm's was something that was from the Netflix prize that we use deep neural networks we also try using Markov model of graph algorithms clustering Lda ensemble forests Gaussian processes the list goes on I think pretty much the only thing we mean don't use right now is svms and being someone who really studied a lot of kernel stuff in grad school it kind of makes me sad and I'm sure we just haven't found that right perfect application for it yet so when we try to think about taking all these models and putting them into production there's a lot of design considerations you have to take into account both on the side of your the product the recommendations and then also on the software side so we need our recommendations to be personal I mean we could throw in you know um personalized popularity and that that might do you know they better than you know random or alphabetical but that's you know we can do a lot better than that so and we particularly want to make recommendations accurate so by using personal you know rekka by using algorithm the person sizing we can try to make very accurate guesses about what people want to watch and build our and display those but you know being accurate isn't enough you people have a wide variety of interests and we need to make sure that we cover a diverse set of them when we're showing recommendations so they we can't just say well you watch comedies ninety percent of time and documentaries ten percent so the most likely thing in watch is comedies Muenster only the others you have to make sure that we balance the diversity of the person and also showed the diversity of things that maybe they haven't expressed it on the service we also need recommendations to be novel if we're just showing you know the same things you've already watched or very popular things that you've seen in other places the recommendation system isn't very useful so making sure that they're novel and and help you'll find things that are new they wouldn't be able to find otherwise is very important finally which is very important for the software side is the recommendations have to be fresh that to be responsive to new actions a new data the user provides us and also what's going on in the world around it so that we can quickly respond to new new events and new trends to keep the recommendations up-to-date because still recommendations look really bad from a user perspective on the software side we have to deal with the scale we run at so we have you we run primarily in AWS and that provides a lot of nice scaling capability but we really need to be able to run make sure running things that on a scale works for the whole user base we need to be responsive so if we put something in the compute path people are sitting there waiting for the recommendations to come back so it needs to be able to happen really quickly because we run in the cloud especially it needs to be very resilient so if the systems will go down and you know we can't take down the whole Netflix site we need to be able to handle failures in a graceful way we have to be efficient because you know we are a business we don't have infinite resources so we want to make sure we're not you know wasting a lot of extra computation power that we don't need and finally we want to be flexible so building machine learning is great iterative and requires a lot of research and trying things out so making sure that the code and software is very flexible to support that is important and of course we're looking for is that you know happy intersection where we can kind of meet both the needs of the software and the recommendation application luckily we have a great software stack that we build on as I mentioned we run Netflix 100% in AWS but we also have open source a lot of technologies on top of it if you want to read about them you can read it on the Netflix tech blog they just scribe a lot of them but we run primarily in sort of a Java ecosystem and production and that means that in the backend we also tend to things like spark and whatnot so where are my recommendations so the first recommendation is be flexible about where and when computation happens so here's a blueprint of our system architecture I'm not going to go into too much detail but the main thing to see here is that we have kind of three different regimes on line near line and sorry off line Nearline on line so off line is or we typically do a big batch processing so I run our big machine learning algorithms and we have things like spark and Hadoop that we can use on line is where we put things that are in the the actual user request flow so synchronous computation that we have access to the most recent contextual data about what device people are on what time of day think requests are coming in and then in between we have this near line system where we can update based on asynchronous events coming in and this is a great place to put things like online learning and then evaluating model parameters that are user based so to walk through kind of an example of how to take an algorithm and then think about breaking it up to be you know a place that to get kind of the optimal performance you want in terms these characteristics I mentioned earlier we I'll kind of walk through this example of matrix factorization so for matrix factorization we might start off with a sample matrix where you say collect sample of a big matrix of user by video data from the big play data archive that we have offline we then run some algorithm that would decompose that big matrix into you a small user by factory matrix and another small item by factor matrix of course with some regularization terms and once we had gone done this big batch algorithm what we do is we typically just throw away the user factors that we used in that sample and then only publish the video factors and then in our Nearline system where we get events about someone has you know started or stopped watching a video we then can take the item factors we have and then resolve to figure out the the most fresh values for the user latent factors in the system and once we do that we can in Nearline also compute the again the dot product of the user item factor to get the set of recommendations for the user and then store them in a reliable cash like eb cache which is our layer on top of memcache D and then once the next request for the homepage comes in or we send a polling request back to have the the device pull we can in the request loop do any kind of contextual filter we can we want and then actually use that to serve up the final recommendations recommendation to design application software for experimentation so I'm gonna walk through an example to film it process that I've seen so many times and I've seen a lot of you know heard a lot of people runs this problem other companies so typical thing happens give this great idea for machine learning model you go out you find some data of how you're gonna use it you do a lot of offline modeling in your favorite language like our my MATLAB Python and you iterate a lot of time and you make your model perfect and you get that perfect final model that you want to put into production you think it's going to be amazing and a/b tested so then you start implementing it in the production system in that world maybe lives in more you know Java or C++ and then you start finding problems you find discrepancies in the data between the online and offline world you find that Owen the real online system there was this post-processing logic you weren't taking into account when you're doing your experiments you find performance issues in your code and you find discrepancies between the online code and offline code so you spend a lot of time in these loops kind of debugging all these things and these until you finally resolve all of those that you finally actually see what the output of your model looks like in the production world and it doesn't look like what you expected it to look based on the you're getting your initial evaluation and the great model you had so then you go back and try to figure out all was my idea wrong was there something messed up in the data or it's something in my code you know it needs to change and so you spent a lot of time kind of strapped in these cycles and each time you go back you have to kind of potentially go through all these you know loops again and so you can have put a lot of time doing this and so the solution we have kind of come up with that Netflix is to really you know make sure we share as much data good code and data as we can between production experimentation really lean towards production so make sure so it's very easy to do things with the production system in the production code because again developing machine learning is iterative so what we do is we rather than trying to push the lab out to production we try to take the application we're going to put this software in and really pull it back into the lab make that software so we can do machine learning on it and share as much as possible and of course you know you're you're never gonna have that library has everything in it already in your application but you can do is make sure you have kind of very well-defined points for pulling things out and taking things back in I thought that ran into sorry yeah okay yeah so so having as well-defined formats can make sure that again that you you maximize the amount of the existing code and minimize discrepancies as much as possible and so it just helps you this problem of avoiding dual dependencies so when you have your experiment code if you then make your copy of it that runs in in production and then you know there's a wall between the two and there's typically even different people using it and they iterate on it it's very easy for these two things to get out of sync so if instead you have a very big shared library that you use both for production experimentation and then the experimentation and production code themselves are really just the drivers figuring out which parts I need to use to run my experiment which parts I need to use to serve up production it can eliminate a lot of these discrepancies and make it much easier for you to try your ideas on real data and real world scenarios to make sure you're gonna have the maximum impact and here I mean this means where we put in the shared engine is pretty much everything we can and all the models the features the training algorithms the metrics we have etc recommendation three kind of goes along this line make your algorithms extensible and modular so you know I think a lot people they start getting start with machinery and kind of go with kind of like the black box tools that are out there so you know they have you know make it really easy for you to take your data and you know Tuukka few parameters and get out some something that you can do some prediction on but what we find is that a lot of times those black box algorithms don't really work that well when we try them in our real problems and real production systems and we spend a lot of our time really tailoring them to the the problem domain that we have and you know it's a pretty simple idea but you know the the thing to do here is that you know if you if you have these kind of black box things are very hard to tweak you spend a lot of time either rebuilding them or having to learn their internals and building very you know kind of brittle solutions but if you're able to build kind of these white boxes where you have a lot of you know extension points where you can customize and tailor how algorithms work to your problem you can it's become much easier to develop the algorithms you build on top of them and in particular when you do this making sure you have really nice inner faces around the model and the algorithm itself so you can easily swap them in and them out and also do things like compose them together and if you kind of applied this you know principle all the way down what we find is we're building stuff where we really try to provide you know good building blocks to build on top of so we don't try to start from scratch we try to build on you know buildings for algorithms kind of up in nice little units that we can always reuse kind of the next layer down if we find that something isn't working at the highest level so starting from you know you know a good matrix library linear algebra might library that provides the good primitives that you want like vectors and matrices having good statistical tests you know and so you know distributions and things like that you can build on you know having abstractions for things like models features metrics all the common terminology machine learning as well as you in different domains things like lost functions or things for optimization or you know abstractions for neural networks etc and then you know kind of at the highest level that's where you kind of can build your domain specific components and sometimes you might you know you can just kind of plug in something to customize it and then by using familiar apps concepts from the domain for the abstractions really make it the job of the software to figure out how to put them together so you can express your idea in kind of a way that's most natural and leave the the composition to a software engineering process and so as an example here is some code that from using open source library I created with some people at Sandia before I joined Netflix and we actually used parts of it within the Netflix system itself so it's a Java library so I apologize for the verbosity of the code but the basic idea here is if you want to take something simple like random forests and and change how we're gonna do the splitting criteria in the nodes to use some custom information we might know about the various features the idea is you want to be able to take the code and you know be able to drop in my custom splitting criteria you know I you know I'm usually a very patient person but for some reasons machine learning algorithms get very impatient so I really just want to see how this algorithm is going to perform after an hour to get the results and of course you know the metrics that kind of come baked into the algorithm or not what I actually want to compute things I've cared about things like diversity so I want to plug in some custom performance reporting and then at the end when I'm done I want to be able to inspect the whole thing and understand what's going on and so but it's providing all these extension points you can make you know your library is very flexible so you can tailor them for for various different applications fourth recommendation describing that your describe how your model the input and output transformations for your data along with your model so we think about building image software for machine learning a lot of times what we're doing is we have you know some existing application and then what I think about it there's just a missing hole there's some big problem in this application we need to solve and the only way to solve it is using machine learning the question is you know what are we put in that hole so we go off when we build you know our great machine learning model and we decide ok let's plug this in and what we find out is that well that model we built you know typically requires some kind of you know say vector gate coming in there represents the data and then the vector out that represents the predictions but then we go back to the people who build the application and they're like well yeah but the thing I want to have is a user and the thing I went out is the set of recommendation so how do I go from that to your internal vectors and whatnot right so the question the problem is that there's this gap between the two you know we like to talk about you know features and and then there's you know but the application teams like to talk about you know real structured entities so the question is who is this application code or model code they can be very natural to kind of drop in and say well the machine learning box is already there so clearly it's application code but what we found is that if we do this it actually creates a very tight brittle coupling between the two and that you know really what fills in these gaps is really how about we think about is feature encoding how you doing this representation and then output decoding and those really are part of the machine learning algorithm part of them that are the material approach and the machine learning software so it is much easier if you then describe how you're doing those things as part of your model instead of leaving that up to the application layer and creates then a very nice interface and decoupling with the applications you put the machine learning into because the application still gets to deal with its kind of high-level concepts and print domain primitives and then it's really part of the model description to figure out what to do with that and that gives you tons of flexibility and the types of models that you plug in there just as an example example of this again the recommendation domain you know if you had a system we have a system like this you typically would have the high level API be something simple like here's a user and some videos and rank them for me and then we can load in a model description that says hey I'm rancor that scores videos individually I have set of features that I'm going to use and maybe those features are parameterised you know then you say something like a simple in your model to score it and by having this description it's very flexible and we can kind of plug in and out a lot of different you know ways of competing features a lot of different algorithm there a lot of different models for doing scoring and whatnot recommendation five maximize a single machine before you distribute your algorithms this is a common problem you you implement your software your code you get it working on a small scale now you need to scale it up on larger data but it just you know it's not going to it doesn't work so of course you know you you're working on this big data these great you know spark doop tools out there so hey that solves scaling problems right but the reality is like really beyond trivial and machine learning algorithm scaling them is actually very hard because there's a lot of commute typically a lot of communication overhead in them and so often if you want to scale an algorithm you have to make a lot of changes and approximations to do it so before you go there I I mean I recommend just trying to squeeze as much as you can out of a single machine first you know memory you're doing things on a single machine and communicating through memory has a lot more memory bandwidth than if you're trying to communicate over a network and I think you'll be surprised at how far a single machine can go as an example today Amazon you know announced this new instance type that has two terabytes of memory and 128 virtual CPUs I mean I don't know how many people are working on things were you actually like the binary memory representation is more than 2 terabytes I mean if you have more than that then yes you probably need to distribute it but make sure you know you're really maxing out what you can do and when you do that I mean first you know do simple things like actually profile your code and think about memory cache layout you know so small changes could have a very big impact I've seen time and time again someone transposes a matrix and it goes from something like ok this takes 100 milliseconds to this it takes 3 milliseconds so thinking through these things you can again squeeze out tons of performance even just out of your just a simple CPU and then once you get that working it's very easy to typically go to multi-core so if you're doing SGD type methods using things like hog-wild which is just lock-free multi-core SGD can work very well when you have a lot of data and then also considering a you're doing things that make where it makes sense to use a GPU or you know where these maybe someday soon your Google can released their new tensor processing units that they also really announced this week you know squeeze as much as you can out of those and then kind of go distribute it and then when you do distribute things you can start with the outermost layers first so for us if we're trying to train models for different regions start you know by separating out those problems and have those be distributed and then for each model we're trying to train we're going to want to make sure we have the right hyper parameter so within this one native issue regions you do distribute your grid search or you know distribute your how you're doing your Gaussian process hyper parameter optimization and those are begin it's super easy to prepare elective should be compared to the actual internal algorithms so typically you know this is kind of you know what the typical type of Arbor texture we would land on for something like neural networks but you know if we get to a scale where that doesn't work then we would finally decide okay now let's actually try to tackle this as a distributed problem we find that very few you know of all the problems you have very few of the ones we have actually need to get to that level recommendation six don't just rely on metrics for testing so I see this temptation image when people are developing machine learning software which is that machine learning has this built-in testing process which is that we use validation metrics to test things right we build our test you know we have a holdout sets we've run our algorithms we try them out and so when you're building software and you're kind of doing in this loop you kind of said ho look you know I I put this in the data came out it's better NASA made software has to be correct it has to work it has to be great and instead be very misleading because you can kind of feel like oh yeah I'm doing great doing great but then the problems we hit this problem of like oh now the metrics didn't go up well what went wrong was that the code was it my data was the metric was that the idea was wrong a lot of time people will just jump straight to the idea or they'll spend you know a lot of time going back and they written tons of code and figure out was that last thing I did and what was wrong and it could be you know with something very deep that's been the code for a long time and so I find the reality of testing is that you machine learning code in files tons of intricate math and software math and logic you know in dealing with you know rounding issues in different corner cases you know the just a simple thing between having a plus sign and a minus sign where I've seen you know it's a the math someone derived or the paper they've been trying to implement had it wrong and it can create totally different outputs and again it's not going to just crash on you it's just going to could be you know just slightly off or slightly worse than you expect and so the only real solution here is I find is just to really unit test things and at first it feels like it's an overhead but once you kind of if you combine this with again building building block so you have good unit tested building blocks you can have a lot more confidence in your software and that means you can get a lot more things done in the long run and I'll call it that testing your metric code is especially important there's like one thing to like test the make sure is like rock-solid it's geometric code because that is what your ultimate ly going to use as an arbiter for all the other decisions you're gonna make and we like to think about is like you're gonna if you were gonna business you're gonna make business decisions based on output this metric you're good they're gonna say hey we should try this idea or not we shouldn't try this idea based on those metrics so you want to make sure that those are really operating the way you expect and are very robust to various different corner cases of course unit testing isn't enough you know we're putting together you machine or software you putting together your large pieces your large chunks of code you're dealing with your real data you want to make sure that at you know that you've actually have system level tests and if you're doing if you're kind of plugging the software you're making sure at a minimum when you're as you're developing that you aren't breaking the existing systems you have doing some kind of new checks across builds to make sure that the output stays the same when it's supposed to say the same I think it's at least like a minimum thing you should do so that's in front of my high-level recommendations I'll kind of go with a couple final thoughts so I really see you know machine learning as another way of solving computational problems you know kind of in parallel to software development it's offered about something you do when you you have a problem you know the solution and you can you kind of you can just break down the code and logic to solve it you kind of go through typical process of you write your code you know and then compile it you test it you deploy it this is kind of standard software engineering and then when we build do machine learning we actually go through a very similar thing but instead of saying well I don't know how to solve the problem but I know I have this data that I could use to solve my problem here rather the writing code we kind of think about developing this algorithmic approach I was talking about earlier the different areas and then we you know we rather than compiling it we train the code rather than testing and we use their validation metrics and then finally we have this deployment step which i think is an area where there's still very little work that has kind of been done but I think if you think about these parallels you kind of see some interesting things which is that again if you focus a lot on the training algorithm that's like being very obsessed with compilers when you're doing software development I think the powers are important but you you know the real important thing is again that how you develop that whole algorithmic approach was like you know when you write your code and of course when you're doing machine learning each of these sub steps does involve software development itself so the takeaways for building machinery software so machine learning is an iterative process so make experimentation easy take a holistic view of the application where you're placing machine learning so don't think of just the you know that one little box but actually once you put machine learning into an application that whole thing becomes a learning system and you're going to want to apply machine learning principles to the whole thing design your algorithm is free modular look for the easy places to paralyze and optimize for a single machine first and testing can be hard but it's worthwhile that's it we're hiring so if you're interested you know both on the machine learning and on the software side so come talk to me time for questions yep and you said that without of course any talk about systems residence that push synthetic actions on the other side examples so the question is kind of for full system testing kind of what techniques do we use yeah so it can be hard one of the things we've done is that we've actually built a system that will take the online micro services that are our algorithms actually would kind of call to get all the data coming in and we snapshot them so that we can kind of have very consistent views of like this is what the data actually looks like when we get a request to do a computation and so we can use that to kind of then feed through you know the rest of the system from the very outermost layer and see what happens to the whole thing and we can do that you know to train models we can also do that for basically you full level system checks and putting large sets of components together and make sure that that what comes out you know look it looks as close it looks exactly like it should in production so if we find you know regressions and issues that we can detect them patiently so dude so question is you know we have a very limited set of inputs for the user so what do we do in terms recommendation new suruc on so forth brand-new yeah so kind of the item the item called start problem news and the thing is coming what do we do so I mean interestingly for us you know for our domain the item called start problem isn't as big of a deal because we we're not like doing news recommendation where things have a very short life span you know we movies and TV shows professionally created ones have a very long life span we license them for years so you know if we on the first day we might have very little data we do know things we have people you pay people to you know professionally tag you know the movies into various kind of micro genres and whatnot like that what mood it is and you know you know who them what the profession of the main character is week of all this metadata to kind of get things started but from there we typically kind of just let the system kind of over a few days evolve to a stable state and it doesn't take that long because again we have a large member base and and so the algorithms can easily kind of pick up on it we but we do have some cold start features that we can use as well if like the metadata okay fooling around on a lot of different levels you showed an example yeah of course yeah yeah I mean that's it so the question is you know kind of how much of this how much of the code is custom and how much is kind of reused and where do we cut where do we draw that line what's the decision may signal yeah we try to use kind of as much things that are out there that make sense obviously being on the JVM that has certain limitations of course the you know the library I showed earlier the foundry actually you know it calls through to things that when you have Blas in law pack installed in the native you know libraries compiled it would call straight through to the Fortran so you know some level we try to pull as much of those things up as possible and make sure where you've reuse all of that goodness but when we make when we decide it's really kind of we just kind of we have to look at you know how easy would it take for us to take this software as it is and plug it into our system versus how hard is it for us to build it ourselves and you know for some things you know on the back end it can be very easy to plug things in to kind of try out lots of ideas and so you try to make it very easy to kind of get data out of our system try out and at the end there kind of bring her back in but in in the online world it can be a lot harder because we kind of a you know we want to make sure it runs at scale and is very robust so that's typically where we might run some of our own or when the other thing is if you have something again that's very textbook there's probably a solution out there if it's something that's very custom very tailored we want to build it ourselves but we want to you know have to build from scratch we want to build it on build on top of a kind of a good set of things that we already have yeah their tuitions we're alliterations we're my training where can't actually production so I think if you're doing in the case like that if you're doing live retraining you I think you have a you have a similar problem you want to solve it's just about there you would get there again having the if you're doing online learning having that feature description with your model means that you know again you can you can publish say your initial seed model and then if you want to update it you have again the full description of the set of features you need to train it if you were to do it you probably weren't also published you might be able to publish even what the algorithm you want to use a train it and they can make that very modular so you can plug it in so I think for online learning it works that it works the same way and it's and it's really about you know how the data comes into the system and making that interface release to pay for yep sure so the question is basically what where we look at in our AP tests so the primary metric are one a/b testing is retention ultimate like that you know for us or a subscription company like I said before if you know retention means people pay us you know money at the end of the month and so if we can move that that means you know we've definitely done benefit to our business and we find again with the recommendations they move retention that's the reason we we do it so much retention though is is hard to move it's you know it's once people kind of are subscribed for a long time they typically you know if you've been to the Netflix for six months going to the seventh one it's pretty likely so we also look at as things our queen or correlated with retention like how many hours in efflux you streams so that's a little more sensitive and then we also looked in you four different if we're you're kind of working in different areas looking down then at the kind of for each area metric up saying if you're going in your you're changing algorithm and search i'm going to make sure the search metrics are improving not going down from your change and those types of things but the primary drivers are kind of what we think it was because as retention and then streaming out another really of questions maybe you slightly because I never take my personal example I hate the system if I came to say Oh bleep most of the time outside probably that UI so I'm trying to understand the question exactly so the question is kind of if you're doing a/b testing yeah so you're saying because we already have a recommendation system is there a bias that would make it such that if we took it away that we would maybe that might actually come up at our Navy tests like that maybe so again the way we got there was through a be testing of you know I've seen kind of time and time again we've got into an area where that's not personalized right and we personalize it it's better right we could going algorithm without without an algorithm and then we put in something it's better I'll give him an example one of the recent things we kind of announced was that we we're actually trying to optimize the images we show people so the kind of box art representation on the homepage you know for house of cards we could show a lot of different images that sharing to say house of cards and what we can do is actually try to optimize those and we can find better ones and again we av-test that and again we see you know meaningful results from that such that we're willing to go and you know talk with the studios and get better images and try out lots of them to find good ones so you know going to a very deep level with these algorithms and and personalization is just you know even beyond kind of traditional recommendation we just see time and time again you can make the make the user experience better once you collect relevant data and can use it effectively any last questions Thanks