Scale By The Bay 2018: Julie Pitt, Applied Machine Learning: a Netflix production
Recording: Scale By The Bay 2018: Julie Pitt, Applied Machine Learning: a Netflix production
might want this a brief trigger warning this talk does not contain a single line of Scala code so if you were expecting that and maybe end up throwing tomatoes at me make sure that they need to include heirlooms tomatoes like I make a great spaghetti sauce at the end of this right it's August 2017 and shelwyn has a problem she needs to know what netflix titles people are talking about on social media and what are they saying about them so the good news is Shawon has a superpower she's a data scientist and so she sets out to build some models using Python NLP techniques we're gonna skip to the part where she has a model prototype working in her local development environment because up till now it's been a magical journey through fairyland with feature engineering and exploratory data analysis and choosing her model parameters and so forth what she doesn't realize is that it's about to get dark because Shawon needs to get her model into production and the first thing that she realizes is that she needs to scale her models up to the full data set and not the sliver of data that worked on her local development environment so she realizes she needs some compute resources shelwyn goes to her friendly software engineer Anu who says that's no problem at all we have a compute resource orchestration system called Titus it's even an open source project and it's a lot like kubernetes so all you need to do is create a custom Titus executor farm out your work and then bring it all back together when you're done so shines a bit daunted but she persevered and digs into the documentation and writes a custom Titus Secutor then she felt realizes what her next problem is which is she can't get the data to load very quickly on all of these compute nodes going through these sequel interfaces has all kinds of challenges either memory challenges or concurrent job challenges so again she goes back to Anu and they work together for a week and they get data access running at a reasonable speed so now she has the full model running one the full data set but that's not enough because she needs to train the model on a regular basis which means she needs job scheduling so once again she goes to her trusted partner Anu who says hey we have this job scheduler you just need to read the documentation figure out how to do your job configuration and get it scheduled so once again she'll win goes and figures all this out and she's done right but then her boss reminds her well you know we're not doing machine learning just for the sake of machine learning we actually need to get this into a business application so enter the web developer who's writing the business application who Shoen approaches and said I need to get my model in here and the business application developer says no problem just give me a rest endpoint for your model rest endpoint so she again figures out how to write a basic flask app and that takes some time okay so she's done she can celebrate she can move on to the next project and it only took four months not bad right except she's not really done because now the models in production and how she has to think about DevOps for the model but what happens if she retrains and there's a problem right how does she do historical backfills how does she continue developing this model while not messing up production so I'm telling you the story not because it's unusual or because it's sensational but instead because this is the everyday experience of your average data scientists in fact how many of you have similar stories okay a few people I'm not just making this up here let me just retell the story very quickly so Shoen needs to build a model in order to do that she needs a data warehouse she needs compute she needs a job scheduler versioning of her data collaborative tools model deployment oh and then she can think about feature engineering machine learning libraries and now she's ready to have a fantastic intellectual conversation about monads and type classes and lambdas no not just kidding she actually has to spend some time thinking about the actual work of creatively coming up with a model to solve the original problem which was understanding social word of mouth around Netflix titles so the observation the key observation here is when you think about it the lower down the stack you get the less the data scientist actually cares about any of these details that's kind of depressing right because we just went through the story where she spent most of four months dealing with all of that stuff there's a good news though which is that the lower down the stack you go the more leverage you can get out of building infrastructure so this is exactly why we built meta flow which was designed to be very opinionated about things at the bottom of the stack that scientists don't care about and very unappealing toward the top of the stack but scientists do care about they do care about working in a development environment that they're familiar with Python or are they do care about being able to use the models and the implementations of those models that matter most they don't want to be forced necessarily into using tensorflow maybe they want to use pi torts or fxg boost or vocal labid or any one of these implementations the key principle behind meta flow is that it's designed for humans and humans that have diverse skills and are solving diverse problems just take a cross-section of the data science team at Netflix we have one scientist who has a PhD in physics and studied light coming from a nearby binary star system we have another scientist who has a political science and government ph.d and did some studies around policy making decisions in a collaborative environment and another person has a PhD in cognitive psychology and did some studies around semantic cognition so these are very different people with very different backgrounds and they're also solving different problems for Netflix some of the less well-known challenges at Netflix and machine learning are things like well what content should we put in our catalog what what content are going to resonate with our members and bring them joy and there's a lot of interesting work on what would the demand model look like for content that nobody that's never been made that nobody's heard of before we have challenges around how do we make the streaming experience itself phenomenal how do we deliver the best bit rate to our customers to give them the best video quality on their device on their network and then there's a whole bunch of super fascinating challenges now that Netflix has a studio function as well for example we get a lot of scripts coming in and there's a lot of NLP you can do on those scripts to learn about you know how might you organize a physical production around those scripts for example automatically highlighting you know when hair and makeup persons should come up on board or automatically calling out maybe when we might need to look into a legal matter so there's a diverse problems done by diverse people what do these people have in common well they need to build models we require some prerequisite knowledge of all of our data scientist and so here's the the long list of prerequisites they need to know what a function is and they need to be able to write a function so a scientist needs to understand this concept of input compute output and they also need to be able to run a basic script on the command line this is a little bit basic amazingly though you can come up with really powerful abstractions on top of this and the abstraction of choice for meta flow is a directed acyclic graph or dag and if you've looked at any machine learning framework you probably have seen this abstraction before because it's a great way to abstract machine learning workflows that involve batch processes that that can take a lot of time so how would a scientist structure their code using meta flow the basic theme is that they write in vanilla Python code and they add a few little decorators so that meta flow knows what to do with them so really the first thing that they need to do is they need to extend this meta flow flow spec class that gives them all kinds of facilities to do all of this dag stuff then they need to write structure their code as individual functions and annotate their individual nodes in the graph with this step decorator and to create the structure of the graph then they just need to call this self dot next method which creates the arrows in the graph it's pretty basic and you might think wow that really that's it like that's there's nothing novel here there's not really any new ideas here so on this team on the machine learning infrastructure team on Netflix we we really measure our success by usage our data scientists using this and the answer is oh and I've skipped that part going back here all you need to do is run it on your local environment okay so the answer is yes say it scientists are using metal flow oh nope I skipped ahead you can also write models in our via a thin wrapper on top of Python so the the incredible thing is that we don't need to rebuild our infrastructure in our we can actually create this very thin wrapper using a reticulate library and all of our our users are happy so we no longer have to worry about this war between data scientists we're like oh is are we a Python shop or we have or we an our shop so this kind of resolves that what about scope shot ok tomatoes are flying my way they already um we can talk about that afterwards ok so anyway so what's so what's so special about this why is that why is this even why are we even talking about this because it is being used so we launched our 1.0 internally and since then we have 134 projects using meta flow so something you need to understand about Netflix and why this is relevant is that the the culture and Netflix is based on freedom and responsibility so we hire responsible adults and we trust them to make decisions that are in their area of expertise so there's no mandate no VP stood up and said you know thou shalt use meta flow this is really a testament that it actually is useful in practice now I want to get into what would be the day-to-day development experience in meta flow the first thing that a scientist might run into is that their job fails or that there's a bug and so they might need to kind of do a quick development cycle and rerun a particular step in their flow right then and there and they might need even might need to pull in an engineer when it looks like maybe an infrastructure based problem so what they can actually do is resume any step in this flow without waiting for the the entire job to run again because some of these jobs can take a long time I mean in the good case maybe it's a thirty minute job but it could take hours so the last thing you want to do is sit there waiting for hours to try to reproduce your bug and the reason why you can do this resume step is because meta flow is backed by an immutable data artifact store and its content addressable so any till the exact state of the the flow at this step B can be reproduced and very quickly both by the developer on their local environment but also by somebody helping them remotely so that's a very powerful feature right then and there that is one step toward kind of reproducible data science okay remember when we talked about shell when and how she needed to get access to compute resources in order to scale her job with meta flow that's very easy and very straightforward in order to get this compute resources all you need to do is add this decorator in this case we have a plugin for Titus you can also create a plug-in for kubernetes and it requests hey for this step I want to see sixteen CPUs and a GPU or perhaps you're doing a memory intensive job and you need lots of memory and the beauty of this is you just run it on the command line again you're not leaving your local development environment to get access to these compute resources what if you need to scale horizontally and you need many different jobs let's say you're doing a parameter grid search well all you need to do with meta flow is set up your parameter grid in this data structure and you call this kind of self next method with this for each parameter and what the for each parameter does is said for each item in this data structure create a new tap new parallel task so this step a over here is going to be instantiated for each item in this grid so now we get this kind of automatic horizontal scaling and the real powerful part of this all is this join step where meta flow will automatically join together all of these parallel jobs without complicated sort of async logic in there and one thing that we wanted to look well at was well how useful is this feature and it turns out that 40% of meta flow projects are using Titus the interesting thing is that many of them are using it within the first couple of days of starting to use meta flow and and that tells us something really fascinating which is access to large amounts of compute resources is critical early in the development process of a model as a data scientist you no longer have to say well I'm gonna create this arbitrary smaller data set that's a sliver of the real data set in order to Train and then I get the real data set and then I get surprised by you know what actually happens there no you can get access to full scale right off the bat another challenge that shelwyn had was getting access to all of her data on these distributed computing outs and we've had that challenge too and we realized that there's a particular access pattern for a lot of these machine learning projects that is different from maybe ad hoc data access in the data warehouse that typically goes through a sequel query engine we realized that much of the data this is really kind of by the Partition level or by the shard level and the these machine learning workflows need access to data in batch so rather than going through a query engine where you have to like submit a spark job or submit a presto job then it does its thing and then it accesses s3 on the backend and then it's pulling all that back and it might run into memory issues or concurrent job issues rather meta flow provides this very easy way to access a table and access specific partitions of a table and it will basically go directly to s3 right using the same hive metadata as the direct sequel access so the result is we get 10 gigabits per second and we are we are where we're accessing data at speeds that are very difficult to achieve through these sequel interfaces now let's talk about deployment because we can't just build our models we need to get them out there and we want to first of all like show in ran into she needed to schedule her jobs which meant she needed to kind of like take her job and then map it on to this job workflow or scheduling service well meta flow will take care of that for you so what metaphor does is that it already knows the structure of your dag and it can map it on to the job scheduler dag interface it'll create the configuration so it's very easy to do this directly on the command line so again all of that reading of documentation that shelwyn had to do she no longer has to do it and we also wanted to know how many of our workflows get into production and it turns out about 1/4 of them get into production which in a research oriented environment isn't too surprising because you're gonna have a lot of ideas that fail but the amazing thing is within you know a week the vast majority of these projects are getting their first run in production which means as a data scientist you can you can figure out what are the production issues your job might run into right after you start developing it and so that's going to shorten that four-month development cycle considerably we also have an interface that looks at your job metadata that you can easily pull up in a notebook so Netflix uses interact for notebooks and you can actually look at what are all the runs of all your flows and you can even look at the data that was in your flow at a particular point in time so you can access this immutable data store so in this case we had this step that had X as a value at that point in time that had a value 0 and some time later we had this step X or this value this data value X and it had a value of T so we can get access to the exact state which again reproducible data science getting you toward that shell wins final step was getting her model into a web service and we've once again made that easy with meta flow so the shaman doesn't have to worry about like any of the intricacies of rest how to provision how to deploy a web service all she really needs to do is implement this class that says hey here's a web service expect I'm creating a micro service here and she needs to put her calls to her her rest endpoints in there implement them as functions and then meta flow will create this endpoint and she can hand it off to her web app developer so that was a very quick tour through meta flow you know we've been we've been taking this very human oriented approach and so far it's been working well but I will say that we have a lot left to do there's a lot of work to do kind of to complete this developer lifecycle for models much like in the last 20 years we've spent a lot of time thinking about what the software development lifecycle is like I that's the big challenge for data science coming up so I want to just end with a special thanks to violet to Lois who gave a very similar talk at Q Khan a couple weeks back and he created a bunch of this material so I think there's time for questions and if you don't get your question in then I'm happy to answer them I'm not going to be at the Netflix booth a little bit later oh I love that question so the question was does meta flow tell you what your jobs costs the answer is not yet but that's definitely on the roadmap interestingly enough what I what I find is that scientists are actually more worried about being too expensive when they don't have to like hey I'm gonna use this compute resource and it's going to cost $5,000 a year and like you know if you think about how much a scientists cost that them spending the time worrying about this five thousand dollars over the course of the year assuming their job actually runs for a year you know it's not actually effective and we certainly found that that so far with these hundred thirty-four projects the cost of data storage is actually considerably low so it's not a kanata concern yet but that's absolutely something that we'll want to build into the platform yeah [Music] so the question is with meta flow is the rest API the only way to deploy your models and the answer is no there's quite a number most projects actually don't use the REST API so again with creating a job you can create any number of different types of artifacts right and you have that freedom and flexibility to do so so in some cases your job might do some bulk scoring and it's going to produce a table and that table is going to be consumed by it let's say a dashboard for example so there's all kinds of flexibility for creating other types of artifacts it's not a question over here yeah so how does meta flow scale in terms of data set and then how do you transfer data between those steps so in terms of a second I mean is there a more specific aspect of scaling the data that you have in life the data size I mean like I said it's interesting because there's quite a diverse variety of projects that are not like you know massive data size and usually the bottleneck is actually just the speed of loading the data and so we haven't yet hit that kind of upper limit on like how much data we can handle yet but I'm excited to hit those use cases in 2019 and then the other question was how does meta flow get data between the steps all save the the details for offline but they're the immutable data artifact store comes into play there yes so the question is are we thinking about a federated data model so you might be thinking about the fact that well Netflix is a global scale company and Netflix operates in AWS regions across the world a choice that's been made up until now is that the data warehouse itself is living in one region and so I think we would have to evaluate a lot of these use cases are not necessarily like real-time use cases and so we have at this point we haven't seen the benefit of you know federating at across regions so it looks like I'm out of time and happy to answer questions I'll be at the Netflix booth thank you all [Applause]