scala.bythebay.io: Davis Shepherd, Building a Machine Learning Orchestration Framework in Scala
so I'm gonna be the last talk between you and lunch anyway a little bit of myself my name is Davis I work as an engineer and the personalisation infrastructure team and I'm gonna talk about kind of a marquee project we were working on inside the personalisation team to deal with kind of the collection of challenges that we've we've we've had to deal with when trying to personalize the Netflix experience so to kind of motivate this a little bit talk about okay what does it mean what is the what are the goal of personalization and Netflix of course now there we go so our goal is to help members find content to watch and enjoy so we can create this virtual cycle of satisfaction or attention amongst our members it's sort of the core tenant of what we want to be able to do so now what does it mean to actually personalize the Netflix experience sure some of you are may be familiar with this page and in essence everything that you see in this page is a function of personalization algorithms not only the rows that appear but the order in which they appear as well as the title the order of titles within each row so the goal here is to make sure that you are as close as possible as to what we think your you might enjoy watching at any given time on the experience and all of these rankings and recommendations are driven by machine learning pipelines so what does kind of a machine learning pipeline look like and Netflix and this'll probably look familiar to anybody who's done this we start with user selection you know sort of getting our sample data what we want to use to inform our model and then we have to label that data we have to draw it through feature generation feature encoders you know some notion about how we label that data and then that's fed into obviously training our model or maybe potentially a collection of models and then we go through a process of validation deciding okay does this meet criteria for prediction accuracy have we found a stable sort of minimum for our convex optimization and once all these things are sort of satisfied then we can sort of publish the model for use in online systems now you know this seems kind of neat and tidy and straightforward but obviously in reality things are never quite that straightforward and it's a little bit more heterogeneous for example when you actually get your data that may start as a hive query you know deciding ok I need access to active members and and needs to access with our current sort of data warehousing solution and then we may have a collection of Python software to decide ok how we're going to separate this data into training test validation and then that'll be fed into potentially a myriad of model training pipelines whether it's regional or global model and each of those may potentially leverage different technologies they've different requirements a global model is gonna need a large a larger quantity of data so we need to leverage spark and use maybe a more approximate sort of machine learning approach whereas the regional model we can use something like our to sort of do a more direct optimization and these pipelines need be able to run concurrently because obviously we need to be able to meet our time budget for retraining these models and then once all its that done then we need to go through a process potentially of model selection and that'll use potentially yet another sort of technology stack whether that's just a Scala application stand-alone application and then you know potentially again a separate stack for publishing a tender to into our online systems so this sort of motivates this or the issue that we've kind of run into which is that there's a lot of different technologies that we need to interface in a cohesive way to ensure that we get sort of reliable easy-to-use access and the ability to experiment with in personalization so fundamentally in machine learning space it should come to no surprise there's a ton of innovation and that breeds heterogeneous environments to run on you know whether that's spark or R or you know docker applications or graphics API so we needed a platform and sort of a framework that could both orchestrate and execute across these heterogeneous environments that said SPARC is used very heavily inside of the personalization team so that was one where it was sort of a we had to have a batteries included solution for running spark and these workflows so that was definitely a hard requirement for us now these first two points kind of mandated we wiII be able to sort of separate orchestration from execution we should be able to sort of change out how we sort of farm out the work from the actual implementation of orchestrating these workflows now we also to support a bunch of different teams whether it's search or video ranking or you know you know there's a similar video team so we need to be able to support multi-tenancy right and be sort of conscious both of our time budget as well as cost budget for for running and training these workflows and you know there's also sort of machine learning constructs in particular something like a parameter sweep that may mandate that we have like a 30,000 sort of branch fan-out for a sort of concurrent training right so there's there's sort of a scale aspect here that we need to be very conscious of and with these challenges in mind we came up with our maze on workflow system now this is a general-purpose workflow orchestration engine and scheduling framework it currently designates execution to resource managers like mazes or yarn but fundamentally it's optimized for machine learning pipelines and visualization and we actually put out a blog a little while ago if you want to follow this link or find it on our on our tech blog if you want to read a little bit more in depth about sort of all the features that it has and I'm just going to kind of touch on sort of the high points in this talk and I should point out our plan is to open source this in the future so our hope is that once we have you know what we feel is a concrete solution to the set of problems we've solved we feel that they are applicable to the industry at large so the architecture is it's it's so maze on is a as a scholar application that sits as a mazes framework it manages sort of state persistence it implements obviously the mezzos framework itself we have a collection of plugins and extensions to enable this system to interface with the various subsystems inside of Netflix whether it's the you know the data announcement system or the various places where we might be able to ingest data and that's we need to be able to support the ability for users to come in and say oh I need to use well X I can do that very simply with a simple extension and thereby tailor Maison for their particular use case and then execution itself once we've sort of captured that state and how the flow of the workflow is gonna behave the actual execution is delegated to a maison mais au s-- cluster running sort of a custom a maison executor to enable us to sort of communicate nuanced information about the the runtime in these workflows back to the actual Maison service for surfacing in a in a UI for the user now talking little bit more about this plugin an extension system you know we wanted to be able to support a lot of different sort of sub step types you know so the hope is that when you know the algo engineers come to use Maison it's sort of batteries included right you know we already have sort of de facto solutions for the various sub pieces they may need to sort of glue together and you know a lot of this was sort of aided in particular by the fact that we could use sort of Scala's rich sort of tricked rate system and multi inheritance so we have this this notion of sort of a thin cake pattern where we glue together various sub implementations and it's very easy to sort of create these new pieces of execution for the particular use case moreover all of these pieces need to be parametrized obviously so we needed to sort of support a rich parameter system you know in the simplest case you know hype wery needs to be motivated by a timestamp like when you can actually fetch your data and then this is going to emit sort of the location of that user data set which will be consumed by steps downstream and you know this sort of process can ad nauseam conservative as you know simple useful information need to build a flow through a workflow so once you have these sort of these steps parameterised then you get to the part where your work role actually has to run and so when a step is actually executing we need to ensure that we surface as much sort of useful information to the user as possible and that comes in the form of you know runtime logs as well as access to like you know sort of new ones progress information as well as artifacts about you know maybe okay what's sort of your feature importance information for that particular step run or how many records are due process where's the location of the user data set that this this thing has emitted and all of this sort of necessitated us to use this sort of custom executor that I mentioned earlier now one piece of this artifact system that is particularly useful and important has to do with visualization so a big motivator for this was the notion that you know very often in ml pipelines there's lots of intermediate data that you want to inspect and not only for the instance of a single run but over the course of multiple runs so we want to be able to chain together all this information from a single step so that you can sort of track its behavior over time whether that's something like feature importance or convergence time and understand how it behaves over time in the system so you can identify useful signals on something like feature importance is is a great way to sort of potentially inform future hypotheses that may go into other a/b tests later now as I alluded to earlier workflows are rarely as simple as do a than P then C so we needed to sort of include a number of structured content there constructs for how to sort of stick these Legos together so in the simple sense obviously we can support something as simple as do a than B then C but we may need to we also need to support this notion of sort of a fork/join notion within a workflow whereby you need to run several sub flows you know weight on their weight in their completion before proceeding something like soar where you have sort of conditional flow through your workflow that may be mandated by you know the output of an ingress step and we also need to support this notion of for each which is sort of where that came from that requirement of of need to do like parameter search right so the actual fan out of your workflow will be in degree will be sort of controlled by you know particular tuning parameters you've decided sweep that I want to run for this particular workflow and we also knew be able to support loops you know in particular if we get to a validation step that may indicate we need to actually go back and rerun a subsection of the workflow again or expand our user set if we think data quality is low or something like that and now we have two ways for user to sort of construct these workflows first is through the web UI we have essentially just a graphical way to represent these and manipulate them and create them and that's that's useful for sort of debugging these workflows prototyping things and sort of getting used to sort of how the system behaves but the more common way that people actually our users actually create and manage these workflows is through a DSL and this is where the sort of the power of Scala was was sort of made clear to be through of a correct choice for us was the ability sort of define a rich type checked fluent system for defining these workflows and managing their lifecycle over the course of development and this is how the vast majority of users describe and manage their workflow definitions so we'll talk a little bits of like what was the scala stack that we ended up using to build this application so the web server is built on top of a KH TTP making heavy use of the routing DSL and spray json you know we started to dip our toes into using the actor system on the construction of our actual Mae Zoe's framework so this is sort of one of our larger projects in scala within the team and so we are sort of taking a measured approach to trying out these new new libraries you know we were sort of wary of big heavyweight frameworks and and we wanted to sort of be able to test out actors in a small and a small way we also use scauldy for dependency injection and our future work is to actually extend our initial experience with actors is quite good and we're looking at actually using the SF FSM trait to sort of restructure most of if not all of our current state machine with a hope that we can leverage sort of higher-level concurrency for workflow processing because you know as the state machine group sort of organic as as requirements came up and we feel that with FSM we can take a much more measured approach to read ascribing the state machine in a very sort of clean reusable and manageable way those members everybody likes numbers our our current primary maze maze on cluster manages you know hundreds of them current jobs you know Anna given point will run between 100 and 200 workflows with thousands of steps and the execution these workflows is farmed out to a cluster of about 750 nodes with 6000 cores and about 40 terabytes of memory and this cluster essentially obviously manages the runtime of the steps themselves the SPARC frameworks that are sort of started as a as a function of those steps firing off and a few odd docker applications as well and we have a few smaller clusters for users to sort of test out in isolation if they like as well so that's it for maze on this is my contact info if any of that sounds interesting you what he knows we're we're hiring as well in the personalization team so if any of those problems sort of excite you or you be interested in please come and talk to us yeah there any questions correct although to be clear it's not a dag it's because we do allow cycles but yes so the question is we use DSL to define our graph yes yeah so the the maison workflow system itself actually has I'm sorry to repeat the question you know if you modify a DSL definition how does the lifecycle of that update actually manifest itself into the runtime system so the runtime system actually has a rudimentary sort of versioning system so you can sort of track state and you sort of manage a prototype and then you can sort of make that active so you can independently version your DSL in sort of a traditional build system you know under under version control and then when you have a stable build you can essentially say okay this is my new workflow definition I'm gonna publish that to the runtime Maison system so that will actually publish a new version based on your new DSL definition that'll get uploaded the server along with any sort of changes to your you know runtime artifacts that you may need and then that will be made active for future runs of the workflow so the first part is entirely up to the users right they I mean they could define a DSL but the actual uploading up to the main zone servers is merely an API call so it'll pull down compile their DSL translate it in the appropriate API call and then push that to the server yes so the question is are the various steps separate spark jobs and the answer is yes so we actually use so the question is is you know you know are you able to share a spark context between two steps currently no and there's good reason for that you know oftentimes we want to be able to sort of isolate resource requirements for various steps because spark itself is is not great that's sort of adjusting its resource consumption over the course of a job dynamic allocations getting there we're getting close to that solution but it's generally easier to make small focused spark jobs that use exactly the resource footprint they need and emit data for downstream steps so so the question is I'm not sure I totally understand the question does it have to do so is does the DSL enable us to do think yes yeah I mean you could you could like you know define like your definition your workflow made hard-code what your sort of parameter spaces and then I'll do a Cartesian search across all those combinations or something like that absolutely yeah I mean it's it's it's very he's very pluggable and moreover it's it's it's very generic cray the idea is a step is really just a unit of execution we have some batteries included ways for how to define these steps but at the end of the day your DSL is as this Scala code right so you could define any sort of complex flow or how you're gonna manage the life cycle those dsl x' is very fluid so yes yeah so the question is how do we manage multi-user workflows now do you mean multiple users that are concerned about the same workflow or ah so typically you know we we don't build tooling around trying to sort of do like like strict version control for multiple users for the same workflow definition I mean we give sort of atomic updates for updating a workflow definition but the onus is more on the users to be able to say okay we're collaborating on the definition of this workflow and you know the tooling inside a Netflix is very heavily using sort of automated build systems and and and version control so those tools sort of solve that problem for us yes in the back so the question is where do we maintain state our current solution for state maintenance is Cassandra you know the tooling for Cassandra Netflix is very very good that was a very very easy solution for us to use right away so the question is is this datastore a pluggable thing yes absolutely the the actual API for doing state maintenances is a simple key value store so anything that can pull in key value you can store your state in yes yeah so to be the to be totally honest I mean this we wasn't we didn't have total clairvoyance before going into this with Scala I think a big motivator for us to choose Scala was the spark interoperability and we already had a lot of people who are on board with language right and and saw its merits you know just in general software development and when we got to the point of the need for this DSL to describe it it was immediate like Oh like this is an obvious win yeah so that was I can't say that we had like an immediate like yes we're using this for the DSL but it was it was a it was a lovely discovery uh no I don't I don't recall that groovy was evaluated yes that's a great question so yeah so that so the question is is uh how does a stack up against air flow so have we considered our flow the answer is yes you know air flow is is very much for to solve a similar set of problems and the our our primary issue with with air flow was that because of its very Python centric approach it's very difficult for us to actually integrate spark jobs into air flow and because spark was used so heavily inside the personalisation team it was almost a non-starter for us so you know we felt that we could take a better approach that targeted our set of technologies and then sort of branch from there to sort of cover everything it is possible I mean I've gotten scholars yeah what did you see common pain point you're saying like amongst our workflows and so there's that and so we basically give so the question is when we expect to open-source that our hope is for 2017 we can't be terribly more specific than that I mean we have a lot of obviously internal pressure to cover some features and things like this and open sourcing a project in and of itself is a bit of a task but if anyone's an interesting enough interested in helping us do that please again come talk to us but yeah they were shooting for 2017 yes that's a great question so he's asking about whoa so the question is what is the interaction between the people actually building the models and people building the infrastructure and I mean that's a hard problem you know it's so we sort of have a separation between infrastructure and model building right now but you know with the the collaboration we want to sort of make sure we have sufficient overlap so that people who are building a model are sort of aware of the infrastructure requirements but yeah it's a it's a common problem we run into those building models are potentially don't want to be concerned too much about okay what does it mean to be sort of resource efficient inside of a spark job and so we try to build as many tools as possible to try to abstract away a lot of for the computer infrastructure and try to build those abstractions up you know it's it's not perfect right now but Maison is sort of our first step to try and and and sort of bump that abstraction layer up because before that we essentially was as was much harder than it is now yeah it's not Sagan yes it's it's latter so it's just Java calling or it's not a terribly sophisticated integration it's not a super common use case but it was enough that we felt we needed at least include that and you know there's not a lot of coordination involved in that piece so it doesn't need to be terribly sophisticated yes ah so the common use case is so oh sorry the question was is what was the use case for loops and the use cases you know these these workflows will go all the way from ingesting data to publishing a model for available in an online system and the the loop is a way for us to encode instruction about if you get to the validation step which is sort of the last piece before you decide to publish is it okay to publish or not and we didn't want to just like terminate the workflow and start all the way from the beginning we wanted to be able to start from a memorized piece and rerun from there to say you know expand so I'd say I got to validation and you know data quality was too low so we have to pay we got to run back and rerun that step with an adjusted parameter to sort of get where we need to be yes users still have any control over how that dashboard is laid out and if so how is that so the question is do users have control over that dashboard layout and the answer is not really for the visualization piece they do but for the most part what's presented in front of the workflow is mostly a function of I mean I guess there's some subtle control when you write a custom step you can define certain interesting pieces of artifacts they'll be emitted and those will be presented in the dashboard along with other relevant data but for the most part it's it's it's fairly rigid so we haven't seen a whole lot of user friction or or control there oh I see we're saying god I got I got it okay got it sorry so then can repeat the question again in that new context yes yeah no so yeah I mean so okay now we're getting to sort of market questions about Netflix so yeah I won't go too far down this road the answer is no you know there's a lot of pressure for sort of an adding advanced features in a Netflix but generally we've wanted to make the interface is sort of seamless and simple and approachable as possible and so we sort of erred on the side of a simpler side of the interface I mean we do hear requests for more advanced sort of user interface but we've yet to have a strong answer for why that would necessarily improve retention engagement across the sort of the wider user base so we see we obviously take input what our customers are at the end of the day every single literally you know idiots a pixel in that interface that you see perfect manners remind them who are trying to do all sorts of a/b tests around it so once we get data are our key metrics being you know more streaming people are watching more more attentively to are staying warm and so we use that you know to figure out where the end of the day providing to entice the customer has its downsides do so that's something that if yeah so that's a great question so the question is how do we deal with logs right because the stuffs running all over the place there's lots of interesting information so maze owes itself gets us a little bit of way there so we utilize the mazes API to surface the logs from the actual execution of the step on the mazes cluster to the UI but moreover with things like spark right you might have you know ten or a hundred instances of those separate log places and it's all that problem we use elastic search and log stash sort of pull them into a centralized place so they're searchable and and indexed by your workflow so say I want all logs from the output of this workflow step bam and you can sort of search through them and explore them understand what's going on yeah there were questions all right [Applause]