scala.bythebay.io: Davis Shepherd, Building a Machine Learning Orchestration Framework in Scala
you I'm going to be the last talk between you and lunch anyway a little bit of myself my name is Davis I work as an engineer and the personalization infrastructure team and I'm going to talk about kind of a marquee project we were working on inside the personalization team to deal with kind of the collection of challenges that we've we've we've had to deal with when trying to personalize the netflix experience so to kind of motivate this a little bit talk about okay what does it mean what is the what is the goal of personalization and Netflix of course now there we go so our goal is to help members find content to watch and enjoy so we can create this virtual cycle of satisfaction and retention amongst our members it's sort of the core tenant of what we want to be able to do so now what does it mean to actually personalize the netflix experience sure some of you or may be familiar with this page and in essence everything that you see in this page is a function of personalization algorithms not only the rows that appear but the order in which they appear as well as the title the order of titled within each row so the goal here is to make sure that you are as close as possible as to what we think you're you might enjoy watching at any given time on the experience and all of these rankings and recommendations are driven by machine learning pipelines so what does kind of a machine learning pipeline look like Netflix and this'll probably familiar to anybody has done this we start with user selection you know sort of getting our sample data what we want to use to inform our model then we have to label that data we have to drawn through future generation feature encoders you know some notion about how we label that data and then that's fed into obviously training our model or may potentially a collection of models and then we go through a process validation deciding okay does this meet criteria for prediction accuracy have we found a stable sort of minima for our convex optimization and once all these things are sort of satisfied then we can sort of published the model for use in online systems now you know this seems kind of neat and tidy and straight for but obviously in reality things are never quite that straightforward and it's a little bit more heterogeneous for example when you actually get your data that may start as a hive query you know deciding okay I need access to active members and needs to access with our current sort of data warehousing solution and then we may have a collection of Python software to decide okay how we going to separate this data into training test validation and then that'll be fed into potentially a myriad of model training pipelines whether its regional or global model and each of those may potentially leverage different technologies they've different requirements a global model is going to need large a larger quantity of data so we need to leverage spark and use maybe a more approximate sort of machine learning approach whereas the regional model we can use something like our to sort of do a more direct optimization and these pipelines need be able to run concurrently because obviously we need to be able to meet our time budget for retraining these models and then once all its that done then we need to go through a process potentially of model selection and that'll use potentially yet another sort of technology stack whether that's just ask all application standalone application and then you know potentially again a separate stack for publishing a tender to into our online systems so this sort of motivates the this or the issue that we've kind of run into which is that there's a lot of different technology that we need to interface in a cohesive way to ensure that we get sort of reliable easy to use access and the ability to experiment with in personalization so fundamentally in machine learning space it should come to a surprise there's a ton of innovation and that breeds heterogeneous environments to run on you know that spark or are or you know dr. applications or graphics API so we needed a platform and sort of a framework that could both orchestrate and execute across these heterogeneous environments that said spark is used very heavily inside of the personalization team so that was one where it was sort of a we had to have a batteries included solution for running spark in these workflows so that was definitely a hard requirement for us now these first two points kind of mandated that we we be able to sort of separate for castration from execution we should be to sort of change out how we sort of farm out the work from the actual implementation of orchestrating these workflows now we also to sport a bunch of her team's whether it's search or video ranking or you know you know the similar video team so we need to be able to support multi-tenancy right and be sort of conscious both of our time budget as well as cost budget for for running and training these workflows and you know there's also sort of machine learning constructs in particular something like a parameter sweep that may mandate that we have like a 30,000 sort of branch fan out for a sort of concurrent training right so there's there's sort of a scale aspect here that we need to be very conscious of and with these challenges in mind we came up with our Maison workflow system now this is a general purpose workflow orchestration engine and scheduling framework it currently designates execution to resource managers like mazes or yarn but fundamentally it's optimized for machine learning pipelines and visualization and we actually put out a blog a little while ago if you want to follow this link or find it on our on our tech blog if you want to read a little bit more in depth about sort of all the features that it has and I'm just going to kind of touch on sort of the high points in this talk and I should point out our plan is to open source this in the future so our hope is that once we have you know what we feel is a concrete solution to the set of problems we've salt you feel that they are applicable to the industry at large so the architecture is it's it's so maison is a as a scholar application that sits as a maze O's framework it manages sort of state persistence it implements obviously the mezzos framework itself we have a collection of plugins and extensions to enable the system to interface with the various subsystems and sign a netflix whether that's the you know the data announcement system or the various places where we might be able to ingest data and that's we need to be able to support the ability for users to come in and say oh I need to use to Lex I can do that very simply with a simple extension there by Taylor Maison for their particular use case and then execution itself once we've sort of captured that state and how the flow of the workflow is going to behave the actual execution is delegated to a Maison may Zoe's cluster running sort of a custom a Maison executor to enable us to sort of communicate nuanced information about the the runtime and these workflows back to the actual Maison service for surfacing in a in a UI for the user now talking little bit more about this plug in an extension system you know we wanted to be able to support a lot of different sort of sub-step types you know so the hope is that when you know the algo engineers come to use Maison it's sort of batteries included right you know we already have sort of de facto solutions for the various sub pieces they may need to sort of glue together and you know a lot of this was sort of aided in particular by the fact that we could use sort of scholars rich sort of trick trait system and multi inheritance so we have this this notion of sort of a thin cake pattern where we glued together various sub implementations and it's very easy to sort of create these new pieces of execution for the particular use case moreover all of these pieces need to be parametrized obviously so we needed to sort of support a rich parameter system you know in the simplest case you know hi query needs to be motivated by a timestamp like when you can actually fetch your data and then this is going to emit sort of the location of that user data set which will be consumed by steps downstream and you know this sort of process can ad nauseam can serve as you know snippets of useful information need to build a flow through a workflow so once you have these sort of these steps parameterised then you get to the part where your workflow actually has to run and so when a step is actually executing we need to ensure that we surface as much sort of useful information to the user as possible and that comes in the form of you know runtime logs as well as access to like you know sort of new ones progress information as well as artifacts about you know maybe okay what's sort of your feature important information for that particular step run or how many records did you process whereas the location of the user data set that this this thing is mitted and all of this sort of necessitated us to use this sort of custom executor that I mentioned earlier now one piece of this artifact system that is particularly useful and important has to do with visualization so a big motivator for this was the notion that you know very often in ml pipelines there's lots of intermediate data that you want to inspect and not only for the instance of a single run but over the course of multiple runs so we want to be able to chain together all this information from a single step so that you can sort of track its behavior over time whether that's something like feature importance or convergence time and understand how it behaves over time in the system so you can identify useful signals something like feature importance is a great way to sort of potentially inform future hypotheses that may go into other a/b tests later now as I alluded to earlier workflows are rarely as simple as do a then P then C so we needed to sort of include a number of structured content there constructs for how to sort of stick these Legos together so in the simple sense obviously we can support something as simple as do a then B then C but we may need to we also need to support this notion of sort of a fork join notion within a workflow whereby you need to run several sub flows you know weight on their weight in their completion before proceeding something like or where you have sort of conditional flow through your workflow that may be mandated by you know the output of an ingress step and we also need to support this notion of for each which is sort of where that came from that requirement of need to do like parameter search right so the actual fan out of your workflow will be ending will be sort of controlled by you know particular tuning parameters you've decided sweep that i want to run for this particular workflow and we also knew be able to support loops you know in particular if we get to a validation step that may indicate we need actually go back and rerun a subsection the workflow again or expand our user set if we think data quality is low or something like that and now we have two ways for user just sort of construct these workflows first through the web UI we have essentially just graphical way to represent these and manipulate them and create them and that's that's useful for sort of debugging these workflows prototyping things and sort of getting used to sort of how the system behaves but the more common way that people actually our users actually create and manage these workflows is through a DSL and this is where the sort of the power of scala was was sort of made clear to be through a correct choice for us was the ability sort of define a rich type checks fluent system for defining these workflows and managing their life cycle over the course of development and this is how the vast majority of users describe and manage their workflow definitions so we'll talk a little bits of like what was the scala stack that we end up using to build this application so the web server is built on top of a KH ttp making heavy use of the routing dsl and spray json you know we started to dip our toes into using the actress system on the construction of our actual mezzos framework so this is sort of one of our larger projects in scala within the team and so we are sort of taking a measured approach to trying out these new new libraries you know we were sort of wary of big heavyweight frameworks and and we wanted to sort of be able to test out actors in a small and a small way we also use scauldy for dependency injection and our future work is to actually extend our initial experience without ropes is quite good and we're looking at actually using the SF fsm trait to sort of restructure most of if not all of our current state machine with a hope that we can leverage sort of higher level concurrency for workflow processing because you know as the state machine group sort of organic Azaz requirements came up and we feel that with fsm we can take a much more measured approach to read ascribing the statement seen in a very sort of clean reusable and manageable way those numbers everybody likes numbers our our current primary maize maze on cluster manages you know hundreds of them current jobs you know in any given point will run between 100 and 200 a work flows with thousands of steps and the execution these workflows is farmed out to a cluster of about 750 nodes with 6,000 cores in about 40 terabytes of memory and this cluster essentially obviously manages the runtime of the steps themselves the spark frameworks that are sort of started as a function of those steps firing off and a few odd dr. applications as well and we have a few smaller clusters for users to sort of test out an isolation if they like as well so that's it for maison this is my contact info if any of that sounds interesting you well you know is where we're hiring as well in the personalization team so if any of those problems sort of excite you where you be interested in please come and talk to us yeah there any questions so you give the dsl to the fire attack correct although to be clear it's not a dag it's because we do allow cycles but yes so the question is we use dsl to define our graph yes yeah so the the Maison workflow system itself actually has started to repeat the question you know if you modify a DSL definition how does the lifecycle of that update actually manifests itself into the runtime system so the runtime system actually has a rudimentary sort of versioning system so you can sort of track state and you sort of manage a prototype and then you can sort of make that active so you can independently version your DSL in sort of a traditional build system you know under under version control and then when you have a stable build you can sense you say okay this is my new workflow definition I'm going to publish that to the runtime Maison system so that will actually publish a new version based on your new dsl definition that'll get uploaded the server along with any sort of changes to your you know runtime artifacts that you may need and then that'll be made active for future runs at work whoa so the first part is entirely up to the user so I they I mean they could define a DSL but the actual uploading it to the Maison servers is merely an API call so it'll pull down compile their dsl translate it in the appropriate API call and then push that to the server yes so the question is are the various steps as separate spark jobs and the answer is yes so we actually use so the question is is you know you know are you able to share a spark context between two steps currently no and there's good reason for that you know oftentimes we want to be able to sort of isolate resource requirements for various steps because spark itself is is not great that's sort of adjusting its resource consumption over the course of a job dynamic allocation is getting there we're getting close to that solution but it's generally easier to make small focused spark jobs that use exactly the resource footprint they need and emit data for for downstream steps yes pre-processing so you parameters of your data mining models is it also let you go back and say get more sophisticated pre-processing driving that so so the question is I'm not sure I totally understand the question does it have to do so is does the dsl enable us to do think yes yeah I mean you could you could like you know define like your definition your workflow may hard code what your serve parameter spaces and then i'll do a cartesian search across all those combinations or something like that absolutely yeah i mean it's it's very he's very pluggable and moreover it's it's it's very generic right the idea is a step is really just a unit of execution we have some batteries included ways for how to define these steps but at the end of the day your dsl is as this Scala code right so you could define any sort of complex flow or how you're gonna manage the lifecycle those dsl's is very fluid so yes yeah so the question is how do we manage multi-user workflows now do you mean multiple users that are concerned about the same workflow or ah so typically you know we we don't build tooling around trying to sort of do like like strict version control for multiple users for the same workflow definition I mean we give sort of atomic updates for updating a workflow definition but the onus is more on the users to be able to say okay we're collaborating on the definition this workflow and you know the tooling incident Netflix is very heavily using sort of automated build systems and and and version control so those tools sort of solved that problem for us yes in the back so the question is where do we maintain state our current solution for state maintenance is Cassandra you know the tooling for Cassandra Netflix is very very good that was a very very easy solution for us to use right away so the question is is this data store a pluggable thing yes absolutely the actual API for doing state maintenances is a simple key value store so anything that can pull in key value you can store your stand yes yeah so to be totally honest I mean it wasn't we didn't have total clairvoyance before going into this with Scala I think a big motivator for us to choose Scala was the spark interoperability and we already had a lot of people who are on board with language right and and saw its merits you know just in general software development and when we got to the point of the need for this dsl to describe it it was immediate like oh like this is an obvious win yeah so that was I can't I can't say that we had like an immediate like yes we're using this for the dsl but it was it was it was a lovely discovery uh no I don't I don't recall that groovy was evaluated yes that's a great question so yeah so that so the question is is uh how does a stack up against air flow so have we considered our flow the answer is yes you know air flows is very much trying to solve a similar set of problems and the RR primary issue with with air flow was that because of its very Python centric approach it's very difficult for us to actually integrate spark jobs in the air flow and because spark was used so heavily inside the personalization team it was almost a non-starter for us so you know we felt that we could take a better approach that targeted our set of technologies and then sort of branch from there to sort of cover everything it is possible scholars yeah what did you see is your primary pain points that I might let get that ubiquitous across common pain point is you're saying like amongst our work clothes like so so what made you feel like I'm going to try to tackle this problem by creating rather than just making your scholar jobs so one of the things that actually when we started out with with the mace on air flow wasn't left bit mature about a year and a half ago that's that's one of the things and the other thing is because of the way we have service systems spark is to get in the first-class citizen natively and so it's not just the orchestration lab also being computer that's blundered in together you can see a lot of steps that you should think you can be with spark so that's one one one thing that stands out the other thing is also we have loops and so you know if you have if you have a standard CAD processor and you won't have loops in there and there were some use cases where we felt that we needed to support loose like when you had to something fail you have to validate go back and we're on the pipeline and so yeah there's there's that there's like a million virtual systems out there and so we basically try to give her our needs and it got enough adoption that we found that yes poster so the question is when we expect to open source that our hope is for 2017 we can't be terribly more specific in that I mean we had a lot of obviously internal pressure to cover some features and things like this and open sourcing a project in and of itself is a bit of a task but if anyone interesting up interest in helping us do that please again come talk to us but yeah they were shooting for 2017 yes that's that's a great question so he's asking about was so the question is what is the interaction between the people actually building the models and people building the infrastructure and I mean that's a hard problem you know it so we sort of have a separation between infrastructure and model building right now but you know with the the collaboration we want to sort of make sure we have sufficient overlap so that people were building a model are sort of aware of the infrastructure requirements but yeah it's a it's a common problem we run into those building models are potentially don't want to be concerned too much about okay what does it mean to be sort of resource efficient inside of a spark job and so we try to build as many tools as possible to try to abstract away a lot of for the computer infrastructure and try to build those abstractions up you know it's it's not perfect right now but Maison a sort of our first step to try and and and sort of bump that abstraction layer up because before that we essentially was was much harder than it is now so model for extensibility and based on is such that you can write custom steps and so one of the things that once people start playing with Mason and they figure out that there's just piece of code that is repeatable and use across multiple pipelines then they can essentially like to make that into a custom step and that can be reused for anybody else and so that's that model is everything improves the reusability and collaboration but in general we build a platform there and the data scientists and we other researchers chipped in to write the core code for so we had that in mind as a vision I'm very wanted to do and so we actually structure the coat off in such a way that there's a netflix pacific version and there's something that could eventually be open source and so that extensibility was there but we didn't pay much attention to it until we started running into the copy and paste problems and you're saying it yes can you speak to the our integration is it just free day off disc or is it java car passing data yeah it's not say again yeah it's it's it's latter so it's just java calling or it's not a terribly sophisticated integration it's not a super common use case but it was enough that we felt we needed at least include that and you know there's not a lot of coordination involved in that piece so it doesn't need to be terribly sophisticated yes ah so the common use case is so oh sorry the question was is what was the use case for loops in the use cases you know these these workflows will go all the way from ingesting data to publishing a model for available in an online system and the loop is a way for us to encode instruction about if you get to the validation step which is for the last piece before you decide to publish is it okay to publish or not and we didn't want to just like terminate the workflow and start all the way from the beginning we want to be able to start from a memorized piece and rerun from there to say you know expand so I say I got to validation and you know data quality was too low so we have to okay we gotta run back and rerun that step with an adjusted parameter to sort of get where we need to be yes so the question is do users have control over that dashboard layout and the answer is not really for the visualization piece they do but for the most part what's presented in front of the workflow is mostly a function of I mean I guess there's some subtle control when you write a custom step you can define certain interesting pieces of artifacts they'll be emitted and those will be presented in the dashboard along with other relevant data but for the most part it's it's it's fairly rigid so we haven't seen a whole lot of user friction or or control they're always here saying god I got I got it okay got it sorry so then can repeat the question again in that new context yes ya know so yeah I mean so okay now we're getting sort of a market questions about Netflix so yeah I won't go too far down this road the answer is no you know there's a lot of pressure for sort of it adding advanced features the netflix but generally we've wanted to make the the interface is sort of seamless and simple and approachable as possible and so we sort of erred on the side of a simpler side of the interface I mean we do here requests for more advanced sort of user interface but we've yet to have a strong answer for why that would necessarily improve retention engagement across the sort of the wider user base yeah nothing is not either do it so we obviously take an input for our custom to that serve at the end of the day every single literally you know areas of pixel in that interpret that you see have product matters behind them who are trying to all sorts of AP tests around it so once we get data are our key metrics being you know more screening people are watching more more attention to are staying more and so we basically use that data to figure out where the interesting to be in the end of the day providing to a choice to customer has its downsides too so that's something that we feel that we're getting good coverage and getting good streaming and we tried to optimize those experiences yes yeah so that's a great question so the question is how do we do with logs right because the stuffs running all over the place there's lots of interesting information so maze owes itself gets us a little bit of way there so we utilize the mazes API to surface the logs from the actual execution of the step on the mezzos cluster to the UI but moreover with things like spark right you might have you know 10 or 100 instances of those separate log places and to solve that problem we use elastic search and log stash sort of pull them into a centralized place so they're searchable and an indexed by your workflow so i say i want all logs from the output of this workflow step BAM and you can sort of search through them explore them understand what's going on key part is it's indexed by the first lady yeah then you can leave there were questions all right go eat lunch