SF Scala: Alexandros Bantis, Scaling ML Experimentation 5x at Tubi

SF Scala: Alexandros Bantis, Scaling ML Experimentation 5x at Tubi

Recording: SF Scala: Alexandros Bantis, Scaling ML Experimentation 5x at Tubi

please welcome Alexandros mantas from where to me and this is our opportunity to find out what really does to really behind the scenes his talks gonna be on scaling up ml experimentation had to be fivefold so I'm going to talk tonight about scaling up ml experimentation 5x and Beyond so basically I want to talk about how Scala is helping to lea to become data-driven organization and so specifically two projects so the first one is ranking service which serves up batch compute recommendations to personalize a user experience and then the other one is popper engine which schedules and runs ml experiments and my so ranking service has been in production since the beginning of the year and we've helped to scale up the velocity for the ml engineering team by 5x and popper just came online for us maybe about a month ago and it should increase it many times more so my goal tonight is to kind of talk about some key things that have helped it to be successful and so I'm hoping that there will be some interesting things in there for you so before we get started let's talk a little bit about to be this is the home screen that you see when you first land people go to to be to watch movies for free no signup no subscription no credit card you just you you just go there and you can watch it so we have over a thousand advertising partners we have over 200 content partners we have over 15,000 titles and over 20 million active users so a problem so according to a study by Netflix people will spend about a minute trying to find the title that they're interested in and after about a minute then they just give up so personalization helps users find the title that they're interested in as soon as possible so that's kind of like the core thing that I'm working on so here's our home screen what we want to do as much as possible is move recommendations left and up right so our goal is everything that we think the users going to want to see we want to shift up to that part of the screen so that's kind of our problem that we want to solve so this is kind of the old way of doing things it was built on nodejs and Redis so the experiment engine Facebook's plan out to run any experiments analytics receives experiment results Redis is a persistence layer and also an in beta in interface to the homepage service so homepage service is what builds that home screen that you see so the home so in order to run an experiment the home page gets a request and then it makes a call to the experiment service and says okay what do you want me to do you wanted me to go left I'm going to go right it gets a response from the experiment service it fetches what it needs to from Redis then it delivers the the results to the client and it publishes an event and Linux so to run a be experiments the the ml engineer so first they got to actually create the the model in spark data pipeline and then you publish it to they would push it to Redis and then you gotta update the code to the homepage service in order to run the experiment you also have to update the configuration of the experiment engine and you got to restart both of these services in order to run the experiment so maybe it's time to upgrade a lot of these applications were built when to be was small scrappy startup a lot of choices made at the time that made very good sense maybe right now they don't make such good sense and more it's time to revisit some of these things so we had the usual suspects for upgrading better fault tolerance lower latency throughput more maintainable code but actually in this case they weren't the catalyst for change so we had a people problem because every experiment was an endless cycle of please and I'll get right on it so so every time an ml engineer wanted to run an ad experiment they had to ask back-end engineers to make code and configuration changes and this created PRS and tests and code reviews and testing out in staging and deployment windows and this is all suffer team that already had their own work to do so this is the new way personalization is extracted out of the homepage service because homepage serviced as many things in order to assemble that homepage the personalization is just one tiny bit event so that was extracted out into ranking service so now the ml Engineer pushes pre-computer rankings Torretta s' from and then ranking service ingests them it uses to handle a request from the homescreen the experiment engine was replaced by popper engine which not only runs a B experiments but it also have a UI and you can schedule experiments in the future and of course the new system is more fall more fault tolerant as lower latency our throughput more maintainable code but the biggest win is that the data engineer doesn't have to ask anyone to get right on it ever again so tyrannic a B experiments again they just create the model publish to Kinesis and update the UI that's it and so they lived happily ever after until the next feature request so there are a lot of things that contribute it to success with this so we have testing tools like ammonite docker and akka we actually use akka to load tests akka Scala is just and I hope I'm preaching to the crowd here but scholars just an awesome language for that makes immutability easy and is functional it's object-oriented Sylla DB is just is a wicked fast C++ rewrite of Cassandra and gr PC and protobufs just very performant serialization is much much faster than then JSON plus a compression and you get a certain level of type safety also with it and an akka as well so I could took talk about all these things but I I want to focus on just three things domain modeling I think this is an area where Scala really shines Sylla DB which has been a big win for us here a to be and akka the bid parts so so let's dive into domain modeling and ranking service again it's used doing pre computer rankings that is getting from the data pipeline so the foundation of this is an ID it's a string value and it has an enum of ID type and this we kind of have because of ERP C and protobufs so we have a lot of different systems here iOS Android ITT OTT like for each one of them they have a different idea of what and ideas some string UUID uppercase UID lowercase integer so we just unified everything as a string and a target represents a non-empty set of ID's so we can do things like we can not only express the idea of a device ID but we can express the idea of a target as being a set of device ID and say platform or device ID and user ID just different lots of different combinations so a ranking contains three parts it contains data metadata and the target and the model state here is really the key thing because we have a way of expressing within the domain that something is either in a release date or it's a candidate to be experiment and we so rank and service saves these based upon the target and then also either its release or it's the model name and so the result is that we don't have to do any code changes in order for data science to do new experiments they don't have to tell me at all I only kind of learn about it after the fact so here's the protobuf again it's like a very simple and the kind of simplicity here basically has allowed us to use the same kind of model and we've had a lot of changes during the year and we've really had to change the the actual API all we've made almost no changes at all so that's really helped us to keep down the amount of code change that needs to be done in order to handle new features so this is a kind of small sorry so this is like an actual ranking class and basically we have to extend this idea of a ranking and we have to implement one function and we have this new set of functionality so and again this is kind of like the database schema that very similar so when some when we have a new type of ranking that needs to come online we basically have to maybe implement a new ranking function create a new table in the database maybe a few other changes and we're good to go and because everything is very strongly typed within the system there's no strings or ins the compiler really gives a lot of support as to making sure that we're making changes in a very safe manner so so again I think that like this is this is where Scala really shines for me is just the ability to take some sort some domain and express it with great clarity and precision and the compiler and maybe some tests are going to really help make sure you're not doing anything crazy all of this could have been done in any language but the best I really strongly believe that the best the best way to do this kind of stuff is with is with Scala so so far as the persistence we had we considered a lot of choices so SQL is very mature it provides a lot of structure in our case though we're storing protobuf binary of a ranking we already get that structure from Scala and from protobufs so we also looked at MongoDB and we use MongoDB here it it provides a very rich query language based on JSON so they're scaling limitations here in terms of Rights because all they all have to go through the masternode Redis is used extensively at to be we have a lot of experience with it it's simple it's fast they're scaling limitations with it in terms of Rights because they all have to go through the master but okay so in terms of Redis the things is that it you have to do extra work for high availability and disaster recovery and it's it's limited because it's all in memory so we needed a solution that would scale to hundreds of millions of rights per day have very low read latency high availability disaster recovery and cassandra ticked almost all of the boxes but it's not known for high performance reads typically you would add a caching layer maybe a cache warming service as well or you would make your application stateful and so this would have become an a cluster so we ended up going with Scylla DB so why well one thing is that it's the same as Cassandra it's the same cql language the same tools same storage format the same drivers so that creates a lot of simplicity which for us is is an important thing because I'm the development I'm the DevOps I'm the Sree I'm I've wear a lot of hats so to the extent I can use a tool that helps me to be more productive it's a big win so so it's the same but it's also more simple so don't get me wrong I understand how to tune the JVM but again we don't really have an infrastructure team here so I would really prefer to spend my time building things rather than tuning big clusters of database so it has a smaller footprint which is also very nice you can run Scylla DB on basically the the largest possible instances that AWS has and you can get a hundred percent utilization out of that so what that means is I'm never going to have to be spending time in front of a dashboard managing 30 40 50 60 70 instances it's always going to be a cluster of three instances as as the needs grow all I need to do is scale up the size of the instances and then finally in comparison to Cassandra we have a bunch of init scripts that will do all of the kind of tuning and configuration for you and like their philosophy basically is that any kind of setting a machine can machine or script can figure out what is the correct value at run time that at when it's first being installed or maybe it shouldn't even be configurable and the system it can just tune it automatically and again the biggest win is that it's faster than Cassandra for the reads so right now the of node of for instances and they handle about a hundred million writes per day from batch jobs generated and spark and air flow and at peak they're processing about twenty thousand writes per second and even at that load the performance is really good the average is about 400 is 600 so during peak load the performance will jump to about six hundred microseconds on average the P 95 is will jump to about 1 and 1/2 milliseconds and the P 99 to about six milliseconds and we're working on bringing this down and and incidentally like one of the reasons that the average is fairly low is because Silla DB has their own caching built into the system so that's why when you're doing this you kind of you don't need to implement a caching solution because caching in memory caching is kind of built into the system so like what's the secret sauce for it so Scala uses a shared heap and that's like that's one of the great things is because of that shared Heatran immutability becomes very simple because you have all of these pointers that you can pass around and so immutability becomes much less expensive but there are problems you can't just throw more threads at a problem it doesn't necessarily scale well in Cassandra each JVM is a node in the cluster insula DB each CPU is a node in the cluster so there's no context switching there's no contention each node the cluster is an asynchronous process pinned to exactly one CPU and if you have other processes running on the machine you at startup you would make sure that there's say 1 or 2 CPUs that are dedicated to other stuff running on the OS so queues are a fact of life we can't escape them it's just a question of where we want that queue to live so in Scala uncontrolled futures result in runnable queuing up inside of a priority queue of an executor service and all of these solutions acha acha streams zio Scala said tasks they're all about moving that queue closer to the application so that we can do intelligent things like cancel a task that's being queued up for too long the same problem exists with Cassandra it relies upon queues in the kernel space so Scylla has all manager of schedulers that prioritize some tasks over others can cancel tasks can do all manner of slicing and dicing to make things more performant and as safe as possible so it's just it just a really not it's been a big win again because I was able to keep my applications very simple as stateless applications so A to B we make use of type of libraries like cats but we're in auto shop so akka introduces some level of complexity because the developer needs to work think across three different paradigms object-oriented functional and also actor based and that that can be a little bit scary we can use akka streams but while it abstracts away the actor paradigm it also introduces yet another DSL so let's take a peek at popper engine and see how just plain dumb old actors nothing fancy can add value manage complexity so recall that popper engine is a place where the ML engineer can schedule experiments run experiments it's very fast it has a p99 of three milliseconds and it's fast because of the simplicity so a namespace is a place where experiments can run with certain guarantees it has state so it has there's experiments that are running within the namespace ones that are running now and ones that are being going to be scheduled in the future and it also has segments and these basically represent and a claim of allocation of the of the space so say that it's the namespace of all users who use the Android to be app so the more segments that an experiment has ownership of for the time it's running the more the more people are going to participate in the experiment so this is the bare-bones interface of the namespace class the namespace is immutable it's an immutable state machine and so the API is again like a very simple version of the interface it expresses like two calls so one is you can evaluate evaluate is very simple in most cases you're just generating two hash values and you're using them to do array lookups and update on the and changes the ownership of the segments and so experiments that are going out or coming in or their changing phase now the fact that namespace is immutable provides very good guarantees but concurrency is tricky and anybody who tells you otherwise I've got a got a bridge that I want to sell so what do I do so I could introduce concurrency into the class so that it would regulate its own state but then in that case it's no longer immutable I could what else could I do I could I could just put all of this state inside of an actor and then I don't have to worry about that but then if I could just could have made namespace an actor but both of these approaches are not satisfactory to me it's the namespace it's a very complex business logic and I would need to mix functional and object-oriented and actor kind of paradigms together in the same space and just thinking about that gives me a headache I don't I don't want to go there so so instead what I'm going to do is I'm going to have a separation of concerns so the so I'll have one that is object-oriented so the namespace implementation class is object-oriented it's an immutable class the namespace companion object is going to be a purely functional so no side-effects referential transparency so everything that I can possibly put into the companion object I do so that I can do this space that's purely functional that gives me very strongly typed methods that I can use to help me actually execute the business logic in the in the object-oriented class and help me manage state so it's referential transparency strongly-typed all these things that provide value from functional programming and then so here is a here's a bare-bones namespace actor so there's no functional programming here there's no object oriented paradigm it has no business rules its only purpose is to provide a safe and simple context for the namespace class and and really as actors go it's it's quite dumb and I like working with dumb actors because it helps me sleep at night so as in conclusion the the core problem that needed to be solved with people we used engineering to solve the people problem not a process problem people problem and that's that's really great when you get a chance to do that we helped another team become very very productive and we did that primarily through good domain modeling we could have done that in any language but Scala because of value types and algebraic data types these immutability object-oriented functional is like a perfect fit for that so silly DB is a great tool to have in the toolbox or ranking and service it allowed me to use fewer components not have to manage not have to deal with complexities of having a stateful application and it's been very very stable and reliable for us and finally like Scala is a polygon language it which is both a strength and a weakness so having a clear separation of concerns object-oriented code in classes and functional code and companion objects and actors where necessary just providing the illusion of synchronous programming is a big win and these are the some of the key reasons why Scala solidity and akka they're helping to be to become a data-driven organization [Applause] questions thoroughly like a high-level library to get to the to these databases to Cassandra and ancilla or do you have to do the low-level calls to these to the database first and is it Java or is it Scala specific so it uses it uses cassandra drivers so you could just use whatever implementation you feel most comfortable with there are so there's a Java there's the jaw there's the data sax java driver there are scholar wrappers of that and actually Scylla has a fork of it as well so whatever libraries you would use with Cassandra you can use with with Cilla's whoa any other questions just it's blend out still in the picture [Music] so so we are not using it so basically with with the system that we have right now we the experiment engine it it works as both a scheduler with plan out when you want to start running a new experiment you need to update the configuration yeah so right now I have aa per engine is running in production right now and it knows experiments weeks in advance so they're already there I don't have to do anything when you know it at midnight of whatever they when the experiment goes live it'll just go on I don't have to do anything so I'm sure that anybody who's working with kind of data the models like you have is going to see that you're you don't have uniform data analysis like you're gonna have as if Ian's distribution where some shows are far far more popular than others how is like that kind of zipping and distribution impacted your system performance so we haven't we we haven't had any problems I there's so there's been like a couple of small incidents so far this year but in every case it's been a people problem it's been me I push code that wasn't a hundred percent ready but still the DB have had zero downtime from it and the application has just never gone down I've never lost any data it just it's just always there questions you framed this as solving a people problem where you kind of cut back end engineers out by letting data scientists do things directly do you ever have problems where things became too easy for the data scientists and there was sort of a you know you need if they find you found you're needed the back end engineers to make sure that the code was you know not going to consume too much resources or something else I don't know so so I think like the philosophy that we've tried to have that we've tried to have is that like so what we do is very hard it's not easy conducting an experiment is not easy you have to make sure you're answering the right question you have to make sure that you're measuring things correctly you have to make sure that the the different systems can synchronize with each other there's so many things that can go wrong and so you need to be able to focus your mind on quality so in the same MO so I've been a big advocate at our organization of Scala fix Scala style Scala format because I don't want to spend my time with code reviews talking about minutiae I want to I want other people to think about my code to find out if there are any quality issues if it's actually going to work as intended and that's kind of been our objective here as well is in the same way that Scala style and Scala format takes your mind away from these little details that are just a distraction popper engine and all these different systems are not there to make it super easy to run crappy experiments on the fly they're there to ensure that we don't introduce friction and you can off and the data engineers the application developers can focus on creating quality expect it's anymore questions great well thank you for coming tonight to be to be our recruiting so please thank them by going and seeing them and saying you want to work for them you're right I'm here tight and if you're too too shy to do that come and see me private perk and another introduce you to them thank you so much and before you all go next week we do have more talks for you we are at Twitter Twitter's RSC team I believe we're gonna open source fast scholar compiled project called RSC which I don't know much about are gonna be talking about that product there the creator of metals will be at that event though I couldn't convince them to do a talk but maybe you can all convince him on the day so again please come and meet all out there next week's Twitter event thank you so much thanks for coming thank you [Applause]