Scale By The Bay 2019: Gabor Melli, Large Scale On-Demand Low-Latency Near Real-Time Predictions
[Music] welcome I have been a playstation now for about two and a half years and this is an update on our journey and maybe I'll get a read from the audience first who among you have deployed a machine learning model into production awesome and which who of you consider yourself ml engineers yeah I am reaching out there because at the end I kind of describe the work that were its marketing for an ml engineer we're advertising for my engineer and I keep trying to formulate a model for the kind of work that that involves so I I'll share that with you momentarily what I mean first by this low latency so under 20 milliseconds faster than the eye blinks and my near real-time is you've done something and within five seconds we we can now react to it and large scale meaning tens of thousands of requests per second and that's and that's because we're Playstation where we push the boundaries of play we want to be the best place to play and now we can do so much with data we started kind of humbly 1994 with PlayStation 1 and that was the third most successful console of all time just last year we receive we sent out classic and and I have one to give away but I was trying to find ways to do this fairly what is this do any of you have a birthday in the remainder of the year three if you want to playstation then raise your hands so I see one hand two hands okay so there's oh three how about let's be a little fair in November okay we have a winner so after the talk come on over and get your PlayStation Classic and that was the third most successful console the second most is the ps4 we have just broken through the number of sales there so on a good track where we've got a nice precedent we've got lots of data now and we have this ambition of continuing we now have virtual reality headsets and a subscription service where you not only do we want to sell you not because now you're a subscriber we want to recommend things that you can play because it's all all there for you also we want to prepare for the PlayStation 5 which is going to have a solid-state drive it's gonna have 8k haptic feedback and it's gonna be backward compatible so right away you're gonna get all that a gaming experience from the PlayStation 4 so this is sort of really what we're preparing for but for now we're we're focusing on the ps4 this is what I can share with you and that's the low latency experiences that we can deliver that are personalized kind of very relevant to I'll start with a kind of clear one games recommend games for for people maybe also because your friends are playing them we have a rich social experience inside there within those 100 million people they often play with the buddies and it starts with a person so we started a year and a half ago to build a profile of our customer and one that is non-identifying so no names no addresses so my team can work with this data because it's non-identifying it's how you've been playing with a random key is that for the user that we can then send to the console we also want to take care of the context so it's a day time are you on the phone we have mobile experience we have a web experience we have a console experience and from that be able to submit from the tens of thousands of games that we have in a decreasing order pass it through a model get the scores and all in under 20 milliseconds 50,000 times per second for all the users that are on the network worldwide certainly if another person comes along in the same context they should get their own personalized scores that are most relevant relevant is a important that we define what that is we work with the business in some cases it's by but actually we make it even intend it to be more interesting as you buy and you play we don't just want to recommend you stuff that you throw away we want you to play so we wanted really engaging also we want to take care of the context let's say it's a different time of day you moved temporarily that should change also you've turned off personalization and we can still at least personalize for you know basically the time of day what country you're in or one other way that this is important is let's say your brand brand new user there's very little information about you we need to support that I'm setting it up for giving you like a taste of the kind of ml that we do but there's many others but this at least makes the concrete for you so for this we've built our own internal relevance scoring platform enterprise-wide so the intention is that any engineer can come along and use this core this platform to deliver their own scores self service on top of it or below it the support and we've done developed three other platforms I'll talk about these two a little more certainly the nml platform we've built our own we're on AWS on the cloud so we're leveraging some of that componentry there I'll tell you more about it all over and then this larger framework with online experimentation our data lakes platform our data ocean and the ingestion of data into that ocean so this ml pipeline is intended to help any ml engineer and maybe even data engineers to also deploy data assets offline and online so not just kind of in our batch mode but also in the real-time ability to retrain with there's plenty of challenges but I thought there's how do you facilitate the deployment of these both patch and streamed the data comes in we want that five-second latency on the updates of information how to create these machine learning data sets machine learning data sets for training have this interesting probably you need to do tight time travel so you want the information from the past to predict the future so you need to have this facility to quickly talk about the past versions of the data and current versions of the data so you can have that your target be in the future there's no information leakage and plenty of monitoring this is a high level design we keep evolving it so this was now already changed since this one but you can see the role of s3 Jenkins github to commit the code and run the the inference code and kubernetes for prediction so this is the ml platform then the customer and item intelligence button this is an abstraction based on the description of data that we've done in the processing so we want a nice way clean way predictable way to talk about the users or the customer and these items that we need to predict so very large everything's going to be an s3 in park' files but also the information that we need real-time that the information that's going to be read within one or two milliseconds to run to the model that right now we're using aerospike also what I don't show here is we're using either spark streaming or soon flink to update those those values plenty of lessons there for us about how to keep this information fresh and also some governance around it so that we can know which is the right version of attribute we have different teams creating the term age and and they have many of them we want one so we want this is all that kind of information that that platform needs to support there isn't really anything out of the shelf that just delivers this feature store for these two entities so we built it ourselves now this was a relevance platform that I started with just mentioned some of the challenges to regularly scores or hundred million active users we don't know who's gonna turn up next minute so we would have to rescore everybody we started out that way where we score on everybody every night but that gave us fairly slow-moving predictions and also we set all our resources so now we're on demand since May now the person comes and we get this information for them how do you personalize less active user so I'm gonna focus on this one this is one of the one of the important challenges that we encountered right away so in these recommender systems there's a long tail of users that don't really have a lot of interaction they show up every other month or every two weeks they have fairly minimal engagement and a lot of items that are maybe falling out of favor but they still have a some life solutions like collaborative filtering who's worked with collaborative filtering here and a few of you who are really great at the head and trunk of this distribution a matrix factorization or even around neural network methods and only use interaction data but we really also wanted to be able to be have a good experience for these people and for these items and so how to do that we made use of our rich profile so based on your tenure how active you've been in the last year all that information that customer profile can now be used by our algorithms because we're using those features for storing those in aerospike so that the neural network model or the random forest model can can make use of that information same with the items we can see how long the item has been around what genre is the item so even if it does if it's not a lot of there's not a lot of activity right now it can still use those other in bits of information so our customer an item Intelligence Platform is really critical for delivering the intelligence we found that helped us address one of the challenges to build this relevance scoring I won't go over all these other challenges to deliver the platform I can talk with you offline about them what I want to change topics there is the actual engineering on these three components the data ocean the a/b testing framework and our ml platform so I'll move these around a little bit and these are the components that my engineers often work with most for doing development for automating their jobs are created jobs automating them and then monitoring them alerting on them get that call in the weekend somehow our inference model didn't deploy properly and it's down has to be part of our tooling then these are the key to us that differentiate the kind of ml engineer we need to be able to explore data need to be able to process data to extract the value valuable information a predictable manner doing the feature generation training data deploying the inference and then doing a/b test I'll go through these as well well to characterize how we do it and why we do it so identify data sources there's this data ocean has a ton of data we need tooling to be able to find it nothing too sophisticated here Alisha and is what we use we use presto sequel to be able to do aggregations so this is typical data science work analyzing data now we get into some of the challenge of doing something every day in a very repeatable way or every hour or even online when it's extract join so we need to for example parse JSON to extract the values that we need from that that object and then clean them join them with some maybe product catalog to make them really useful Parkay is 3 or maybe and and here by the way where we are using our Adina engineers often do use Scala with spark to do this work I don't mention it here by that for the online version of them we spark streaming or flink to to push the data with that we consume through Kafka so our challenges are the part of the unit and acceptance testing and this is there isn't kind of a formula for describing for putting the specification of these data assets and these transformations that we that we need to receive so we're working through that that's not much there not much of a tooling component thereby more of the process so if you have any suggestions at welcome them how to detect job failures that are actionable versus ones that are gonna retry data quality monitoring is continues to be a challenge and tuning tuning for now the spike jobs so this is one of the tasks important as that our ml engineers - we are for for some of them we were referring them to a data engineering team but often were the first people who do this now comes the feature generation so here again we use spark here it's mostly the data scientists kind of had so they they prefer to use PI spark to do this but we do have this component a feature store library and again pushing this into aerospike and into flink job error handling is is an important challenge because if the feature doesn't get updated now we're not getting giving the customer their recommendations that are really relevant to them or maybe there's a data a rule around that did they buy this game recently oh we can't tell you because we had a job failure is 24 hours late and staging environment so we develop this code and staging but our staging environment doesn't have a full copy of our theta ocean or data leaks that's quite challenging that it's hard to predict the behavior when we go from staging into production and it's pretty difficult to always get these snapshots of an entire data ocean or data lake environment a training data is now this is what you would most of you maybe know for as a machine learning and data science I'll go through this quickly I spark the ML libraries of spark are very handy we're also experimenting with Kara's and tensorflow they haven't quite found a way to beat this mark ml processes yet but we're working at it and one of the challenges here is that it's a very notebook environment moving from doing this development and notebook way world to then committing that code and having that stability and environment is still in the area where we're working through so I'd love to hear from you any any approaches you've made to solve that finally we have a model every week it's gonna be retrained or actually yeah every day or every week we were still figuring out what the value is between doing it every refreshing daily or weekly and here the path the data comes from aerospike the code itself for inferences in Scala we've serialized the models read them into kubernetes docker environment some of the other techniques that we need to familiarize ourselves with is G RPC and protobuf or for communication between systems when we deliver that the recommendations having a realistic test data is difficult shadow testing monitoring the output so we make prediction predictions predictions 15,000 times a second and we found that it's important to monitor where those predictions are trending off all of a sudden day they head south or north we know don't know why but that's unexpected so we need to to put a lip put an alarm on that finally a/b testing this is not too much doing involved here just statistical hypothesis testing skills in development yeah Mon Jenkins github as you develop to create these libraries we try to put some automation in here with libraries and with operations with Prometheus co-financed n su X matters because we need to be alerted when these systems will go down because they're driving customer experiences and all of this to say that these are all the tasks that our ml engineers do one of the things that well we quit quickly can recognize from this is that the ones I have highlighted here these are the kind of traditional data science e skills and the others are more software engineering skills so how do we combine those two we've one way we've done this is by holding an email University training within SAE to discover which software engineers can also perform those machine learning tasks those data science tasks and we've had a really good success with it here's a couple of our classes and we're off software engineers that are now ml engineers because we discovered that they already had latent talent in doing these tasks here we trained them up on doing these tasks and now they're part of the ml team and so I recommend well for your organization's to to discover and the ML the engineers inside the organization that can maybe aspire to and actually have a skill in in ml as well so that is our journey so far a review of all that sort of methods that we've decomposed the problem and some of the technologies that we're using along the way more to come but for now that's all I have for you thank you [Applause] [Music] all righty and we've got time for a few questions I will pass this microphone around hi you great talk you mentioned a be testing sure in at the end when you're serving models have you also thought about moving forward on expanding to say a contextual bandit because your pad some contextual information about each user sure yeah it is on our roadmap for our an online environment testing framework to have multi-armed bandit contextual bandits because the richness of the information that doesn't well an a/b test is just a fairly slow way to try to identify this it's for now we're still at a beat s work but yeah it's on there on the way are there any other questions can you tell me can you tell us a little bit about what you do to get some of that personalization in the tail because that can be pretty darn tricky too to get interesting features and recommendations out in the tail sure so sure that there's a long tail of many many users many many items that don't have a ton of interaction and that's because we're focused on a specific experience that say we're at the store and there's not with a game you're gonna buy a game every few months it's very slow moving information but what they do do is they interact in other experiences they play they send me messages to their friends they like posts so all that information that it gets summarized in their profile so all this other behavior around them is not condensed we're not quite at using an embedding space to condense that information in some obtuse way we literally have feature engineering of their most used other experiences how much they play and how much they interact socially it's an example of how we can then characterize the person to say aha you're more of a sports person over a shooter kind of video game any other questions following that do you you know this anytime recommendation happens Isis like you can just get into a funnel and closed-off - is there any metric or interesting thing backed takes like a linear combination of genres into account for that recommendation like epsilon and multi are embedded for example not right now because we want to be very general in our description of the person so that it can be applied in so many differents experiences that we don't want to tailor the information that we store for the specific engagement that the person has so one day we may say on that there's more about there's additional value in actually using what they do in this store specifically for them and then testing with with the multi-armed bandit to see what bits of information are the most helpful so not for now any more questions well don't forget your classic and thank you [Applause]