SF Scala: Nikhil Tibrewal, Playlist Recommendations @Spotify

SF Scala: Nikhil Tibrewal, Playlist Recommendations @Spotify

Recording: SF Scala: Nikhil Tibrewal, Playlist Recommendations @Spotify

um so let's get started um good evening everyone thank you for joining us at the Spotify San Francisco office tonight for meetup um I'm here to talk about playlist recommendations at Spotify so I'll start with some some uh inform about how these are generated and then dig deeper into how we use scalding um throughout our playlist recommendations feature to uh to do some pretty amazing stuff and then I'll sort of you know tell about oh sorry sorry now okay hi uh so yeah uh today I'm going to talk about play recommendations that's Spotify um initially I'll talk about how these are generated and then we'll dig dig deeper into how we use sculling to um do some pretty amazing stuff for this so who who Am I'm my name is Nick Hill tial it's pronounced as Nick Hill um and I'm a data engineer on the Lambda Squad Lambda Squad is our team's name in New York and uh we work on the most awesome machine learning features on Spotify um I graduated from Cari melan University in Pittsburg in 2013 got a Bachelor's in computer science and an additional major in economics um joined Spotify right after I've been with them for a year and a half and I've primarily worked on playlist recommendations in my time there um just to give you a sense of Spotify numbers Spotify started in 2006 and we're available in 58 markets right now we have over 75 million active users 20 million of those are paying subscribers um we have a catalog of over 30 million songs 20,000 added new added every day um we have 1.5 billion plus playlists and a terabyte of logs are logged every day at least more more than that I believe um so this just gives you an idea of the kind of scale where I'll operating at and the kind of challenges we might face um going forward and making Spotify more awesome um just a quick overview on the kind of recommendations we offer right now we have the good old discover tab it's been around for a while and we have um track album and artist recommendations on on there and we have radio we have related artists for example this related related artist for Ellie Golding um and then we have discovered weekly this was launched a few weeks ago and and and has created quite a bit of Buzz um and then something we have something new that we've launched recently is playlist recommendations on something called a now strip I'll tell you more about that in in a couple minutes but this is the you know newest feature that we've added where we're serving uh playlist recommendations so what exactly is this now strip well this now strip is essentially a list of music human created uh playlists and shows for you right now this is the kind of music and and content that Spotify thinks is is best for you at this very moment and so along with this with this you know content of human cated playlists that are created by at editorial teams um we add we sprinkle through some different kinds of playes that we believe are are are good for you according to your taste so these are the recommended playlists and these are sprinkled throughout the script to tell you sort of to to give you sort of a machine learning base plus human created experience because that's where we believe the real uh real catch is um and this is available for some users you may see it now you may see it later on but it's it's it's being rolled out so so now you're probably wondering how these playlist recommen recommendations are really generated let's go into that next so at the at the base level you know the first thing that I want to note is that we only recommend human cured playlists and there's a few reasons for that we have about you know thousand plus of these playlists but the main reasons are that these playlists have really well-designed cover images cover images that are depicting what kind of content is in there sort of you know giving a user the sense of a really inviting us playlist content in there and then there and then then these playlists also have thorough descriptions that are describing what kind of genres are in there you know what when when we think these playlist might be appropriate to listen to um and then a title that reflects what kind of content is in there as well as an example you can take this playlist that has a pretty nice image of you know people having fun on a Sunday I think and U there's a title and then the description is you know telling you what kind of music is in there in this case as Indie pop but then the reason that we have to go through these playlists is that there's other kinds of playlists in that 1.5 billion that are um like this one the title here is literally and the cover image is not telling you much about the playlist you know there's there's there's it's a it's a mosaic of four albums in there but you can't really get much out of it so we can't really recommend that to a user and be like this is your taste you know they'll be like no that's not my taste um so let's go next into this recommendations pipeline that is that is generating all these recommendations the first step in this pipeline is the candidate generation this is the part where we generate a list of candidates candidates are playlist for each user and and then we we do we do more steps on that so but at the base level there's this there's this concept of generating track vectors that are generated from collaborative filtering and a few other models but basically you know these these track vectors are sort of a representation of of the kind of features that these tracks will have and these are completely Laten Laten feature Vector so we don't really know exactly what the features are but uh this is how we represent each track in this in in this n dimensional Vector um I I can dig deeper more into that later uh off um presentation but uh we won't have time for that today um and that's not really the focus of this talk but come grab me later if you want to know more about that um and then there's the concept of sort of vectorizing playlists um what this means is that playlist vectors are derived from track vectors track tracks are the base content of of a playlist which is where we have the most information about a playlist so you would you know sort of come up with with a play this Vector that is derived from these track vectors and then the next problem comes the next problem if it loads the next problem that you would like to solve is you would try you would want to find playlist that are close to each each each other right so for that we use something called anoy um this stands for approx my nearest neighbors Oya the Oya part was added recently before that it was something something it was it was created by Eric Bernson um who was who who used to work at Spotify as well but basically this this shorts these playlist vectors in an nend dimensional space uh using the approximate nearest neers algorithm through which you can query for for closest playlists but to query for closest playlist you also need the user uh listening history right and that's where we use this concept of a user vector um again that is also derived using the user listing history from the tracks and other content that they've uh that they've streamed and in this case you might be thinking that you know you what do we do what do we do over time like what about tracks that they used to listen to long time ago many many weeks ago well those tracks we would wait less because we would like you to get recommendations that are more like um more or closer to something you've listened to more recently so we would add some sort of a Time Decay to those tracks and you know give you more recent recommendations and then there might also be a concept of you know users that are having different kinds of taste so that let's say a user might be listening to metal and Indie these are completely different sort of genres and if you combine them there might be some discrepancy there so there could be a concept of separating or generating two different vectors for for a user based on that um and again we won't dig deeper into that but uh that could be done as well um so so once you have these us of vectors and playlist vectors that are generated in the same space what that means is they're both generated at the base using the track vectors you can use the user Vector to query the playlist Vector using the ano tree I I just talked about and the way you do that is just you know ano tree. get nearest the seed Vector here can be the user vector and K can be the number of playist that you're that you're trying to get the K closest playlist so you know this this part gives you the candidates for a user based on the nearest neighbor's algorithm but then what more can we do from these candidates to give you better recommendations this is where the ranking model comes in that uses a lot more data to rank these candidates and sort of surface the more um appropriate recommendations to the top um for example we would use a genre information between the users and or between the user listenting history and playlist to to surface the playlist that are more closer and this distinction especially especially appropriate when you think about Chinese pop versus Korean pop you know they're pretty close but you would like Chinese pop to come up further for a Chinese person than Korean pop or if if that's the kind of music that they listen to um and then there's demographics data for um you know realizing country and age groups and then there's playlist popularity as an example take John who's 21 lives in USA likes rock music his recommendations should compose of rock playlists that are you know popular in America and amongst 21 year olds or so these these should be the players that we believe should you know surface towards the top and that's where this ranking model comes in but then there might be the problem of you know this ranking model always ranking playlist in a certain pattern and the user getting the same playist again and again so we would add some sort of postprocessing shuffling steps that uh avoid this uh repetition over multiple days and times um and just to give you a number of where we're operating today we have 90% daily active users that have Rex and our our efforts now are to you know improve these re and get the next 10% 10% that are probably the users who are who you know recently joined Spotify so they don't have enough listening history for us to be able to get them rex or the ones that have not been active for a while um and we're not able to get them something recent so let's take a little step uh elsewhere to infrastructure and the kind of uh Technologies we use in the recommendations pipelines to uh to make it more um robust and better um we used Luigi to manage the workflow this was also something that was developed at Spotify by Eric bernardson and uh this is this takes care of you know dependency resolution job scheduling fa tolerance so all of that is taken care of us and all we have to worry about is writing jobs and just scheduling them once and then Luigi will take care of all that and our entire pipeline is written in scalding this is Twitter's framework open source framework for writing map jobs in schola and this is something that I'll take deeper into next um and then we run all our jobs on a Hado cluster composed of 1200 plus nodes it runs about 10,000 jobs a day um and then we have Cassandra as our as our primary backend storage that um that serves all these Rec and then just for playlist Rex itself we have about a dozen nodes and at the very top we have Java back and microservices that are serving these recommendations and doing the shuffling logic that I just spoke of so sculling let's let's dig into that next and and and tell you how we're using using that and what kind of problems we've had with that how we've tried to solve them and how it's benefited us scaling is comprised of a DSL that makes map produce computations look like scola's collection API and is a rapper for cascading to make it easy to Define jobs test and data sources on an htfs um scalling with respect to playlist Rex we used to use Python back in the day um before we switched to to scalding and that posed a lot of problems primarily our our inputs and outputs Were You Know Tab separated and or in Json format and you can clearly see that how that that's probably not going to scale as the complexity of of your jobs are going to um go up um you'll have to keep track of your scheme of changes you'll have to keep track of where the items are and just catching errors early is is just something that python cannot cannot give you um and then it's also hard to write test because python didn't really have a robust and strong testing framework for for um map rce jobs as um scaling does so scaling provided us with wild time a checks you do being written on top of scalding or sorry schola and um you know having that that benefit of of having type checking helped you catch errors early in terms of schema changes over time and just being able to manage those changes really easily and then we also use Park and Aver for input in input and output Neville my colleague will dig deeper into that next or in in the next uh presentation but basically Park and ab made it really easy for us to be able to um write these inputs and outputs something that we learned about part as we started to shift over to that was that par hurts performance when your when your data structures have really fat columns so in the sense that they're very nested the kind of data that you're trying to access is all the way deep and an accents for that and an example for that is this data structure it's a playlist and at the bottom you can see it's it's an array of items but let's say you wanted to you wanted to access the added by attribute of each item that's really deep in there and that's really going to cause your job to use up a lot of memory and um it could happen that job could just wait for wait for a really long time um trying to get resources enough to to do those queries um something that scholing has also helped us with is consistent data quality when you write Hado jobs you would write you would like to write them once and be sure that over time as these jobs run um your data quality stays consistent and what we've been using for that is Hadoop counters we have rappers in our in our extern in our internal um scaling Library code that help us track flows and sort of count certain records in those in those flows and be sure that every day when this pipeline runs we have the correct numbers and as an example you can see the way it's used is that you define the counter here and whichever flow you're trying to count you just increment the counter there and in this case this counter is counting the number of usernames that start with a so at the at the end end of the end of of the job you would have this counter would basically contain the number of users that uh have a username that starts with a and to to extend that what you can also do is you can add a verifying counter uh check at the bottom that verifies some sort of a minor Max condition on the counters that you've uh you've added to your job just to make sure that your job is actually doing at least acting sane and to even make that more robust we've added some sort we've added u a basically monitoring system on top of our counters are using our existing mon monitoring system with counters so that over time you can see see what sort of uh what sort of a track your counters are taking and adding uh monitoring alerts on that so that if someday your counters take a certain dip you can sort of get alerts for that and act on it fast um another thing that scaling has really helped us uh tackle this pipeline tolerance job whe is a pretty normal in Hado world and it's really annoying with big jobs especially jobs that are you know let's an let's take an example of a job that runs for seven hours and it fails after six hours well then you go and make make an make a change and you make you run the full job again it runs for six hours again it fails again in the you know between the sixth and seventh hour well that's a really really tough job and enter scalling checkpoints what this helps you do is that it it um it dumps an intermediate output of your of your job into hdfs so that the next time when the job runs the that fail job runs it'll start from that same checkpointed place as an example you know this job is using this checkpoint at this at this time this is generating generating the user vectors and the way you define the checkpoint is right here you pipe that check you uh put this pipe as a checkpoint and you use the resulting pipe um into your job that continues and what's important to note is that these two parts the checkpoint and the job that follows happen parall so you don't have to wait for this to complete and then have this this part run something that is also important to note is that checkpoint itself is a map reduce job it's more of a map than a reduce because it is just just writing um uh to disk but it has the same caveats as a as a normal map job where you know because of cluster issues it could happen that it's waiting for a while um and job run times are also something that everyone deals with in theup they're trying to you know get their jobs to run faster and make sure that um things finish faster and the common solutions that we've always used were you know including increasing the number the number of producers or add in code off optimizations but you can only do that so far at some point you have to realize that you know there's other Solutions as well and the one solution that we're using is speculative execution this is more of a setting in Hadoop where what this allows Hadoop to do is that Hadoop will track um how many uh or how long each of your task is taking on average and at some point if it sees a task is taking more than average it's going to spawn up a new task that is identical to that and from then on it's basically a race to finish whichever task finishes first that output is used and the other task is discarded or basically killed um but this can also take up unnecessary resources because you are spawning duplicate tasks and so you have to be wary of that to use this uh sparingly but um for jobs that usually take really long um we've also been dealing with a a lot with memory issues as we migrated from python to scaling back in Python world we used to use something called Sparky this is also developed at Spotify and it's open source now what Sparky is it's basically a simple constant key value storage library for read heavy systems with infrequent large bulk inserts um as you use these Sparky in indices you define them and they replicated to all mappers so that you have quick access to this sort of a uh dictionary store um to be able to look up uh things really fast and usually our sparkies were a couple gigabytes but because of restriction and in terms of uh machine memories we could really keep on using more more and more memor in our scalding jobs so what happened was as as our scalding jobs require more memory at some point we hit this memory limit that we couldn't reduce in the scaling job and the only way we could do that was get rid of the Sparky index and then we realized that if you just do a join instead well it's not that bad it takes up a little more time you write less code and it's just you know it's a it's a very very um very small trade-off for for uh some memory and some some faster code another tool that we've been using is called driven this was developed by cascading and um it's a sophisticated tool that collects Telemetry data from running scalding and cascading jobs on a cluster and presenting them in an intriguing user interface let's see what that looks like so that is the Intriguing user interface um at the top you have this uh this driven tool offers you this data visualization of of your in entire job so it tells you what kind of syn sources are there what kind of syns are there at which point there's a combine at which point there's a flat map which point there's a group by and then once uh once the job is executing you can even see stepbystep breakdown of what's happening in there what's happening at the ma map step where the data is Flowing you can find you can pinpoint bottlenecks like in the case of of these two uh nodes there was a really high read time so you can sort of really understand what's going on in your job at a at a mapper and a reducer level and this has been really helpful in profiling our job some other awesome benefits that I think are worth mentioning about scalding is that there's a pretty active Community with some big players in there and I think that that really helps us developing and scalding because we know that uh we can always rely on these developers in this community and we can contribute to it and learn more from that for example Scola Scola itself has um has a library like numai called scholar Breeze that you know is is is pretty helpful in in doing some uh strong the linear Al algebra and then something else something else also that is important to note is that data pipeline flows naturally follow the functional Paradigm so writing scholing code is basically writing functional programming or specifically writing scholar code as an example let's look at this small job which is getting uh track genres into the stream of all these uh users who you know stream tracks at this point you have a map a group both scalding functions this point you have map and group as well this left joint comes from scalding sorry these are Scala this is scalding so you can see it's basically writing scholar code with you know some scalings sprinkle in there which is really easy to pick up and if you have some background in functional programming or or scalding you're able to pick this up um pretty easily so what is the end result of using scalding so efficiently well it's productivity without sacrificing performance um writing less code you know um spending less time creating and fixing errors and working with an act active Community that's that's like a lot more than you can probably ask for from any um any framework and um since we started using sculling it's it's helped us a lot it's it's helped us be more productive um without sacrificing any performance compared to what we were at python so that's been a pretty big step for us and we've um grown a scaling repo as I think 10 15 times since um so it's been a pretty pretty big step that's been very very helpful and that is all I have for today do question and answer now any questions yes uh how often uh you talked about two vectors user vector and uh yeah so how often the first one is actually up getting updated or is it actually being recalculated each time for each user no so they're updated every time the track factors updated so everything relies on the track factors at the base which are generated every you know now and then whenever we have more data when whenever we have enough data um so so track update is actually independent of users in it's actually updated for all yes yeah so all our recommendations will be using track vectors okay say you use Java for micro services and Scala for so what's the actually ril like using Java for microservices and Scala for other I think we also use we also used to use python as a as a microservices uh language back in the day and then at some point we shift it to Java before we shift it to Scala so the happened to the shift to Java happened before the shift to um Scala happened so then we just stuck with that yes did you look confused in spark and why did you not Cho spk so we didn't look into choosing spark for playlist Rex but we are using spark for other jobs that we think it was more appropriate for in terms of players recommendations scaling was giving us pretty good performance um so at some point you know we didn't um it was it was never a requirement for us to sort of start to look into an alternative when we are getting a really good performance with this right so yes out of curiosity you said 90% of the people do have recommendations how many of them are using the feature um is there any like data on that um I can't really releas that yet but uh that's um also a quick follow so for the other 10% you said since the users are new or they might have recent like they might not have recent history have you considered using their friends maybe as a point to say like oh these their friends have been active and they are listening to this kind of music maybe we could suggest this kind of music to these people as part of the recommendation algorithm yeah I mean that's that's definitely been been uh been part of it but the users who haven't been listening then it becomes more of a problem of having them come to use Spotify and that's a whole different problem right so we do need some streams from them to be able to get that data and right cool thank you guys guys