Scale By The Bay 2018: Bram Leenders, Julien Tournay, Scio data processing nirvana at Spotify
you so welcome everyone we wanted to give a talk about shield the data processing framework that Spotify uses internally for all batch and streaming pipelines that we have and the reason for that is a bit that two years ago one of our colleagues Neville introduced Geo here on this conference and ever since we've been using it and it's now the default at Spotify so with two years of experience and a little bit of pain in our minds now we want to share like the stuff we've been doing with it and the work we've done on shield so for a quick introduction my name is Brom I've been an infrastructure engineer at Spotify for a bit over two years now I've been asked a lot developer for about a year since we started writing pipelines in Scala and this is my first time speaking on a conference so thank you for being here with me right so two years ago we would have started with like a raise of hand like who knows what if I hear I guess by now everyone kind of knows Spotify but in case you don't it's a music streaming company just to give you an idea of the size that we work that we operate on we have over a hundred ninety million monthly active users almost 90 million of those are premium subscribers so they pay us every month we have over 40 million songs that you can listen to and over three billion user generated playlists and to give an idea of what that results in in terms of data every day we ingest over 200 terabytes of client events an event here is for example every time you click on a button that's part of an a/b test or every time you finish playing a song the device you listen on will send an event to Spotify and that's what we tend to refer to as events so yeah that's over 200 terabytes a day we process that with several thousand jobs scheduled every hour Spotify primarily uses batch processing there's a little bit of streaming as well but the core is really a batch unfortunately we don't really have public numbers anymore on how many computers we have since we switched to using cloud infrastructure but as of about a year ago we deprecated or we started deprecating or Hadoop cluster and that had 65,000 course which were maxed out so we kinda had to switch to the clouds alright so talk about here which is when it comes to the singers 45 in a few words here is a skylight VI useful date sample synched works for both batch and streaming pipelines and it's open source on the right side of the slide you have a simple example of what you can do so that's in fairness Watkins so you really text by somewhere you can flat map on it do stuff with its accounting value you save somewhere and then you can context so it works very much like the Scala collection API for example or what you could do we spoke you can find source code on github you're very welcome so you can read the source code contribute code and peels and then make change you want to yeah that's it and it's possible free to use so a bit of context about geo so 25 we have roughly 300 people using it and goes from data engineers to backing engineers and also data scientist so that's very different kind of people like I would say we are I think roughly 3,000 jobs in product range something of that and yeah what's passing string also ops 45 if you be be completing it I think don't Jones is using it was nine CAC if anyone like James yeah yeah also be convenience expert if I actually using CEO to run jobs and polishin okay so that comes with a few constraint for us so I think the team that is maintainable for 25 so I wanted to last years we've done a few things using Spanish buying production scale what are the names like not everyone can know most Scala or spin writings colorful ten years and especially when you're working with data engineers or and data scientists engineers usually the no skeleton your functional programming or they know Eclipse basics of it oh that's but data is coming is not their primary focus they have to do it but basically a job is more about math statistics backup stuff not recoding so the north experts and there to be experienced that means to do things for us the first one is documentation is important because we have a lot of users but my squad is actually a small team we only four people working full-time here so that means that when someone is going to use Shiro is 25 they should be able to do anything they have to do by themself they should have needed our belt the flat muskrat doesn't have to take the hands and guide them to use using shield they should just bring documentation and I figure out everything themselves the other thing we want is type safety and while looking sky is pretty doable the reason we want type safety is because when you're writing a pipeline especially a batch job running it is going to take hours so you don't just write your code then deploy Giants and see what's it's doing that would be stupid so he wants you to take every defect every problem with your column very early so one of the choices remain in Sheol is to have enough type safety that's most of the common mistakes will actually be covered by the Starfighter you don't have to write tests or undergoes the compilers we need to work for you no of course is better if you write this doesn't appear for that well since we've learned - is that one of the issues with your going to have your work your right distributed program is civilization so when you want stone here your instances to library that can be very complex and on the JVM your line issues with that so that's one thing we want to make easier with shield so we are now going to talk about an example of the pipeline job that we are reason if you months ago as what if I like that so here's an example of how we shield and the benefits we get from this right so like I said about a year ago we started like heavily deprecating our Hadoop cluster and before we started doing that the majority of our jobs were written in crunch Java framework and also a little bit of scolding every like for the functional programmer enthusiasts so yeah that was predominantly the framework we used to we used to work with but we and this was also the framework that the anonymization pipelines of our event delivery were written in so as part of the Spotify privacy policy the internal one at least we try to encrypt personal data whenever we store it and we encrypt it with encryption keys that are specific to each user so no two users should have the same keys and that means that as part of our event delivery we take the set of all incoming events and we have to join that with a set of keys for each user so naive implementation is on the left it's kind of simple just of events you join with keys based on user ID then you're encrypting you save it but of course it's never really this simple because the join is quite expensive to do so the challenge that we're kind of facing is that we're dealing with events we have about 400 event types each event type run sets its own pipeline and the biggest events can be several several tens of terabytes per hour so that means that we have like on the one hand we have billions of events worth like several tens of terabytes and we're trying to join that with hundreds of millions of keys that's both very i/o intensive and very CPU intensive so we spent a lot of money just shuffling data around the naive implementation is kind of what we used to having crunch of course incarnations a bit more for both Java and you work with factories supplier some things I need a white screen to fit everything the coat was a bit longer but in essence it just did a fool shuffle join of the events and the key data set so we rewrote it and she oh we spent about three weeks going from the original idea like hey we can make this better to fully running this in production and also turning off all the old systems and we noticed that we now have much more complex pipeline logic at the same time the code is a lot simpler to read it's fewer lines there's type safety so it's a lot it's a lot safer to change even if someone doesn't know the full context and we can now use a joint strategy that's way more advanced than a shuffle joint so one of the things we try to do is we we pick a joint strategy based on the size of the incoming events so if you have a large event like a be testing or use a different strategy than if it's a small event like video playback error which might have only dozens of events per hour another thing we try to do is we predict which users will be active in a given hour and then those users we will load on every machine that runs the job and we load it in an off heap hash map using a product called Sparky which is also open source also by Spotify and this allows us to super efficiently join users that we think will be active and the remaining users we still do a full shuffle join for but it's only very few users a few percent and this allows us to do exactly the same logic but it's fifty percent cheaper than the original code right so the other things talk about what we're doing for the next version of here which we're going to reuse new few weeks so it's gonna be she owes you know seven we are currently running shield there were six operations went by so it was coming so we were actually breaking stuff for the first time I guess ever in India so if you're migrating functio there are six two zero seven you can compile immediately so what we do is we reward equity clients that's that's not you guys but it makes it much nicer and then bit more consistent with the rest of shield we reward only are using shield that's a much bigger change so that what we wanted to do is to make are using Shearwater Eric so that not only you could write I use herself and next time what what she or is about to do easily but also confetti recipient had to deal with everything testing in shield so in all the rest of share what you were using others you had to deal with with test if you right now you you need to define what's supposed to do when when when you test Nikki ensures you're selling that that comes automatically you don't have to deal with it they'll seen is iOS are no generic and there's new with a living s collection and she'll context that allows you to just read and write and you only need to know those two methods and then you personally Rita interest-free androids whatever that is you don't need to learn all the different methods at our next image here the other biggest change we are making is waitressing this concept of new colors that are we going to talk about so corners are what you using beam when you need to move data between workers which happens a lot so basically any time you want to you're processing some things out I need to move that former worker to else and another worker you need to turn that data into an array of bytes and that's what colors do they just take your class here instances turns out to bikes you send them all the networks then the other worker gets to need advice and can dis realize that it's your class that's basically the way that this is working in shield zero six is that it's using a librarian chat like on Friday so a few words about prior witness probably already yeah okay few people so preggers like this magic Java library that uses reflection to guess how to steal a large data and just know it's data at runtime the reason why we're using that these videos that's what everyone is using so if you speak or spa or anything Davis crater and the other things like cry do is is easy ish to use it's so what magical it's not the implementation is what complex but as a user you just say hey I need to see you like this do it for me you don't have to be very explicit about what to do after that sighs that makes CEO is it used but it's not a perfect solution we wanted something more time safe because of the Steve Santa Steve is one of our concern and doing things that one side is not exactly typed in so it's a belief itself moving from the dynamic cry of galaxy to something that's started and known by the compiler is to get a bit unpredictability so basically whenever you're going to see lies or just realized something it's better if you know what's the the the program is going to do ahead of citing he wants to know what to buy time how it's going to be sterilized you don't free what something magical you want something that's easy to test so if you can just say aim keeping your color for that type and then write tests for that that's great that's that simple I mean we also think that's fast obviously we are processing terabytes of data if we're slow it's expensive and we want something that stays automatic despite the fact that it's not happening here on time anymore so that's pretty important points we don't want to bother our users especially data centers which seems like civilization which is like a very multi concern that the framework should be doing that for you shouldn't be thinking about that all the time and yet to chew the efficient so that's that's very big challenge so we decided to do something that that doesn't automatically at compile time using a library called Magnolia so whenever you use she gives you a seven and you need a color for it sorry most of the time is going to generate or derive it for you study but maybe it don't do it for reasons and in that case is going to use cryo again which should be fine hopefully so we get like a much better option than than before instead of using probably all very dynamically all the time 90 95 percent of the time we're going to have something very static and nice and then what it doesn't work well too bad basically worse right so first how it works in in she'll so if you use a map function of the collection in this collection you used to have the signature on top so she'll need each other runtime information about your class so it had a class time so that way when you were trying to move something of type you from the worker to the worker cryo could get to type information at runtime get a color for you so your lines everything and that works so know what we did is we say very specific instead of saying you need a class that you need formation by the time we say you need a cover you need to know how you see right that thing so we replace that that typed on string from a plastic to a color so that when you map over something of time cue you can have put up you that's it no way no yeah so this is little demos of what she is doing so say you have a single class food and string and in engineers if you query the compiler when I write color of food what I'm looking for is an implicit color of food so it is going to be looking into the interests it's go so essentially what it's doing it's like asking the compiler hey can you provide color view I think about a that gets it okay and can because it derived it automatically so the compiler actually has the logic internally to Jerry to color food because knows what you sterilize a string it knows how to store is an integer so obviously it knows how to slice a product of the string and integer right that's easy and you didn't have to do anything now on for a slightly more complex example what happens if in self an integer I have something called a local it's which is a telecast I cannot write that obviously so in this class Spanky foo and inside it has just like slightly weird ideas okay so now I ask the compiler can you give me a colorful funky food and the copilot said well yeah I can't but I don't mean what to do is we that's our giving me a local I mean I'm not sure what this is so I'm gonna use cryo for that time now if you are using something like normal increased since color the compiler would be thinking and say to you I don't know how to sterilize thank you food you wouldn't give you information about the source of the issue it will tell you well it's local I don't know what to do with that in that case not only are we providing a full back so that that doesn't fail we also telling you precisely why it didn't work like why why is it weird so that you can actually anything I've got a following on yourself and you know that everything is nice and in case where that's which fade up front sign you wouldn't know that oh I have to look at this because that was weird alright so we wrote a few benchmark to make sure that every single at least as fast as we you subscribe so we have this likable simple kissed ass next it and we we wrote small micro benchmark that just serialize and destroys instances so on the Left we have Chavez realization so you just X none in Toronto so you like simple every guess class is serializable test that it's very slow and very bad and also the output is actually very good that's pretty bad that's a very good option that's why everyone's you can cry you and know that then we use cryo with registered classes so that's typically what you would do if you were using your six pin spot that's what you're doing so it's not faster the outputs is smaller when you're still rising so it's it's kind of okay and what if we describe but we explicitly matter if your products and I are cool at all times so that's a lot of work for a super cast like that that would be probably 30 40 lines of good so it's typically something you don't want to do so it's a bit faster becoming is significant faster and the other slightly smaller so yeah that's better now what if you use being so Bemis is concept of coda which is not cryo and beanies even faster and again you have to write everything manually so that's all the work oh great and the output is even smaller and now what we these is that a compile time we derive the theme color for you so basically the center it is it's very fast but you don't have to write that for itself the compiler has to work for you and as you can see it's almost as fast as writing the beam cover yourself and it's even faster than custom franco's and the earth which is very small that's great so now the question is all right that's micro benchmark classrooms when you actually run that on a bye-bye so we did a few tests on the pipeline and that's like a will be a placement like someone that kids no parents wrote that my time and as you can see we use much less work of myth less workers but we also go from 60 minutes run apartheid was given that sense to roughly 45 minutes so it's about 25 percent faster just by upgrading sheet Co to 0-7 and I think that's what all right so that's it if this sounded interesting to you be sure to check out the github repo also you can reach us on get there which you can also find the link on the github repo or you can send Julian messages on Twitter I don't really use Twitter so you can use the same handle but at Spotify and at Spotify dot-com and you can email me on that and if you won't work with us on she'll check out Spotify jobs.com Thanks [Applause]