SBTB 2014, Sasha Ovsanskin: How LinkedIn Uses Scalding for Data Driven Product Development
hi my name is sasho sanki and i'm i'm with linkedin you guys have a good launch yeah okay my mine was good thank you as well so i'm uh i'm going to talk about scouting which is uh a great thing we'll talk about oh wait how do i yep like that um and uh uh okay here we go i wanna i wanna talk about data-driven product development and i also wanna talk about the scouting which is a scala dsl for hadoop development and i i want to show you how this these two things come together in a very nice way and why you should uh check it out so um let's uh let's have a little show hands of uh how many of you do work on a service that touches users okay great so um so that's your service and this is your user and hopefully you provide some value to you to the users and that's that's why they like your service uh next question is uh what is what is it you get from this service so uh first thing you get from this uh for from the service is a data so uh if you are uh if you have many users working with your service uh you have a lot of data um uh getting out of that and there's a lot of things you can do with it so you can do some processing of this data and you can do you know the get some insights about how your search service is working for the user and hopefully by processing this data you can turn your service into an amazing service that uh gives a lot of data to the user and then at the end obviously you will profit or improve your service so that's the goal uh we were working uh towards at least in linkedin and this is how uh it works in in in our our situation so we have a so-called online world uh which is a services that users are interacting with uh um and uh we have a um so-called offline world which is uh runs mainly hadoop but but other systems as well and there's a lot of data that is coming uh from from the uh online service into into the offline uh this data comes from uh let me see if i can no i can't okay so yeah there's a lot of data is coming uh via the etl processes uh from the databases and uh there's data that is coming um from tracking and logging of the services and all of this data uh ends up in in the hadoop file system hdfs uh uh with with this wealth of data there's a lot of things we can can do and we can run different operations that perform analytics on the data and then people will look at at this results of this analytics or look at these reports and get some insights about how can we improve things and then based on that we would create some data products some some some algorithmic improvements that we will what we think will improve the your user experience and uh these improvements will travel back to the online world end up in in some data that that could be serverable to the users online uh also there's a uh there's part and this part that i'm i'm working on which is basically up using this data to uh get some some interesting uh insights to you back to users themselves and send them via messaging via email or the push notifications so that's uh that's how we use uh that's what what we mean by data driven product development at linkedin uh obviously we uh with with the amount of data that that we have coming from online we were facing these scaling problems and we need to we need to figure out how do we scale this uh um how how we efficiently work with the data at this scale uh also um the complexity of the data is uh is a pretty pretty significant and growing because with the amount of uh products we have there's there's a big variance of the data that is that's stored offline and we need you need to process with and it is also um also problem to deal we have to deal with so uh i just wanted to know that that uh in the time i have i i can't talk too much about this so if you are if you're interested in modder i'm the the couple of links uh that talk about uh linkedin big data ecosystem by jake krebs and another one uh on grid operations by ellen wheaton hour so check them out if you're interested so these are the problems scaling and complexity that that we're dealing with and uh we were looking for good ways to to solve them so that's the next question is what what does half all to do with the with scala where our skull conference right so next i want to talk about scouting uh scouting uh is a scale-based dsl for hadoop map reduce jobs and as far as i understand we had we had a great talk by mate in the morning about spark as i understand scouting uh was inspired by the um kind of straightforwardness and simplicity of the spark interface and uh as creators of uh of scouting the group at twitter they tried to to reproduce the same kind of intuitive interface for hadoop map reduced jobs and they succeeded with that it's built on a java framework for hadoop development called cascading uh which is uh has been around for a while and and it's it's a really um good platform to build uh this kind of kind of thing so the the api um let's have a nice show of hands how many of you had a chance to work with hadoop uh quite a few people and uh there are different ways you can work with hadoop the somebody has matched the big as one of the tools and you can write jobs in java and sometimes it gets very technical and gets done on the physical level somewhat reminiscent of what databases look like before sql and things like that so the the api that uh skeleton uses is uh uh is a much higher level so this is a little example of that you don't have to understand all of it but uh just looking at this you you can see that how you uh read the input file that is passed as an input parameter then you do flat map operation on it breaking it into words and then you uh group by words and then you count these words and write right the output so that's that's the level of the interface uh we're working with uh and as such it's a it's it's a resisting and powerful and uh this is this very very important for that so how does it help us with with the scaling so because of the huge volume data we need to run this data in distributed manner there's no other way around it so in order to do that we we can we can you know go go a low level and specify for every particular operation how many computers would it run on and what how they will talk together um there are even apis for that but that will take a lot of time to do so uh it it'll be nice to have it to have a high level description of these algorithms and uh obviously functional programming uh is is a one nice way of doing it so that's uh that's one of the ways that how um how this house scouting and scala can help with that so uh here i'm i'm showing an example can anybody uh do us can are you able to see the code in any way or well you don't have to be uh to see the uh to the very detail we won't go over over in very detail but what you can see here is that you uh you have operations like group by join then values which is uh some sort of projection then another group by uh then count uh count the account the uh the results of the grouping and then uh map and then uh again group and and so on so this is actually part of the or really a call that is running in production so uh it's not it's not an artificial example i just took it from there and uh as you can see there uh the uh you you think at high level and behind underneath this you there are actually uh three hadoop jobs that kicked off in the process of this thing and uh they're executed and it's all happens uh happens without yeah or you you're doing it explicitly so that's uh that's the advantage of having it at high level uh another problem is complexity so there are a few things we can do to deal with that we there's a lot of data there's a little updated data file so we we can think of some reasonable way to organize this data and in some some sort of reasonable file system and that's that's what we do have certain practices to to organize that i won't talk about in more detail and uh for every every file we need to know what what data is it is within the file and what's the format what is what's the schema so uh to for that we use avro which allows you to specify schema for the file and it's actually embedded with the file itself so that means that if you if you're just looking at your hdfs with all the million files in there you found something interesting that you can you think you can work with uh next thing you do just extract schema if you uh from from that file and you are able to to work with this thing in type save manner and then where the way they talk about scalar ges how uh how you can create create a code on on the fly and look at all this documentation and look at the methods that are available at every given point this is the same kind of uh operation that you have here as soon as you have um you have scheme of the data and you compile this schema into into java classes and this java classes can be accessed uh directly uh by by the scala code you just uh you just have have the same level of uh ease of working with that and you can you can modelize you can share share the code between different different pieces of code which is which is very important when you have a high complexity so scouting is great in this race i respect as well i'll skip that because i'm almost about out of time so the killer argument is that uh what's the best api for mapreduce it's the one that has uh map and reduce primitives inside it which is basically this is a valid scalar code which does a sum of squares of numbers uh it's not it's kind of a joke but uh still it's nice to to have a map reduce within within the base basic api so right now uh the way we use scouting a little bit more than a year ago we started doing doing jobs we have thousands of production lines of code written and running in production producing products that some some of you are interacting with so there are dozens of flows we have internal scouting usage group and we are trying to be active outside of that um many many engineers in our company are you know became more interested in this and uh from the learning point of view um it's it's a a known problem that that it's kind of take a little bit of effort to learn scala it's a very very rich language with uh with a lot of features and stuff but uh specifically uh if you if you work with hadoop development and you use scouting uh you all you need to know is a smaller subset of scale which is uh which makes learning a little bit uh smoother for for for people and we have like new college graduates uh getting up to speed quite quickly with this thing so basically in summary uh for data driven product development uh scouting is uh is a shut up um it's a it's a it's a great tool and you should check it out uh plug for linkedin uh waihari and thank you very much