SBTB 2014, Ed Fine: Kiji and Scalding -- the Easy Way to Get Started with Data Science at Scale
hey so today I'm gonna be talking about key Jian scalding and how I use key G as the easy way to get started with data science scale you know so I won't first of all I wanted to tell you what key G is key G's a a modular an open source framework for collecting analyzing and serving entity centric data in real time so that means that you can use your no sequel systems and build out big data applications that can serve out things in real time these could be recommender systems these could be a lot of different systems but I'm gonna focus on another on a another aspect of key G where we can use it to really start doing data science on Big Data and I wanted to thank Vitaly and Sasha who earlier yesterday and today have been talking about using scalding at LinkedIn and yeah so key G's history K G comes out of using Avro and HBase and Hadoop and now recently we've been adding Cassandra key G is combines all of these things and wraps them together in a way that you can build out big data applications so today I'm going to be talking about the motivation that I've touched on the big theme of my talk is that it's easy I'm gonna take a brief data detour on our data model and how you use key G Express to start doing data science and then how how you can start working with it locally and then bring it out to your cluster in the cloud so in terms of motivation you know sort of the prototypical example that we work on often at we be data is you know customers who bought bananas might also be interested in carrots or bread or milk and one of the interesting things about qiji is that at the heart it's a system for storing knowledge and data about your specific entities so you know in this case it might be we'd be talking about this this customer who's coming over here and what she's bought so before I used we were in a system where I would have big logs with lots of data and we'd be engineers kind of trying to process that data maybe put it into my sequel and then we'd also have data scientists who would be thinking about okay how do I build a big data product out of that and you know we data scientists would be thinking about it in terms of tools that that we use like R and Python you know which work really well locally on us on once you've down sampled your data and really can make really cool models but it's really hard to get them off of the laptop and on to a big data situation where it's deployed out in the cloud and can be served in a real-time production system so I go to the Hadoop store and I pick me up a Hadoop and that allows me to take my logs and instead of and all the stuff that I've put into mice my sequel database or any other relational database and to put it all into Hadoop but now I've got a little bit of a problem because running ops on hadoo makes me scream it's really challenging it's especially as a data scientist I don't have the expertise in that system to really be able to do that you know I'm not I'm sure that there are some folks out here who are great at that kind of DevOps and at running a dupe cluster but personally it's just too much of a challenge and moreover I've really found that the skill set that it takes to really set up and run a Hadoop cluster well are very different than that skill set that it takes to do that data analysis and data parsing and really build those data products there you know some people can be good at one some people can be good at another but it's a very rare person who's good at both so then I find myself stuck as an engineer up there coating away trying to build out my web my data product and I want to build out recommendations and so I need to get recommendations and request recommendations and I as a I or my data scientists have gone through an untrained model but you know maybe we've used something like our and that's not really deployable so what am I going to put in that question mark so it's time for some data science I go back to my I go back to the Hadoop store and this time the sales the sales rep gives me a free add-on qiji okay and he says trust me this is what you need it's easy and so and Here I am up here telling you it's easy but we've all been to talks like this where someone says this is the solution it's so simple to deploy and you come you know tomorrow it's Sunday and you think I'm gonna try it out and you start doing it and oh my gosh it's never that easy well don't believe me believe this guy antonio's was was kind enough to have written this book I've never met the guy and but thanks Antonio's if you happen to see this on the web and if you're interested in this talk and you want to try this by his book so oh yeah I've heard it's easy before and so at the heart of key G is a is a storage system for for knowing about for reasoning about your data but how do we actually deploy it and get started so if you just go here to to our our kg org site three easy steps you know really just a click and a couple of a couple of exports and sources and start installing your table and you're off and running on your local laptop so what will this set you up with it'll set you up with your hadoop system so that you'll be able to type a dupe FS LS and and start looking into your HDFS filesystem it'll allow you to start building key G tables which are which oh look and it'll allow you to start building key G tables which interface with either HBase or Cassandra you can start taking your records about your entities that might be complex objects and serialize them with Avro and even better you can start using scalding to process and think about these entities rearrange them so for data science we've we've got Hadoop analysis via key G Express and and that's Scala and scalding we've got a unified API for reading customer attributes and events and yeah data scientists and engineers can be friends by using this so let's find out a little bit more about key G Express so I'm going to take a brief tangent on the key G data on the K G data model we all know that data munging and rearranging is often the real challenge of doing data science so how do we orient our data so that that kind of munching and rearranging can be done easily well so when you start up a cagey instance you're gonna be making a table and those tables are gonna have rows in them those rows are going to be focused around an entity so the key in those rows are gonna be your entity ID and then about those rows we're gonna have data nothing new here so one of the cool things that you can where it starts getting a little bit interesting is that in your entity ID you can have a composite entity ID where part of it is some hex code that keeps it all unique and keeps all those nice database properties but you can also I add in your own entity ID that describes your data in this case it's Bob and in addition we can have column families that will separate out different types of groups different types of columns that may have different meaning about your data and in addition so even within those we can start separating them out where you know you might have in payments you might have both the card number and an address or you might have with interaction you might have both clicks and searches things like that and in addition what's even cooler is that you don't have to have just one of these so now if you have clicks you can record every click and that's all associated with a single row so that when you want to go get our information about Bob you can get all of the clicks that are associated with Bob and at the same time you can look at all the different sets of recommendations so you can see where this would be really useful because with a single database query you can get all of your data about this single entity and now you can sort of deal with this data that's effectively and end-to-end dimensions so in summary about this this data model you know key G's entity Sentra and we have an entity ID as the as the primary key it encourages using wide flat rows to encapsulate all your information and it also one of the things that I like about it for data science is that it encourages setting up your data in a way that you can easily get tidy data which is a concept that you'll often hear from statisticians so data science with key G Express kg works really well with scalding key G Express is written in Scala it has all of your scalding all of your scalding goodness which is like your flat maps your filters your groups reduces joins and it's you can really because you have all of this you can express complicated flows of these MapReduce jobs through a simple Scala based DSL so this is very much where we come back to the the kinds of scalding usefulness that our two friends from LinkedIn talked about and remember that still functional like Scala and you have all your relational algebra primitives so let me give you just one slide of code a lightning talk is too too quick to actually go into code but you know up here it looks like normal Scala code it's really pretty easy you are you know what we've got going on here is just a simple class where we've pulled some information out of a database we start doing a filter on it we start doing a flat map two on it and a group buy and then take the size so nothing especially interesting going on here in the dot dot dot say I edited that out but we could easily send it down either to back into a different column of the database or we could easily send it out to a text file CSV whatever you want so what data science benefits beyond scalding you can't mmm you can work on ML models you can build out cosign tf-idf there is there's a lot of cool data science that you can do in in order to build out different data products out of this data once you have this in this once you have it arranged this way we've got simple api's and you have Avril records for all your PO Joe needs so I told you I was going to make it easy to go to the cluster so now we've got I've got a key G on my laptop it's working by installing bento box I have my my Hadoop cluster running on my laptop but it's not a real cluster so we'll go on out to either Hortonworks or cloud era and either your own cluster if you happen to have a cluster lying around or else you can go out to AWS and sign up and both Hortonworks and cloud era have really easy tutorials of how to set up a cluster with HBase on it and now how are we going to get qiji installed because it doesn't come in the default installation well we can just go back here and SSH on to our cluster and into any edge node and once more all we have to do is just in this time instead of clicking a download button we just W get and you have to reset for variable names via export so that's about for Linux commands easy peasy and then what do we have to do next nothing we're golden we're flying through the air so at the heart so in conclusion key G contains the components to allow engineers and data scientists to store this data in an entity centric model to build and train ml models and bento box allows you to developers to try out and all of its friends both locally and then easily to take the those jobs that you wrote locally and move them up to the cluster and start running them at scale thanks for listening and I'ma Tweedy data