Guerrilla data-mining with Scala, Scoobi and R

funconf 2013, Alex Cozzi: Guerrilla data-mining with Scala, Scoobi and R

Recording: funconf 2013, Alex Cozzi: Guerrilla data-mining with Scala, Scoobi and R

welcome to the skull symposium the picture shows what a symposia is supposed to look like symposium means to drink together unfortunately intel generosity did not extend to provide large quantity of alcohol so you have to listen to me while sober or which i do apologize on the other hand you also don't see a bunch of geek dressing toga which something to be thankful for for i guess so let's move on and a little bit about me mostly because that way we'll give you an insight how what is my perspective and i'm an applied researcher i started my experience in ibm first where i'm more researcher and became progressively more apply when i moved to yahoo and then ebay now and i'm most interested in data mining machine learning and ranking and as a researcher i see the implementation part of my job not only as a hobbit but it's important part but in which i don't have to do but i want to enjoy so tend to be a bit picky about my tools and i strongly believe that you should pick the right tool for the job that when you look for google this is the two top image you get right and and it's not whether the tool is right or wrong i mean the mac are excellent computer but it's probably not the best way to to cut a a cake right so the consideration will make are coming from a specific context and i will try to make give a simple way to decide how to how to pick a tool so imagine that this is the kind of questions myself frequently do do you know what you're doing if the answer is no in technical term we call this research you just don't freaking idea what you're doing so your objective is to reach understanding right what do research is about somebody tells you oh what about do people buy pink iphones maybe why or how so you want to have a tool that is interactive exploratory incremental flexible and is a lot of ready little resources program packages ready to go without a lot of effort r is fantastic for this kind of use cases much better any kind of scalar stuff out there not because scale is a bad tool but is a different use case when you know what you're doing that's called engineering your problem change is not about finding out what you're supposed to do but making sure that it does correctly and performance level performance and then where you really need robustness scalability efficiency and the ability to use your code because you really care about it will use a lot of time will be deployed in a lot of different environment and our scala really comes into its own is a very bi-modal step so and why are well this is the definition from the r page r was designed for approach in integrated fashion data manipulation calculation and graphical display humans are visual animals to a larger stand and the ability to have everything your fingertip in a very integrated fashion extremely powerful and is very very efficient when your object is to figure out what the heck you're trying to do and then our secret weapon is cran the repository of our packages where as yesterday we have 4 724 packages in our package for people who are not familiar with it is a relatively large collection of code for a specific purpose so if you want to do support vector machine principle component analysis gradient booster trees or whatever was published yesterday on some statistical journal they're normally all there actually when a new statistician came out come out with a cool new idea data mining they probably create the art package even before the article because of the way they research in the first place so for you as a scientist is fantastic because if you want to find out does grow the boosted tree works better the random forest well it's right there and just to give you a flavor here what happens all the time you have some piece of data that you kind of know where it comes from in this case since we are in ebay i sometimes this i am statistic on everything from the camera vertical in ebay basically we have something like hundreds of categories camera lens and accessories and i have how many clicks we got what did we do what we did with them and who bought them impression and all this kind of information and that is just fantastic for walking around with them so sorry ah sorry let me make it minimize so this is this they are ripple you're used to the scala rifle some of you might see the rifle is this particular ui is called rstudio which is an ide for r and well you have all this nice tool right you can read the csv file comma separated right so you just say you're assigned to somewhere else and now a variable i can poke around and this tells me okay this is a variable that has some information there is leaf id all stuff you don't care what we wanted to play around with is i have the number of impressions how many times an item of the specific category was shown to the users however many time was bought how many time was clicked upon clicks there and i calculated uh the silver impression all this kind of cool stuff so what is really useful is remember i don't know what i'm doing right i'm just walking around the data so for me it's really critical to be able to go there and see okay what a distribution number of sales across all this category and i just look at this it tells me for example the mean is a kind of useless matrix here it's a strongly long tail distribution i should not even think about doing them in here you know i can i can ask question like for example given a category does number or click how does number of clicks correlates with impression uh click through impression list let's put on the x-axis impressions oops and clicks plus geo point up was clicks missing thank you that's introduction for you there you go so well nice seems to be a relationship with the number of spamming we show something the amount of time is click upon it looks roughly a linear relationship i can easily check for example i put a smoother on top of it and looks pretty linear if i don't like that i can method method equal linear model see very quickly i tried i i look at the data poked around they get the feeling or the ranges involved can see where the linear model fits my data well huge amount of information that i didn't know before and i didn't care whether this was scalable i didn't care whether this is a type safe i didn't care whether it is functional right that's not what i'm trying to do right now my objective is rich understanding with the minimum amount of effort be interactive accrue information very very quickly and figure out what is going on and the plotting practices that use the ggplot makes extremely nice looking plot which is important to tell to your manager or even the publication you can save on spng to send on a in an email or put on a website you can even put in a web server so that the thing run and crease you know you can very quickly hack together something that looks interesting you can even use one of these four thousand packages for example this one is called tree map that we this is all the camera category so i is a little bit longer line but there's one line or this one just too and one question you might want to ask where are click through um what is what i'm missing here oh i call it the wrong way x allows you to very quickly try something pretty complicated that is we have a three categorical tree structure there's a camera camera and accessories each organizing a taxonomy so we map the number of impressions to the area of the other square you map the color to the click-through for example you map the category structure to these blocks so the entire category as the sub category these lenses and filters contain filters lens adapter and et cetera et cetera let's see this all right so very quickly i can see do for example click through follow that taxonomy structure look at this say no what i would expect otherwise would be for example is this all very bright auto click through and going down it tends to be consistent but it seems there is no relationship between the taxonomy the click through this is important because a lot of time there is taxon you find you want to aggregate on them you want to you think you will learn something more on uh if you know the super the parent of a particular taxonomy or data whatever it is and very quickly you can dispossess you of misunderstanding what it is about many times we run a loop job that transfer a day and 95 percent of the data is all zeros right because it didn't remember a specific use case where for example on ebay a lot of a huge amount of query are empty string and you might think it's stupid but it's not because where the browser you go can go to the category tree only you forget about that and you run your job and your macro dies miserably because all the empty string brass inquiry hits the same map when the thing dies okay this was a little bit giving you the feeling of r what is the time that means like you can even do something a little more fun like um you decide that you and a real data scientist want to do great and booster trees which is for people who are acquainted with that is the method of choice to predict things so what you can do you can say well can we predict uh the number of by by looking at the number of impression plus the number of clicks plus the let's say the silver impression well now seven percent is cheating sorry plus let's say the l2 categories right and i pass my data x and let me see whether this works beans not found what is beans are they called beans let me see how they call it yes they are there for equal okay very good so this again is a you do some assumption but very quickly i can look at my model and it tells me well the number impression is the most important features the number of clicks the second most important features and the level two category doesn't matter at all and try to predict the number of sales again beyond the specific example we almost zero cost i could find out whether a state-of-the-art learning algorith can help me in this particular case and yes is an aggregate it's an aggregate it is basically the entire inventory under the camera category but it's not the raw data so yes would be nice to have every single sale as a separate record i make the fundamental assumption that everything fits in memory which is only helped you so much and so that's bring me to the second part of my talk what is it okay sometimes you need to use something else when you want to this data for example is a text file that is 20k is definitely not our raw data our data is much bigger than that and it lives on hadoop cluster so in 2010 we needed to get this data out so we can actually start marking around in r for example how do you get the data down to reasonable size on out of the loop cluster and at that time there was a what was available in hadoop was the other program in java which is a huge amount of pain otherwise you can use pig which is still around and which reduces some part of the pain to postpone it when it really matters so in some sense pig makes really easy to get started and it's absolutely impossible to debug so you can easily get your toy program starting very easily but when you start to believe oh yes can do this and you start to have real program that breaks down catastrophically and it doesn't really work very well and there is high for the sql people who wanted to just to take the script run it on hadoop it's good for relational theme but in ebay we have teradata and that's a very large database that does sql properly so in some sense if you want to run sql you might as well use teradata hadoop streaming is a way to cheat in the sense you just want the data and you want to do something on the fly and that's okay it works well enough for a relatively simple job again if you need to do something complicated it doesn't scale very well or you do your own um just to be clear what the pain was this is kind of a whipping boy by now everybody saw the map the word count in java it's really painful but the point i want to make is that it's all crap i mean this is all roller plate you take cut and paste you know this is the mapper class the only part where you do anything useful is here tokenizer yeah there's two lines that you do actually something useful all the rest is bullet plate this is the again the second part of the craft is reduced part the only power you do something useful is here right and you're not done yet because there's still another kitchen list so again um we look at this in 2010 say this is really stupid uh mapreduce is a intrinsically functional abstraction with an intrinsically non-functional language which is you get this right and we weren't happy also um so we're looking around and scala was the new new cool thing among the functional on the jvm language was color and closure closure is great but was a little more of a stretch and it doesn't interoperate as well as scala with a java type system and we decided to use call instead now and yeah this is kitten and at the time we are basically running our own and then we move on wrapping cascading with my own really crappy interface then somebody like twitter did something a lot much better job than that scalding so don't use my also we didn't release because we're embarrassed but got the job done and also scooby is a is also even nicer i think api where they are scala all the way there is no intermediate step and what this allows you oh well this is a scala fact in the middle so make sure that everybody's awake and you get this type safety right if you call that uncompile it will not run so this is the way scooby the word count in scooby looks like uh first of all i mean you read scala most of you guys and it's a lot more readable first of all mostly because it's a lot less um i mean i i don't do i need to search color to you guys are you good with that i mean i think you're probably good right so i don't need to to push to the choir but i wanted to make a point if you look at the code here right basically there i just read a bunch of lines i take the lines a flat map i split them i i map each word to a wordplus account then i group by the key value and then i just combine them all right workout tell us it is now the really interesting stuff that is here is where is kubi right it's kubernetes sense there is the import over there and there is this crazy d list here and this persists down there but it's gone and how it's gone is because when you switch to scala you start to program functional right that's the whole point of the or switch to your programming languages and the really brilliant part of programming functional is that what you type here when you take the style this is you do anyhow if i was adding 50 line on my disk i will do this pretty much the same way it's not any different than what i do distribute them on here on hadoop right is the natural style that you write the functional code in scala so the really cool thing is that since i run run it writing dysfunctionally what scooby does behind the scene it can capture all this function make a closure out of them and steal them so to say normally when you write this color code it just runs there and you get a result when you do this in scooby they they cheat because this the list actually instead of executing this thing directly it it closes on them doesn't run them and create a dag behind the scene and only get executed when you actually force it over their skyline that's pretty much the same thing but what makes the really fascinating part of functional abstraction is that you can put all the complexity in your data structure because the function can be moved and become a movable piece when you implement if implemented in java there's a for loop there's an index that goes to 1 to 100 and that imposes a strict order on the execution and imposes that i need to thread from index the other number 1 index array number x and all the array must be made at the same time because this is necessary for the semantic of this operation if you drop a functional style and somehow you obstruct over that you say i tell you the operation to do but i rely on the data structure to iterate and execute that is extremely powerful because now the data structure can change the thing under the feet without you changing anything so what happens there is that the data structure can say i'm not running this one right now i created a closure that i can serialize ship on the 160 machine on the cluster run in isolation and do the reduce through a two step downstream and get back in the same place right so yeah so i think this is really the the money slide uh what i didn't like in 2010 was that hadoopis let me put it in a bit of crappy piece of softness is extremely useful it is very easy way to waste a lot of time on a lot of machining parallel with a little effort but there's a lot of limitations very specific use case works well only for embarrassing parallel problem it doesn't work well for iterative algorithm it doesn't work well for streaming algorithm it doesn't work well for a lot of other different videos that were interested in doing so i was kind of ready to invest a lot of time and effort for me and my team to learn all the details about pig all the details about hive all the details about my produce it's nice is we need it right now but i don't see that as scaring us in all the use cases in the future we will need so in a crazy way when you see that the people in future will need because of distribution because of synchronous programming because of parallel processing will need to reach in a different place to program things like it would be and probably would be functional basically if you can bet on that you can sidestep a solar sense on the current technology by overstepping ahead since i said 2016 let's do scala let's do functional and then they will catch up so it's normally not a strategy you will propose but you can avoid to be obsolete by being much more farther in the future so you basically you bet to find yourself or everybody will finally figure it out in a couple to three years later and they will come out the cool thing is that chris doesn't like to use kobe for whatever reason i don't know what's wrong with him but the next aisle and all this code run perfectly well on scalding after you change this little thing and you change this little thing or other relatively trivial changes so in some sense the fact that you program functionally say okay forget about scooby mapreduce and all this other crap you worry about your algorithm you know you send it a structure magic that contains the data you want which transformation you want to apply to it how you pass them what happens to them and how they are supposed to transform just express this in the most natural form possible and then how actually gets implemented if i wanted to use parallel correct collection i just use this list and i say dot par and i get parallel collection so on the same machine the file fits there i can run in parallel using parallel collection which work fantastically well on a single machine much more efficient than hadoop so if my data set is small enough to fit in a large machine i can go this with parallel collection my code doesn't change here doesn't even know what is run it doesn't care right i just tell what transformations apply to it and i can get my parallel collection code or they say no this is really big we'll never fit in a single machine we need scooby and so we got distributed we might produce kubi takes out the pain to figure out the specific one produce or scooby for whatever reason is broken this week because that happens all the time and therefore we say okay we need the result tomorrow so we can reload this thing to work in scalding okay the other part is that when you have a real language as opposed to language i mean pig and hive are nice and well but they're relatively toy language you know they are toy language because you need to create a udf every time you do something serious whenever you need to pass something interesting you need to do some extra functions normally in java just to get the real job done right they just they're very helpful to structure the get this piece of data here over there and filter and group and a couple of things like that but when you're ready to do complicated stuff you need to escape to some serious full-fledged language the nice part of scala once you bind to function you already know full fledged language with all the you useful file that you need so you can take advantage of information hiding object-oriented programming all these nice things that you want to use anyhow because when your data starts to get complicated you need to use them anyhow to keep complexity under control so this is pretty much the code that we use to figure out what episode of our ebay logs and ebay logs are extremely complicated we are literally handles the different pages that do hundreds of different things and no one really knows anymore what each of them does there is some legend and some a little bit of mutually conflicting documentation about some of them and so it's a continuous catch-up to just understand what the logs actually are saying but in most cases you really don't care about which or the 65 different version of the view item page you are looking at or the search result page you're looking because we're mobile where the app we have the the uk version the e-version where the old version that we replaced last week and where there's a new one we are testing right now right in some sense most because you really don't care you want to know how many times the user saw this particular item like that what you really care about so you want to say okay if it's a search result page just give me all the item version to the user i start the complexity away you cannot do this natal language you need to have what we did here we wrote pretty complicated code actually i can show to you because this is good to give you the natural reality there what is behind that is yeah so this is the kind of code we need to figure out what these events are right so let's say what uh yeah the buy item page can be any of these numbers uh i need to check whether 16 different flags in a specific case because it might fail that way or you didn't check out or it was immediate pay right now or was turbo lister super duper whatever whatever right you don't want to know this thing no one wants to know this thing but we need to figure it out and it's all code attached to that in skyline is all there and really fantastic parkour skal is that allows me to ride that way and i can use uh what in java is just a set of event or class absolutely identical and i need to find out which tags are associated with it which flags are associated with them instead i transform in a different set of types each with their own custom parsing code and thanks to pattern matching well because a lot easier a lot easier to read a lot easier to explain and so we did all this thing for a loop and then happened something really interesting a couple of years later we uh we need to do the much quicker hadoop is about hadoop is a batch system right every day there is something happening you take all your log you copy boom you copy down into your hdfs cluster you have one one and terabyte of new stuff to go through right so you can determine the heck out of it but sometimes you need to be much quicker than that particularly uh i'm working for the machine learning machine learn ranking function we need to train new ranking on ebay and the what we the way we create our training set is whenever somebody buys something on ebay you're there looking at your hello kitty phone covers right and you really try to find a pink one so you scroll down down down down and you do searches and you refine oh hello kitty with a smile whatever it is and then you find it and you buy right now when you buy that in one out of 1000 other sessions people are monkeying are not doing anything useful and they're just clicking around not buying anything but we want to get the single session that there is a conversion event somebody bought something turn around really quickly and freeze the state of the search engine as it is right now because our search engine takes into account what you clicked on what you you know is a continuously changing system so we need to get this event turn around freeze the state of the world on disk just script search engine what was what you saw the last search before buying something so we can learn from it i really would like to put on the top the one you're gonna buy so yeah you should not even scroll down you go to ebay the first one on top is the one you want press buy you're happy we're happy we make more money everybody wins right so that's what we want to do you know to do that i need to learn what i showed to you that you didn't care about you had to scroll through so next time i tried not to do it and so to do that we of course hadoop doesn't work because it's a batch system and it takes one day before you even notice things so we looked into plug into the message bus on our data center to get the streaming mode this event coming in as i said there's a work in that environment so we use our own custom code using akka we take all the back end messages and ship it over to an actor that sits there and and just looks around all the users what they're doing and when the event of conversion happens it turns around grab the last search and send over to another actor that is supposed to scrape the search engine it's not really simple conceptually it's just a couple of messages moving around but the really cool things we are able to take all these parsing code this several hundreds of classes very complicated parsing called the ransom hadoop absolutely unchanged and since we're clever not to make it depend on any hadoop library and using the real-time system now great right in a single fails whoop we have a very complicated piece of code that gets reused in real-time application gizon hadoop they're always in sync everybody's happy and when we add the you know mobile event to the hadoop side magically also works in the real-time side that's where you gain advantage to have a really serious programming language because you can invest in that environment with relatively good assurance that will be it will pay off so that was really nice and so let's suppose you want to get on the good stuff you want to have your people let you use ja scala at work that would work for us i mean if they changed this did this kind of culture change about four years ago they suddenly decided they needed to be a technological company no sounds strange but they decided to do that and there was real culture say let's put the engineer and the scientists in charge and let's see what they're doing so sometimes we get a little bit get away with much more like we say you know we want to use this crazy stuff now we don't support that we don't care we just use it anyhow that what they did the people write in the search engine they say we don't want to use clearcase anymore we hate it we're going to use git i just say why not support it say we don't care we just eat anyhow so two months later he says we support it the the people want to say we running solaris on the back end people we don't like solar we want to run ubuntu on the back end she said we don't support ubuntu so we don't care we're just going to do anyhow and you know it takes a little bit of sticking your neck out but sometimes you can get away with it but let's suppose you want to start in a slightly less disruptive way and what worked well for us was uh start with something important and new but not mission critical you want to give you the chance to fail without a company going down right that's kind of basic level smart we started with a small team of two three people that was really critical because we're small we can move quickly you can change decision quickly people relatively quickly get to the same level expertise which makes nice and you learn things more or less at the same time try to agree on tools relatively early if your friends start to use intellij and you use eclipse is not fun because you spend half of the time bitching about the dot project versus whatever else is on it's not important which one it is but pick one if you're a small team right and three people should be able to agree on your tool at least we pick maven it works really well for us sbt is nice but uh maven was easy to integrate in the in what else we had and it was good enough and honestly i'm not so hot about build system anyhow as long as they get the job done i don't care so they work well git and you pick scala is cool it's new try to pick in a place where it actually makes sense you don't want to write rewrite your legacy application in scholars it's no real point even if you succeed what is the interesting part that you're learning there right another big risk you try not to learn a new thing in a new environment but sometimes uh one of the big plus points of functional approaches make a lot easier specific difficult part of implementation asynchronous distributed parallel those parts in many cases is much more difficult to get them right in java like in our little case in the in the real time system you could do it with thread and logs but would that be much more difficult to get it right in java to learn scala and akka probably because once you're doing was a stupid program it's two pages so yes you can you know you can relatively quickly find out where it works and you can actually show that hey two pages will get it done is dancing is up since one year never goes down as opposed to you are still chasing your lock and your null pointer exception so i think that's a really good point for you pick something new and high risk design risk anyhow so you might throw a little bit more on top of that the beginning don't obsess too much about being pure i mean there is good reason for being pure as home but don't overdo it we all started by writing scale java and scala and all these things it's fine it's a learning experience just be aware of what is there is we're all learning to use our tool try to keep interp interoperation with java simple it's true that you can call scala code from java and it's true that you can call java code from scala it doesn't mean necessary you need to do all the time um for the all our useable code for parsing our logging i decided i will not support java at all first first of all because it's good for them to learn to move to scholar that was the whole point that you could toy i want them to move anyhow so this and not going to make it easier for them to stick to java but also because i don't want to write to the minimum common denominator there is good reason why we switch to functional style the fact that for example i transformed this sequence of events in different classes is is convenient because i pattern matching if i need to read this in skylife if instance of blah blah then cast else if instant sauce blah blah blah and cast is ugly so your api does change if you're thinking that most of the clients are scala or java also whenever you need to interoperate that's normally the most complicated part of the language to know how scalable for java needs to know so much about intel the compiler you really don't want to know is not a job you should be when you're starting out so keep it simple you say in our case scala code called java code and that's it scala calls the scala code scala code will call scala code and if you can try to learn one thing at a time it's good to people get really enthusiastic and you can do really cool stuff but you know the complexity grow very quickly and it's good to get a little bit find your leg first before you start to run around um one really cool thing that i tried to do last yesterday but it didn't work so well unfortunately was to you can take a actor you know aka two o as this cluster mode basically you have this actor that can find each other you have a seed node and you can create your own cluster and i try to do this let me let me show you because it's cool to have it around let me see yeah so where does hadoop cluster run sitting there and i was deciding let's steal the hadoop cluster so what you can do is add this idea you on adobe watch you have a machine that you connect to and that distribute your job to 2000 mappers now what i did is uh what if you stick an actor in each mapper and they all connect back to the command line mode and you create your and you know you take over the cluster right yeah and you can actually do it the sun says i was able to run it locally you can see let me see where i can see where it is yeah yeah so this is just i just took over the activator label and here in uh in a mapper in scuba you can create a new function by calling three implemented three methods the setup basically when the mapper starts get called the setup method then there is the process method when you go over the data that is coming into the mapper and then there is the teardown method so you can stick an actor in the in the setup here this map firing up on some of your nodes and that could connect back to the main to the seed node and create a cluster on top of that blue cluster and it works beautifully in local mode but i find that they never the people in active are too smart they blocked all the ports so i cannot connect back to my thing but if i find an open part i will actually make it run so the point here apart from the apart from being cool is um the nice part of functions that you put all the cleverness in the language as opposed to put the cleverness in a specific framework or the specific api you're using and that makes much easier to use two of them at the same time normally if i use a pig i cannot have communicating process on top of peak it just doesn't work your the framework controls the execution of your code that you just asked what type of zippers you're not in charge of it right and but here in scala instead you have much more since most of these complicated stuff get pushed into apis i can use this could be apis that enables me to create this map and so on my produce jobs and i can also link into the arc api that allows me to create actors and so on and use them at the same time because i'm in control right the is not the framework that imposed its will on me and i can get along as long as i fit in their particular view of the world but i can actually take over and do something really cool so there was another thing that uh yeah i think i already talked about this yeah this is functional programming okay how do you convince your hadoop ops team to let you use kala this was 2010 yeah well that's the way we did it chris knows it but does your hadoop job in java and this is the job in scala so just don't tell them right they won't they will never fight out right they will never figure it out they look they look in the class but they say what is this colorful oh it's just a library we picked up it's nice don't worry about it right don't show them your source code there's no clue anyhow so right we ran for a year without anyone knew that we were learning scala this production in some real definition of production but these are code that actually produce data that ends up in production this production job right as long as it works no one cares so don't tell them why do you tell things right just just use it remember it's much it's much more difficult to object to success right one thing is say you know there's this weird stuff that no one really else is using but we think we're cool kids we want to try out pretty pretty please can we use it in production i say no of course right but you say you know we implemented this thing we took off effort this running system months is up and we love it we want to have more of this you know you can start a discussion a thing we really learn is be careful when you find yourself writing frameworks at least me as a scientist if i'm writing framework means i'm not doing my job resist the urge as much as you can you know try to get what is out there there a lot of people reacting framework for you get online the learning curve is steep more than you expect i mean we all picked up a language overnight it's just syntax most of the time unless you're well-versed in functional programming languages is a different ballpark it's like when we went from imperative to object oriented it's not trivial to get really comfortable with it documentation and tooling are important particularly when you get people who have no clue because they are beginners and you need to set them up our biggest pain point is uh first of all you need to get a you need to get a working eclipse installation because that so you need a lot of memory then you need to get uh all the libraries we need so you need to have a maven network so you can download our libraries then you need to have something on which hadoop which means all your windows machines are useless because just does run on windows i don't want to run on the cluster as much as you can you want to run locally and and then you need the whole thing to work magically right that's a really huge amount of setup and you cannot give them a vm is a development environment vm wasted half of your memory on crap you will never get enough performance at least not now on the vm unless you're a humongous machine which you wasting it pair programming i find it works really well to ramp up people and giving talks writing documentation embrace a bit of elitism you know you don't need to get everybody from java to go to scala actually i strongly recommend against converting stuff you want the new cool kiss and the new stuff to be new but you know there's no point in rewriting what is work perfectly well in java in scala what is not very interesting you know what in general i want to write i'm again i'm programming because it's fun so i want to work that was not even possible to implement 10 years ago either because it was too complicated or because the complexity kills you because involves a synchronous distributed process that you can just not wrap your head around normally so try to embrace the direction and and be selected to where you deploy this technology and then try to become the standard the wiki is your friend so this is the internal page of ebay it's called scarlet ebay i wrote it and the way you become the standard is uh the little edit button so you hold a week right so you press the edit button say scala is the standard at ebay for all hadoop job this is pretty much yeah right and people believe you i mean i wrote this paper i wrote this page and i wrote this nice seven pages tutorial how to get started with the standard platform on hadoop that is color and all the documentation and after a couple of months people from the we acquire a company called hunch in new york they join they're over there in new york and then the guys from hunch comes over to visit the main headquarters say we really love the scala platform it's awesome everybody's using a touch you say cool it worked right they're new there's no clue so they go on the wiki they read what is the standard i tell them what is the standard everybody's on board right works don't tell them right and so conclusion since we are almost there scala is awesome but it's not good for everything be aware what you're trying to do do you know what you're doing or you don't do what you're doing if you don't do what you're doing maybe scala is not the best though first figure out what you're doing r is awesome for poking stuff very quickly try all the possible kind of crazy new ideas and statistics also as opposed to computer scientists statisticians tend to be sensitive to get actual correct results from their process and r is implemented by statistician so normally you get result that you can stand behind so when you do run a linear regression from r normally the result makes sense as opposed to be weird learning scaling function is hard but is but is great but it really pays off uh because once you learn to think in that way all of a sudden a lot of things become easy so because you know you invest a lot of effort in to get your your mind wrapped around the the scala collection classes and the way you do map and filter and all this kind of cool but cool stuff but then you say hey we can do in a disability process with aka just new library pick it up or chris was saying he needs to go and run sql code on teradata i know it's bad but sometimes you have to do it and you say hey there is link i can write it as a scala for loop and it's type safe is awesome right so it really pays off because it's like is a really a new enabling technology in the same way when we move from procedural to object oriented all of a sudden a whole new set of things became easy and is really cool and scala is also where all the action is in these new programming models so spark spark is improving it's going to be much more interesting so you want to go iterative distributed process is a new thing to try out uh stream processing is also there is initiative implementing some modesta so that's where the action is and if you want to be a little bit on the leading edge is a good place to be and finally at ebay with more data than we know what to do with and we need help to figure out what it means so we are hiring and if you are interested to try it out we are definitely looking forward to your application so and i think i will conclude it here and open for a question right um it's kind of difficult we didn't really benchmark against each other honestly i i was using a cascading before what i noticed cascading is a bit more aggressive in running a parallel job in parallel meaning when you have two data sources that you will eventually join uh cascading is pretty good as coding on top of that to run all them at the same time scooby's a little bit less aggressive but scooby is much more aggressive in compacting all the process on the same mapper i had a cascading process that was three in my produce step and when i translated to scooby went in one and i thought was broken but because i figured out they could uh um compress all that and optimize them all out so it can be a trade-off yeah they're all pretty good um yeah and you'll be happy yeah um my objection to scalding is not that there is any particular bad thing in the infrastructure itself but uh is mostly also it makes training a bit awkward because uh you get them on board you say oh we're using this weird stuff called scale at ebay so you need to learn it then and by the way you're running hadoop job so please learn about hadoop and then learn about cascading because the underlying technology or scalding but all the documentation in java so while you're learning scala translate mentally all the documentation in cascade into ja into scala while you're learning this so you say so when you combine um hadoop with r do you have any interesting tips of uh how you get data from the building or sampling uh aggregation yeah we will wish would be more sophisticated than that but we end up doing text files a lot chris implemented but i didn't release it yet a way to read our co avro format in r unfortunately there's a really important mismatch because rs fortran and c code and scala is all java code so it's kind of difficult to interact with there's a couple of interoperable data formats but they don't map well to hadoop again because they're not splittable and so on so you can do it at databases with backloads or you go with text files in general staffing r should be relatively small anyhow because otherwise you kill the machine anyhow so text files do not ideal but you can make it work you need to sample yes sorry i don't do regression scala i mean if you think about i mean there is very few reasons to make regression an entire data set it might be cute but when you do regression of one million points you just increase the error or the square root of the number of points right that's what you're doing so if you take the sample of 10 000 points you do regression you get pretty much the same result there are a few cases in which you really need to run machine learning scale there is few like for example you want to find all the you know like tech statistics or bag of words model or principal component analysis that's where it makes sense to reach scale but most machine learning actually uh i mean medical research sample size and medical research is in the hundreds because you're killing people right if they can figure out with 100 people when somebody's life is at stake we should be able to figure out with 100 000 samples for most of our things that are just people clicking around right so just again if you try to figure out what is what the heck is going on you don't need to train 20 millions more in parallel typically anybody else yes yeah that's a tricky thing we just buy a bigger machine no honestly i mean right uh it's the simplest thing is most cost effect we bought 192 gigabytes of ram this is the simplest thing what that's what you want to do right you invest two years of optimizing the crap out of it now yeah again it's honestly it's cost-effective to just buy larger machine r makes very strong assumptions everything is in memory it doesn't you can still find static falls in between but by the time normally you should figure out why you're doing that and therefore you can invest in doing a performant and scalable thing and r is not necessarily the greatest implementation language when you want to have a robust and scalable system it's just okay thank you very much