funconf 2013, Vitaly Gordon: Data Science Patterns
Recording: funconf 2013, Vitaly Gordon: Data Science Patterns
guys um let's start yes I am yes um okay I'm Vali this is Chris we work uh I work at LinkedIn Chris works at eBay for those of you who don't know my company does a social network thingy you can buy stuff on his company site and but before we start I just kind of I want to know um the talk will be about kind of machine learning map ruce stuff how many of you have written map ruce jobs keep your hand if you use like um some platform like Hive pigs something keep your hand if you use one of the cascading Parts like scalding CasCal log okay and SC and just scalding how many people have experience okay awesome he um so we'll start no slides just kind of live coding cross your fingers hopefully it will go well um so I before we start just kind thing that will make my life easier and actually it is things that we do use in our day to-day which is not part um actually it is now part of scaling but just not officially released yet so one uh thing you can see here is this word local so I'll see scalding and cascading have local modes so actually in order to test our code we don't have to ship it to our hop cluster which is great and especially avable for talk so we won't have to wait a lot of time to see the results um the second thing that you can see here is these case classes that I'm going to use I just made up two case classes and you can see that they extend a record so both both LinkedIn and eBay and other companies use AAL for their data serialization so AAL for those of you who don't know it's a serialization format like protuff Rift and others um and it's extremely nice to use instead of this long topples because we can actually use case class and there is a plugin that was developed in Berkeley that um so AR has a schema and the schema can be kind of pain to describe so all we need to do is just Define our case class and say that it extends a record and it will generate the schema for us and it's really um nice to use this way the reason why we have this vs it's like don't Stone us to death because of it it's not our fault it's because how a works it generates first of all like an empty um empty object and then it fills the data inside it's kind of java thing and I guess um the Berkeley guys didn't know how to handle it so uh if you want to use it then you'll have to set your variables as vars so basically the way to start a scaling job and let's kind of ignore all the Imports let me kind of uh bring uh the screen a little bit up so everyone can see nicely um so I have my class it gets args and it extends job uh which is a scalding job and that is um basically all you need to do and we'll start with something simple and we'll show a couple of examples that go kind of uh more and more into depth and what I'll use is um Wikipedia in order to show our examples and in Wikipedia so that's what this first case classes for is I I just kind of removed everything unnecessary and and we just have the name of the article and the entire text of the article as one big string and we'll use it um in order to show some how easy it is to do things in scalding so let's uh start so one of the things um that we do kind of the map reduce Paradigm for everyone knows there is the map stage and the reduce stage map is basically Transformations the as simple as it is in the scalar collection and actually it's not where the most interesting stuff happens is you know we take um some data apply transformation we get another data most of the interesting stuff in our at least machine learning um careers Is How We Do the reduction because there is a lot more possibilities there and I'll uh our talk will be more about how can we do nice reductions and I'll start with something very simple like for example calculate the average length of a Wikipedia article um so let's just uh do that so I'll just the fine function here and I will load all the Articles um and here is Chris seers here developed this awesome thing called packed our source and now it's actually part of the SC the scalding project it was pushed like last week and you can just read AO um in it with it sorry so that's the way um I read um Hadoop data in AO so it's basically there and plus articles. AO um so now what I have here is this packed a source article is actually a kind of a pipe that has articles in it so every row will become this object article and what we want to do first is to calculate the average average length so one way the most naive way to calculate an average is to do two passes we do one pass for to count uh all kind of to sum all the lengths the second to count uh how many articles and then we divide but in kind of real life machine learning scenarios we almost never want to do um two two passes over our data because just inefficient when we are talking about pedabytes of data um the second thing that we can do we can kind of maybe have a topple or some other intermediary object where we uh use it for both for summing and Counting and then when we finish we kind of divide one by the other and we do as well but since count calculate average is something that is done quite a lot in scaling they have a function for it so first of all all types have this function aggregator and it gets something called an aggregator and will and we'll see what aggregator is in a second and one of the aggregators is called an averager so it um it's worth to mention what is algebra algebra is in kind of linear algebra or not just linear basic any type of algebra library that developed by Twitter and they have a lot of very very useful stuff there it goes hand in hand a lot with scalding but it can be used also outside it doesn't have to if you have a lot of like math operations that you do you need to do and also it has um kind of mini version of what you can find in Scala Z like Trace mono groups semigroups Etc then you can just use algebra algeb and it has a lot of the code for you it doesn't have to be used with scalding so um we use this aggregate um and one of the problem that um I kind of immediately get if we look at the averager it's a monoid aggregator and we'll get to it very soon that has three Ty parameters the first one is the input then it's kind of the um intermediary data which is the monoid meaning something that can be summed and then they do a a present which is uh we take U kind of our summing the object and we display it um in some other form so in terms of map reduce you can think about it as map reduce and map again under reduced result so we averager needs to get um a double so what we will do is um so types uh sorry pipes have just a function map and it's very nicely the same that uh the same kind of code you will do in uh just regular scalar collection API so we can take our pipe of Articles and actually now we transform it transform it over to a type of doubles so now we have a type of doubles and aggregate averager will work and it actually will also return tight pipe uh of a double but that uh since we aggregate all of over all of the Articles we'll get just a single number and that number will be the average length of Wikipedia articles and again here I want to well yeah there is no need to save it as aage we can save it also as T tsv CSV and they have a lot of other formats so sorry not string it should be double double and here is dear plus and let's call it average. txt okay and let's just call function and I'm kind of a maven guy I'm I've been working with Maven for quite some time time now still didn't really get used to SBT so excuse me for that um and Maven has this really nice scalar plugin which we can just do Maven Scola run and now it will run the entire code and let's see kind of what we get so now the you see it's stuck on this compiler plugin which is the the berkele plugin that takes our AO uh record classes and actually does the schema generation for some reason it's kind of slow but hopefully we'll be on with our lives pretty soon okay so we have success and if we look at can we see here cut average txt and you can see we have the number 27525 which is kind of length in characters of the average Wikipedia articles okay that's a nice start but obviously we don't need scaling in order to do that and a lot of the time that um we don't want it to do just average we want uh um to understand how our data looks like uh for each variable and we want to get additional features um for example um the average the count the sum and also the standard deviation uh this the cortosis of our and distribution and actually here H scalding has created something for us as well and I will show you um how we can also generate an an aggregator that is not necessarily um predefined in scaling so we will create now um new aggregator for us but it will use something so let's look at this class uh algebra has a class called moments so moments is a pretty nice if you see the case class here has like parameters and again it um it has also a group uh for it so the groups can take all these parameters and for those of you who don't know what group is is basically like a monoid but with negation and subtraction and it calculates the count the mean the variance well the standard deviation is just the square root of the variance the skus and the C toes so let's try to um but here you can see it's a it's a case class it's not an actual aggregator um so let's see how we can create an aggregator so the way to do it is just to um extend so we have an Let's see we have an aggregator class yeah no I want a different aggregator well never mind let's where is it and we also have a monoid agregator which is let's look at the aggregator first so aggregator like I said it has this map map reduce map stage which is the prepare then the reduce the preparer takes the input and just transforms it to some other type the reduce takes two variables of these types and well reduces them and present just again does another transformation on what the reduction was and since the reduce um and monoid kind of go hand in hand then we there is also the tra monoid aggregator that can instead of the reduce function can use can take a monoid and just use the monoids own plus function which is if we already have a monoid for our data is actually pretty useful we don't need to um lose data again so extends monoid aggregator and now that we will again we'll start with double because that is the data we have and let's take um what we will reduce on will be uh moments and uh what we will eventually U kind of output is list of any and I will explain in a second why why it's any so like you can see it we we have three functions to Define so the the monoid is just the moments group obviously every group is a monoid as well so we can uh use group here groups here and we have the prepare so we need to take this double that we get and somehow to create a moment out of it and actually moments has an um apply function which as we can see takes um takes a single value and returns in uh a moment object you can look at it as the zero as the zero of the of the monoid and we need to also present the data in the format that want let's see if it's okay and well here I'm going to do something a little bit nasty but um to list that's why it's the list any because it's just a topple of five elements and I don't want to write out each element so I just made it a list and now the the code to produce this data is actually again very very similar now that we have this moments aggregator defined again we read the articles we transform it into doubles because we actually don't have to transform it in doubles so one other thing we could do here is just uh start the mon the aggregator with an article and have it just return moments and the same application just we can do it here right so text Dot length dot to double so we can also that that's that's what the the prepare stage is for to kind of get your data in the format it's supposed to be in for the reduction stage and here instead of the averager now we have the moment aggregator and hopefully I haven't made a mistake we I and just list of any and let's call the file moments yes s let's just run this function now wow that's interesting stuck okay that not supposed to happen oh awesome and again clearer we'll do the maven Color Run and hopefully this time we will get so remember one of the um one of the moments is the mean so hopefully in the mean we will get the same number we got before and we will get other numbers as well that is very useful for us to understand how our distribution uh looks like awesome um and now cut moments okay so now we have the elements and we can see actually yeah the mean is the still the same we have it's not the entire Wikipedia in order to it's just thousand articles from Wikipedia The variance is nine and something and this is theunis and crosis and actually now it kind of becomes um um much more understandable how our data looks like something that was kind of hard to understand with just one number which is the mean uh of the Articles and this is a this aggregate pattern is very very powerful so you can do pretty much everything with mono you just you need to think about okay how do I sum and just example that are already exist in scaling so we have uh priority Q um priority Q aggregator so for if we want top n elements it's basically a priority que that um s sums it like we do a monoid from this perity q and it sums two perity cues and gives us the top elements in a in a single pass without us needing to sort the elements F the elements first which is uh much more much more efficient um another monoid hyper log log uh for those of you who don't know hyper log log is this extremely efficient way to estimate cardinality with just like 12 bytes of information we can get a cardinality I mean how many unique IDs for example we have in a data set with just 1% error rate so we again estimation of you estimating how many unique elements we have is naively can be done only by again sorting through every all of our data and kind of doing the unique so Harper log log and they have count means sketch and I I'm probably forget Bloom filters Bloom filters are also monoids and you can just pretty much do uh all this kind of niiz data set using uh monoids and if you want to aggregate your data and apply this monoid on then and also by the way obviously we don't need to do it on our entire set for example I could have done and find me the kind of moments of old old begin for you know just um what the first letter of the Wikipedia article is and the only thing to do here is just to do this group by and do text and like the first element and then we Group by by the first letter and now the group by will be um will get an answer for every letter um in the alphabet so again kind of nice and here I will let chrisy continue sorry M sitting in here somewhere okay uh let's see here hopefully one of these has the what oh hey there we go look okay uh let me clean some of this up real fast cool and there we go all right uh is that yeah okay cool that's visible okay um so I was actually going to go through and uh we kind of gave a similar talk a little while just a week or two ago um and I did some stuff with Mout uh sort of pulled some clustering stuff out but in fact uh actually just a day or two ago I taught an internal class at uh PayPal and we we basically started from what is this Scola thing and by the end of the day we basically wrote a scalding job to calculate tfidf on a bunch of stuff that we we had so I thought we we' just try and do tfidf actually because that's kind of a fun fun thing so um if you've uh oh something just went wrong oh dear let's screw this thing in so I don't do it again come on so if you've never heard of tfidf um it's a pretty common thing for a collection of documents uh and a given term in a document we want to know sort of how how important is that term uh inside that document or inside the collection um and it it's not a hard thing but it's it can be involved uh if you were to go and write this in in a bunch of map produce jobs but I want to sort of come on uh oh we uh it looks like it's it went to sleep on me do you want can you log in real fast pH sorry I'm not sure why uh it just slept so it it's a um yeah it's a pretty simple thing uh but it it has a few steps if you had to do this as map ruce it would be really nasty um in scalding it's going to be pretty easy so um basically there there's three things here that we're really going to need um so again we I'm working on kind of the same Wikipedia collection and we're basically going to I called my things Wiki articles instead um and basically I just to prepare I'm going to read some articles um and just going to rip out the the title and the body and just because I'm going to keep using it I'm going to go ahead and split up the body by uh Whit space as well so we've done some some kind of uh ghetto prep on it already um it's it's not perfect but it'll work for what we want so so TF is basically given a term uh and a document I want to know how many times did that term occur in that document okay so term for frequency so let's see what we can do so we can take our articles and so given a termin an article we have to think about what are we going to put out um and we probably want to put out multiple things per per article right so we're going to put out something for every term in the article so we'll probably use something like flat map okay okay so coming in uh I like to name these things coming in so coming in we've basically got our title and we've got our let's say token array and let's make a function out of this guy so uh what are we going to do so now we're the the sort of really fantastic thing about scalding is is that this sort of articles. flatmap this is the scalding piece right and now I'm just giving it a regular Scola function to operate to use um so let's see what we could do here um basically what we want to do is we want to go through um for everything in the token array we would like to put out uh basically the the number of times that thing occurred in the article okay so we could do something like uh let's see let's make a counting bag and let's do group bu and you know just to be a little bit fun let's uh let's Group by the lowercase things um and then so we've got this guy let's do map values so the when I group by I'm basically going to get a list of all the tokens that match this key um let's just do map values and let's get the length of each one okay so there's our counting bag and then what I really need to put out now is some sort of iterable I'm using a flat map so I'll do something like bag. map so for every essentially count in the bag or account in token excuse me um sorry when we go live we just want to spit this out along with the document title because we're going to use this later so let's do something like token title count okay all right so that's it so if we were to inspect TF we'd see basically we've got a type pipe of string string int okay not so big not so bad so there's our term frequency for every term in document we know how many times it occurred okay so now let's actually get the total number of documents well actually this one's super easy uh we can just take articles right well we we want all of them but let's just to to be kind to our reducers let's actually ask for the keys so remember articles is a is this guy who's a tyght pipe of sorry if that's hard to see but it's basically a type pipe of string and then array of string so I'm going to get the keys I don't need to send all this crap down um I do group all so this is a shortcut for basically Group by unit so I'm I'm saying smash everything down to one reducer um and then size so let me show you size really quick um and then values because I I really don't care about having the unit attached to my value so size just as a quick detour is pretty darn cool um oh shoot we don't have soures down do we so size is basically using uh algeb and defining basically using a monoid under the hood to map everything to I think long and then just summing but using algebra again so it's pretty slick um in fact you can call sum after you group and uh basically it will find if you have a monoid and scope on the type that you're summing on it will just do the right thing and go through and sum using that monoids plus function so this is great um okay and uh so you know what I know for a fact that uh at the end I'm going to use this guy uh as a double so I'm just going to do something like um like that uh there we go cool so D should be a tight pipe of doubles it is in fact it's only one double um so we're actually okay there uh oops you can see that I forgot to cop out some of the stuff so let's let me just uh we'll do this as we go apologies you can see behind the curtain okay um so this guy TD count so this guy is basically I want to know so I know how many total documents I have right so I want to know for a given term how many documents did it appear in right because our IDF our inverse document frequency is basically a ratio of the total number of documents and the total number documents this guy appeared in so it's actually not so bad it looks almost just like term frequency except we're going to flip it so let's go articles and we're going to put out uh again we're going to use flat map and we'll use something like case uh when I have title title array and this guy is a partial function so it really wants this partial function notation okay so what do I want to do um basically I want to walk through uh my my tokens in the array and I want to spit out that token and the title and then I want to count up all the titles that I ended up with at the end right um so there's one sort of sticky Point uh which is I probably want to do something like make that guy a set first so I don't spit out too many things and for some reason it really doesn't like it if you don't specify the type here um and and then this guy I'm just going to map so this guy's like a token oops and then I'm just going to spit out the token and the title okay am I missing some okay we're good so we've got that okay then we want to we want to find out for a given token all the titles it appeared in so I can just say group in this case so group is is kind of sugar um did I miss something here I missed a pen didn't I there we go so uh ins scalding group is is sugar if you have a basically a tuple of aity 2 it will just Group by the left one in the Tuple um and then I'm basically going to just say size again why not um so at the end of the day now this pipe basically is giving me I have a count token and then and the the number of Articles it appeared in okay so IDF let's bring these guys together um so a really slick thing that is in scaling uh is basically this I can say TD count um I have to group this guy again unfortunately it's just a a syntactic thing maybe we'll fix that soon um then uh excuse me I don't need to group this guy I can just cross there we go and basically I'm going to cross this guy with d so I know for a fact that D is just this one thing so what this will do is rather than spawning a redu phase it will basically just take this and ship it off to every mapper uh that that TD count lives on and I I'll have access to it there so if we were to just sort of interrupt right now and look we'd say basically this guy has a a a string long and then it's got this double appended everywhere okay um so okay so then I all I have to do is basically map this guy and I've got a case so I've got sort of a nested business now so I've got this guy which was a token title and let's I just call it uh excuse me token count and then this guy which like do count maybe and we'll send this guy to what do we want so we just want something like token and then we want the we actually want math log um basically count divided by count okay uh why are we not yep everybody's happy okay so now we've got our IDF we just need to get this guy back together um I'm probably running a little low on time are we okay actually you got about 15 minutes I've got 15 okay so let me actually I'll leave this guy filled in um just for fun so basically the last thing is all we have to do then is get the IDF thing grouped up by token that's this IDF group we're basically going to take TF uh and join it on the token again right and then we're going to call map values so in scalding when you have a essentially a a join or a group rather um essentially it holds the key fixed and you have access to the value set there's a a map values and a map value set so the map values is smart and will happen on map side as much as it can um whereas the map value set if you need the entire bucket uh to calculate something it'll actually FL down to reducers so we use map values when we can um and basically all I'm doing here is basically say take the the token and for every token and title basically spit out the the TF IDF score for that guy that token in that document so the term frequency times IDF um I don't need whatever I grouped on I'll throw it away and then I write this guy out and let's see if it runs um so I I've got a little Runner set up um we'll find out if it actually works if not we could always fall back to Maven it's run before so let's see I'm I'm running it in local mode essentially again uh oh oh you know what that's the guy I meant to work in and I just forgot let's try this again all right here we go it takes a little longer than averaging um but hopefully not terribly longer so it's running yeah do you guys have any questions while this uh really giant computation going on yeah uh this one's I think 293 um but it I've run the same thing in 210 and it works just fine yeah yeah the that Avro plugin doesn't work nicely in 210 until somebody decides they want to go and fix it for us and uh and make it work in 210 if anybody wants to do that uh I'm sure both fali and I'd be super happy um is this guy still running yeah other questions uh Z question in the back why you say that run sorry oh sure because you you see basically what I've written here right so this actually spawns three map ruce jobs um in fact I can show you the the graph if you want it'll spit out a graph maybe we can do it afterwards because it's it's nice if we have time um but essentially it's map produces a functional Thing by Nature um and sort of writing it in this functional way is is sort of agrees with me um but in terms of just like General use there's a lot less boiler plate going on here right I sort of describe what I want to do and and let uh scalding and cascading sort of plan it out for me and do it in the right way um the other thing is that in using this sort of typed API um we're really very very close to regular Scola collections code um so almost all of my development I do it like in the reppel or something on a local set and play with stuff and then just kind of change you know how I read the data and how I write the data and it's almost identical um so in terms of productivity I find I'm much much much faster using this yes does it run spark um scalding does not run on spark um I think the cascading guys are looking at running on yarn but SP spark has a really similar syntax so all the stuff I did here you could write in spark with very little changes in fact there's somebody trying to kind of unify all of them maybe it'll happen maybe it won't um but it it would sure be nice because there's another thing called Scooby and another thing called scrunch so it looks like maybe this actually ran um yeah here we go right and so um okay so we were a little ghetto in our our splitting so we we have some weird characters um but then we get down to things like zero um but if we wanted to find something interesting like uh the um it's probably in here somewhere yeah so there's the appears in a lot of places um you'll find usually the you know is is not that important um I know for a fact that from the last time we gave this talk like there's a uh Anarchy was like one of the first articles so we could find like you know whoops let's do it from the start so you know like Anarchy is really important to anarchism and uh uh not as important for uh mom Luke and uh not quite as important for porn actors apparently either um but anyway you you can see what it does it's pretty nice so anyway I'll hand it back to Vali so just one sec okay so let's see now what kind of um what other things we can do uh very nicely with uh the Wikipedia data and uh what we saw here that cred with the tfidf um okay let's go back so uh what I'm going to do now is something that is called I'll try to find similar articles on Wikipedia and the way to find similar articles is again at least one of the one of the ways is to see um how matching their words in their frequency so if two articles have um the same words in the the same AR in the same frequency we can say okay uh these articles uh um are kind of matching and we can produce a score and the similarity function I will use is called a cosine uh similarity basically it takes um all the um all the words in both articles and multi multiplies the frequency of U the pairs of word the pairs of the same word in both articles and sums all this together and we get we get a number um so there is a way kind of short way to do it and there is a long way to do it and I will do the short way and I'll leave you as an exercise to do kind of the longer way because the Shor way is actually pretty nice we will use this scal the scalding Matrix API so scalding has a matrix API um that does a bunch of uh cool stuff and let's try to make it work so um let's see we have uh the class here that article War count that we used and let's article word counts and well we can use it here um but let's just read the data that Chris just wrote let me see if um it's called work counts. Aro so again this the data is saved um in an a format pack our source everything is nice so now um there is a way to transform a pipe to Matrix and let me show you how can you do it so there's map to Matrix and basically what mapto Matrix gets it needs to start from our class and to transform it to a row column and value what is kind of very very nice about um The Matrix API in scolding that it doesn't really tells us that rows and columns have to be integers it doesn't have to be indices we can actually use strings and uh whatever we want because it's a sparse Matrix representation so it's just these topples of row column and value and since it's a sparse Matrix there is kind of a lot of neat ways to make it um very nicely parallelizable between a lot of machines so let's just uh do it so we have this SW count and we need to get work count. um so name is name is orle word and I have it in doubles here the count so now we have um we have a matrix of string string and double and now that we have the Matrix we can do um a lot of kind of nice Matrix operations on it like for example row normalize and we can see here we have kind of free normalization and the reason why um I do the uh normalization here because I don't want it to appear that two very long articles are similar to each other just because they're very long and have you know just by sheer coincidence a lot of similar words I I I want to kind of normalize the effect of the length of the article so I'm normalizing and here I'm using L2 normalization which is the square root of the sum of squares of the frequencies so basically the sum of squares of all the frequencies will be equal one now and now when we compare every article to itself we will get one instead of some arbitrary number that is a kind of function of the length uh of the article so let's now that we have let's kind of call it Val mate and now it comes to so like I said coine similarity is basically just a m Matrix uh multiplication so all we need to do here and instead of multiplying the entire matrix by the entire Matrix I will just um pick one one of the element and like you see it's actually ask me for a string so let's say a Wikipedia article of Audi can try anarchism I think one of one F Chris's favorites um and we can just multiply it by The Matrix but we actually need to transpose it in order to get well this whole row roll column stuff nicely and again think about it it's it's it's just math you know the multiplication is the dot product operation of matrices here and writing this one line of code is well yeah I I just uh really try it at home we will post the code on GitHub in order to see how much code this one line saves it's yeah even in scal even in scalding itself without the Matrix operations so and just because kind of um top let's just pick the top 100 elements because um when we multiply a row by metrix we basically get another row Vector that will only have the titles and the similarity scores so we select the type elements and then like always we write them let's write them to type tsv of string and double and call it deer plus matrix. txt okay that's the entire code we need to write to do um a multiplication to get the most similar articles to the article Audi if we want to get a matrix of all articles with each other what we need to do is just to remove this get get row and that will give us the entire Wikipedia similarity between all articles so um I think that's really really cool um and the fact that again um obviously I will show it on my local machine now but it's the same code that goes into the Hadoop cluster you don't need to change anything besides perhaps this local thing and actually you can also simulate uh sculling also uh knows how to work with local hdfs mode so if I write here GFS it will actually spin up a Hadoop on my local machine and will run it as if it was running on real Hadoop so let's just run Matrix here go to our beloved terminal hopefully it will work and again that will that will take a minute or two so any questions you two minutes okay the source of what hey like we will post it on GitHub and go to the SS col group and we'll have we post anything else oh one thing that's not immediately obvious yeah one thing that's not immediately obvious here also is that um essentially it's doing enti time type checking all the way through that PPE um so if you've ever run things in in regular you know cascading or or Pig your H or something um and you know 6 hours into the job you added a double and something that doesn't add with double nicely the whole thing Mars and you just lost that six hours this essentially prevents that at compile time which is a really fantastic thing and it goes all the way through to to basically write out uh yeah had another that because I know on object cascading everything's essentially an object and you always have to cast back and forth so you guys are you guys are still doing the underlying casting so scalding yeah is essentially it's it's basically the there's a two API in scalding there's like well three there's the Matrix one but there's the typ one and the fields one and the fields one happens just like regular cascading you throw around two bols which just have stuff in them um the type one essentially has been a little bit they're being a little bit tricky and passing around tup that have exactly one thing in it and it scalding is smart enough before it hands it to cascading to figure out uh that you had the types link up all the way through and that there are things that will actually play together so I'm curious because I know uh one of the differences with Scooby was that they don't have that underlying cat thing yeah does do you know I mean I know you guys scolding but does that actually buy anything or or is that just them saying like well we don't have this well no I mean the type and scalding and Scooby are almost identical yeah I mean they're for for very much all functional purposes they're really identical um and I mean at eBay like half the people in our group in fact there's another eBay guys is going to give a tonk uh next in this room he's going to talk about Scooby he he uses Scooby and half the group do Scooby and the other half use a scaling like an internal War yeah it's a a very low grade War um and it it just it all works out perfectly fine actually they really inter out pretty nicely okay so here are the results as as we can see we did something right by Audi getting called basically one after the similarity we can see that the third one is auto racing the fourth one is armored car assembly line IL legally so I don't know why African-American is so Hider but we don't have too many articles so it just does anyone know who August hch is found of the company um let's see and we have this tool on Google and that links to Wikipedia and a first is a German engineer p and actually the founder of manufacturing G that would eventually become Ali so as you can see it's actually work it actually worked so it's pretty exciting about that with fail life and it was a very simple and you know it's hard to stress like how simple this well so obviously we there's some tfidf needs to be done in order know not to get two article that similar because they have the word b or the word a in them a lot of time but once we did the tfidf normalization is a single line multiplication single line and that's it we write it so that's the end any other have foris it's pretty easy like in the TF it's pretty easy basically take that uh like document frequency or term document frequency pipe and siphon on the the ones that appear and you know like 80% of the documents populate a map or something call those stop wordss and then again cross that map back into the the other one to remove them I have more of a question about if you guys run into so so I've run into this not in map R but one thing with this go collections API which essentially this mis matches is it's really really really easy to introduce complexity uh you know exponential complexity um now for me that that means something that should run in you know 10 milliseconds runs in a second but for you that could potentially mean a job that supposed to run in an hour takes 2 days how often do you guys run into that where you got to really go through and like so for me very infrequently um usually that whatever so that it's it's I think it's it's hard to some extent to introdu complexity in like the actual scalding flow whereas if you're still using collections inside of course the sort of the functions you're passing in right use collections code and in that case the things taking you know a second instead of milliseconds it probably slows your job down but my guess is that you can just pay so much IO overhead and time for the h process that in general it's it's a really something I haven't run into much I guess well I can say that maybe because the reason why Chris haven't run uh this as much is because he's been doing it for quite some time now and but it's definitely something that happened and and one example is you can yeah you can blow your data very quickly like for example if you do the UN like a bad join then that a bed joint can really kill kill your job a flat map for example if you take um kind of list that you have and now you instead of a list you have every element in a separate row then you know your mappers can also go back so there is some fine tuning that needs to be done and especially once you go to production and tried um yeah it kind of I think it's easier than Chris describe to shoot yourself in in the food but after you kind of do it a lot it's well it's the same patterns all over and over again so you already know once your job kind of doesn't finish you kind of know where to look at that's it okay well thank you guys