SF Scala & SF Bay Area Machine Learning, Joseph Bradley: Decision Trees on Spark

SF Scala & SF Bay Area Machine Learning, Joseph Bradley: Decision Trees on Spark

Recording: SF Scala & SF Bay Area Machine Learning, Joseph Bradley: Decision Trees on Spark

uh thanks a lot um yeah and thanks tony alexey and sf scala bay area machine learning for organizing course yammer for hosting and everyone for coming um despite the lack of food but hopefully it will be well worth it uh and apologies for this column um hopefully you can still hear and see the slides so this is about decision trees on spark and for those who are machine learning experts what i hope you'll get is a bit more knowledge about doing machine learning and distributed setting especially using spark and scala and for those who are experts in spark and distributed computing uh hopefully you'll get a bit about uh using an example machine learning algorithm uh decision trees and how that interacts with distributed computing and for everyone i really hope this will be useful in terms of basically understanding how to use spark and the machine learning library built on top of it ml lib to do big machine learning so i'll start just giving a bit of background on decision trees i'll start with a much loved example spam detection so say i got an email you know get viagra real cheap send money now to get the way we're going to look at this on a computer is to parse this into a more friendly representation and one really simple one is word counts so here there are two instances of the word get one instance of the word viagra and so on and you might have other types of data you'd include to represent this email for example do you know the address of the sender address known and each of these elements is going to be called a feature which basically says some descriptor about this email and then this entire vector features i'm going to call an instance or an example so our goal with decision trees and other machine learning models for prediction is to take an instance examine this feature vector and predict a label where in this case the labels are either not spam or spam so given this sort of high level goal uh how does a decision tree model actually do this prediction so we're given an instance of an email in the form of this feature vector and a very simple decision tree might be something like this we start at the top node and this is going to test the value of a particular feature is the address known in this case uh it's not and so we're going to go to the right the edge labeled with no then we count how many times does the word viagra appear well it appears more than zero times so we're going to go to the right and finally we'll get to this leaf node which is going to say uh we finished predicting we're going to predict the label spam and so just to put some terminology up there these boxed internal nodes internal tree nodes are going to test a single feature value and these bottommost leaf nodes are going to predict labels so this is a really simple dumb example but it gives you a sense of the power of these models they can handle different types of features here we have a categorical feature uh address known takes two values and continuous features counts of words you can handle different types of labels here we have two categories and so it's a classification problem you can also predict continuous values in which case you call it regression decision trees are pretty interpretable models where you can trace down the tree and sort of see the exact process which led to that decision that prediction and models are pretty flexible they can be small in which case they're very simple or big trees in which case they're expressive powerful and yeah i just want to emphasize that these really are industry workhorses or parts of industry workhorses lots of applications here silly example with spam detection later on i'll do a little demo with digit recognition uh other famous examples include for example the winners of the netflix grand prize for movie recommendation included decision trees as part of their model so let me look at the outline first we're going to start with quick overviews of decision trees and spark and the point will be to show how traditional methods for learning trees don't distribute that well we'll then talk about how learning trees actually works on spark and this will give you an intuition for that underlying computation communication and the constraints you're working under in the distributed setting i'll then move to using ml lib in practice and in particular show a couple demos to highlight issues like model selection and accuracy communication trade-offs and finally i'll mention some active development so decision trees and spark let's first talk about how you might learn a decision tree if you're working with a single compute node you have this training data set where each of these rows is a training instance with its label and just for visualization i'm going to like collect all these training instances at the top and what you're going to do when you're learning a tree is pick a feature and value to test in order to split this data set where for each instance that instance is either going to go down the left path or the right path so we'll partition the data set and see intuitively what you want to do is group all the non-spam on one side all the spam on the other and in this case on the left hand side we have a bunch of not spam so we'll make that a leaf node and say we'll predict not spam if an instance goes there and then on the right hand side it was a bit muddled so we'll split further testing another feature and you know eventually we may say it's good enough we're going to do these predictions at the leaf nodes the key point in terms of computation what you're doing with your data is that you're recursively partitioning the training instances based on what feature uh and split you choose and you might start to see what the problem is when we start talking about how you might distribute your data so in a lot of big machine learning problems you have tons of instances and so it makes sense to partition by rows or the training instances so each worker will have some subset of this data well if you compare this to what we were just looking at in terms of the tree recursively partitioning data to the left and right uh you can see the issue when you start out learning you put your data on your nodes but you don't know how the splits are going to partition your data as you learn so for example with this you'd expect each worker to have some data points which you go to the left some which should go to the right and then on the next iteration a naive implementation of that traditional algorithm would try to collect all the instances which go to the left onto one group of machines and so on so basically you'd end up shuffling the entire data set across the network many times of course we're not really going to do this and so what i'd like to do is give a brief overview of spark and the sort of computational framework which is available and will let us get around this naive implementation so spark at a high level is a fast and general engine for large-scale data processing i came in all of uc berkeley's amp lab and has a really big open source community it's one of the fastest growing apache projects um 250 plus developers from lots of companies and it's included in every major hadoop distribution knob so spark is sort of this underlying uh framework for large-scale data processing and distributed fashion but built on top of it our number of libraries for sql for streaming machine learning graph processing i'm just gonna have time to talk about one of these the machine learning library and within that i'll note there are a bunch of types of algorithms implemented uh classification regression recommendation you know things like movies uh clustering statistics linear algebra and so on decision trees will be applicable to these first two tasks like we mentioned before so spark is sort of this general framework uh but i'd like to delve into a bit about the data representation used by spark it uses what are called resilient distributed data sets or rdds which basically take the form we were just looking at where each worker is going to have some set of partitions each of these partitions is going to have some subset of the rows or training instances and then spark will have a driver or master node which can sort of aggregate information from these different workers communicate with them and handle the coordination so just to give a sense of the types of computations which can be done with rdds one very common one is a map we take each of these instances and we're going to transform it into some other representation for example you just saw with an email example could take a raw text email and transform it into a feature vector the other important operation is an aggregation on each node you can compute some statistics or other information about the data you have there and then you can aggregate these to the driver node i'll finally mention one of the big pluses of using spark as opposed to maybe traditional mapreduce is iterative computation which of course is very common in ml so say we have our rdd like this may want to aggregate some statistics which is what we'll do with decision trees may want to communicate some function of those back to all the workers uh and iterate like this and so the big advantage of rdds is that a lot of this computation is maintained in memory rather than writing to and reading from disk on every iteration uh rdds of course are also the rs for resilient and so they have built-in love failure tolerance which is great for machine learning people like myself not have to worry about uh these details of distributed computation so that gives sort of a very high level of traditional learning of decision trees uh spark and will lead into discussing a bit more of how we actually have implemented this in spark so at a high level we're going to train trees sort of level by level starting with the top node we'll choose what feature to split on and then we'll recurse we now have two nodes to uh examine and we'll train features to split on for each of those and so on now at each of these steps the key choice is how do we split at that node what feature do we use and maybe what value do we choose for going left versus going right so recalling our spam example we want to choose this feature to test here we have a binary feature true or false and we need to ask the question how good is this split and this will let us compare different features different splits and choose a best one so what do we really need to know uh about this well basically intuitively we need to know the number of not spam instances which go to the left uh spam ones which go to the left and same for the right and basically this will let us say if a lot of spam instances go one way love not spam go the other way it's a good split we'll call these the sufficient statistics for this split and these will allow us to compute what's called the impurity or information gain which is basically a function taking these sufficient statistics and giving you a real value indicating how good this split is and note that i'm saying this is i won't go into details about impurity but it it can be sort of a general function because you could have different trade-offs especially when you get into things like multi-class classification where even with binary classification with two labels you could ask yourself is it better to have a left hand side with all not spam and the right hand side very mixed or to have both a bit less mixed but neither side perfect so there are various trade-offs sort of set by this impurity i'll give an example of how this behaves in practice later on but basically you can see that in order to grade the split say how good it is we just need these sufficient statistics we don't need to know everything about we don't need to know something about every training instance on that driver node so we're going to use this aggregation mechanism in spark where all of the worker nodes have some set of training instances they compute stats about the various splits and then once we add all of these together the driver node has enough information in order to choose the best split so returning to that high level view training level by level on the first level we need to choose a split for this one node so as we just said we aggregate some statistics driver node picks the best split and tells all the workers what it just chose then on the next iteration we do the same thing gather statistics about now two nodes driver node can make a decision and so on and the key point here is that we're only making one pass over the data per level and this means that for fixed depth tree the running time is going to scale linearly with the data set size and this is what you might hope for with an ml problem and actually works out in practice as well so to illustrate that here's a little example of binary classification uh there's a 16 node ec2 cluster training trees with six levels here on the x-axis i'm showing the number of features in the data sets and on the y-axis the time to train in seconds uh the number of instances um is fixed for each of these plot lines and you can see it's varying from 16 000 to some eight million but the main point is that these lines are straight so running time is scaling linearly with the number of features or in other words with the data set size uh up here for example it's pretty big data set 224 gigabytes at the top you can see similar behavior in terms of scaling the number of instances here it's the same sort of plot except that now the x-axis is increasing number of training instances up to 8 million and again you can see that the running time is scaling linearly with the number of instances i.e the data set size good so works out in practice and this gives kind of a high level view of how trees yes you have to sort of randomize data in a special way i'm not sure like what if one of those workers right suggests one thing is split right how so the question was do you have to randomize data in a particular way you know if a worker gets an odd subset of the data and with decision trees um rough answer is no um it's deterministic um more accurately there actually is a point when you're using continuous features uh where we do do sampling and i'll explain that later on and there it doesn't matter how your data are partitioned but it's not deterministic um so it's fine if you have you know weird data on one node and normal data on another um good yeah and do feel free to interrupt me and you know ask questions that would be great so now i'll take a look at using ml lib and decision trees there in practice and particular model selection and this accuracy communication trade-offs you can tune so i said in practice so here's uh one part of the api for decision tree uh we have a train classifier function which takes a number of values which i'll sort of use as an outline the first three are really quick basically describing your data we have an input data set which is an rdd of these labeled points the feature vector and the label number of classes which you need to predict this is a classification problem so we need to know the number of categories the label can take there's this categorical features info which is sort of this metadata allowing you to specify which features are categorical which are continuous and while decision tree will run if you don't really input this data it can help the algorithm really know how to handle features properly now we come to the more interesting ones so we've mentioned impurity before which is this function taking these sufficient statistics and telling you how good a split is uh the other interesting one um is the max depth which i'm sorry sagan please oh right so the impurity is a string here uh because we like to use uh simple types so spark supports uh scala python java apis and the original decision tree implementation actually had a specialized class which you would um use to specify the impurity uh the issue with that is that made it a little more difficult to interact between for example to do the python api um and so using simple types makes it a lot easier right so there are very common sort of named impurity functions uh genie entropy um and i'm sorry right so this could definitely be implemented via some kind of enumeration um we made this design decision yeah basically by looking at other very popular machine learning libraries and thinking about what would work best in terms of uh having these three different apis but yeah i agree there are different a variety of choices you could do um for max depth this is the maximum number of levels in the tree it's a really important parameter because basically the more levels you have the more expressive or powerful this model is but also the more expensive it'll be to train so what i'd like to do hopefully i can manage with one hand is a little demo uh looking at the max depth and impurity parameters so yes so there is that last parameter max benz that will come later yeah i will discuss that uh so good um one sec it looks like i need to i think unplugging my um thing from the projector messed with settings a little um good okay so what i'm showing here and please tell me and back if you can't read this very well but i know there's screens back there um is a window looking at the databricks cloud which is basically letting me run spark on a cluster here which i have uh this notebook attached to and basically i want to show you like a simple example to show how to train trees and give a sense of intuition about the behavior and how to set these various parameters good so um i'm uh not great at typing with one hand so i'm gonna uh just copy some stuff in um what we're going to do oh uh that would be awesome thank you great uh thank you um okay so we're gonna be looking at uh choosing the number of bends and the impurity and we're gonna start with uh loading some data and what this data is is this very common mnist handwritten digit recognition data set um and the way we're going to do it is we have a couple files in s3 these are being loaded and it's a training set a test set and i'm actually if you look carefully at the file names i'm inverting them so that actually we have a a lot more test examples and to count those we now have two rdds um which have a training data set test data set and you can see they have um huh they're not inverted sorry 60 and 10 000 examples respectively so as i said this is a handwritten digit recognition data set so this is what the data looks like the label here is the first value in this case of five and then you have a very long feature vector which are basically pixel values pixel intensity values so just to give you a sense of what this actually looks like here's an example image of a four from this data set and you can see it's clearly recognizable to us there's definitely harder examples but um it's sort of a rough grayscale image so now what i want to do is train some decision trees using varying max depth values and impurity settings and yes these are these like genie entropy impurities which are very standard uh implemented by most popular decision tree um decision tree libraries out there and basically i want to give a sense of how these various parameter settings are going to end affecting that held out data test error so what i'm doing here is running uh some commands where each is going to take um a set of max step values uh and it's going to map each of those into a decision tree train classifier call you can see these parameters we are looking at before we're training i guess 16 trees each with different max depth different impurity and we have 10 classes uh the 10 digits which we want to recognize so this is straight going through uh the data sets uh training each of these in sequence and while it's running i just want to define a helper function which is this accuracy and it's going to take a model and i'll put a double which is the accuracy on this mnist test data set so while that's the machine running um i'll also mention that for the sake of nisty later on i'll define a little case class which will put the accuracies into classes holding the depth impurity and the accuracy good so uh this command up at the top finished running and just to give a sense of what this did uh you can see we were training on a series of depth parameters uh depth 0 means we predict a single value i guess 1 is the most common label so we're just predicting that depth one means one internal node so we're testing one feature value and there are two possible outputs and so on so now yeah the feature is their individual pixels and i agree this is just like a simple example where you know you could do fancy stuff with uh hog features or whatever but yeah um so we're going to use this accuracy class and now we're going to compute uh a variety of the accuracies one for each of these trees we just learned good so now they're displayed as a table but what i'm going to do is mess around a little bit and plot it so it's much more interpretable good uh what is this interface um so this interface is then the shoot this interface is in the databricks cloud i'm unfortunately running on a test cluster which other people may be messing with um good but after a little fiddling now have a plot which shows on the x-axis increasing max depth uh the y-axis you can see um is test accuracy and you can see basically as the trees are getting deeper they're becoming more expressive able to model the data more accurately and therefore we're doing better uh in terms of test accuracy you can see in terms of the two impurity values for our impurity functions for this problem they don't actually matter that much both are doing pretty well good so returning to this we have just talked about impurity and max depth and how those tend to affect learning and practice um now i'd like to just mention max benz and i think this is a particularly interesting case where uh it gives you the power to tune this communication accuracy trade-off sorry i can take no noise thanks so much um good so in terms of choosing how to split uh recall that with this binary feature we it can take value true or false and we split either left or right and for computing how good the split is we have this one bin uh what i'll call a bin on the left of sufficient statistics for the group of instances which fall to the left and one bin for those which fall to the right so i'll call this two bends of sufficient statistics per feature but now of course you might wonder what do you do if you have a continuous future for a continuous feature a split is going to look like a test on this number line of whether or not the feature value is less than some fixed value here so what this will look like in terms of binning is your training data is going to fall on this number line some to the left of this cut some to the right and this partition is going to create two bends where to know how good this particular split is you need to have counts for the left and counts for the right of course you can see the issue where if you have a lot of training instances you could potentially split this continuous feature between every consecutive pair of values so you'd end up having tons and tons of bins so a naive implementation would have a number of bends on the order of the number of training instances but what we do and is a common and effective solution is to discretize the data so you can imagine we have this number line training instances lie along it and we could a priori choose to chop this up into big chunks and in ml lab this is done by taking a sample of the data so that it's efficient but somewhat data adaptive and then we could consider doing a split at each of these cutting lines in this case in order to compute how good each of those splits is we basically need counts from every region between the lines here we have five possible bends and you can see of course this is uh much smaller than the number of instances and gives us a sort of trade-off where the more bins we allow uh the maximum number of cuts uh the more like we are to considering every possible cut between instances and so potentially we'll be more accurate but if it's smaller number of bends we'll do less in terms of communicating the sufficient statistics so in terms of communication how much is there well on each iteration we train one level of the tree so for each tree node for each feature and for each bin for that feature we need a set of sufficient statistics so the total number is number of nodes times number of features times number of bends per feature and so the really tunable parameter here is this last one which we set using the max benz parameter so this will let us sort of throttle communication potentially at the expense of accuracy thankfully the expense is often not that great and for some problems you know it doesn't even matter and what i'd like to do is uh first mention why communication is so important and then show why we can throttle it with this max benz so what we're looking at here is the spark web ui which is giving sort of the status of a bunch of jobs which have run in spark as part of learning a decision tree here it's two million instances a few thousand features uh about 70 bins per feature and so i don't expect you to be able to read this but zooming in on on a particular part on the early stages of learning we can see a number of operations happening there's the reduce map those operations we discussed at the beginning those are happening quite quickly at the beginning less than a second and you can see that the data input is about 26 gigabytes and the interesting columns are to the right where you can see on this first iteration where we're training one node uh we're communicating about six megabytes of course that's really small because we're doing this thing of aggregating sufficient statistics instead of communicating stuff about all of the data but you can see as we move up into the second iteration we're training two nodes we have twice as many stats to communicate and so the communication is increasing uh 10 megabytes is small but moving down to i feel i think this is like the sixth or seventh level i forget um you can see that we're having to communicate a lot more statistics here 200 megabytes and although this is still happening pretty quickly and is much smaller than the total data size you can imagine what happens when you get much deeper in the tree so this kind of emphasizes how it is important to keep this communication in place in mind and one way of uh adjusting it is through this max ben's parameter so what i'd like to do again if i uh could enlist your help again and what work for you is stand this side second okay okay yeah thanks good uh great is just a quick demo to um continuing on with the same data set uh of how adjusting max bends actually affects accuracy so uh again what i'm going to be doing is um oops said i was bad at typing um we're going to set the max benz parameter to a set of values from 2 to 32 32 i believe is actually the default in ml lib um but this will show you some extremes and train trees for each one and this training trees is going to take the same kind of format you saw before we have an array of max spins values and we're mapping them to basically a bunch of decision trees which we're going to train this time just using one impurity and here we're training out to max step five um last time we went uh for a whole range of depths but this will let us sort of slice it and look at the effect of max bends and so uh once this is done i'll be able to take a look at for each of these max ben's values the accuracy of the decision tree which uses that amount of communication and then we can take a look at how that actually looks in practice um and so what i'd like to emphasize is that this is a particular data set and so this is a parameter which may actually vary in behavior for other data sets and of course will require some kind of model selection um but what i think this does sort of illustrate is that for a lot of data sets it's actually pretty robust uh so we computed the test accuracy and again i'll want to mess with this a little bit and uh basically say um good um for a variety of values of max benz from 2 up to 32 we have test accuracy on the y-axis and basically they're all getting the same test accuracy and so the thing to note here is that this is hand-wrine digit recognition a very simple models just down to depth five um but that basically it's saying that looking at a black and white image in this case um is doing uh pretty well compared to looking at grayscale and of course real implementations of uh onimnist data are really going to use much more expressive models and take a bit while longer to train but this gives you an example of how actually throttling communication doesn't hurt you that much on this data so going back to thanks a lot yes right so the question is are we gonna see a graph with communication bandwidth used unfortunately i don't have a graph available for that although the charts we are looking at before in the web ui we're looking at the total communication from each round and it is very predictable where for a fixed number of max bends if you double that number of max spends you're going to end up communicating potentially about twice as much data and i say potentially because it could depend on your type of features um you know if you have uh binary features you only need two bends even though even if you set max bends much higher and so the current spark master actually can vary uh the number of bins per feature oh thank you good so the question is how do you actually choose this cutoff for benning in terms of how do you choose the number of the max bends or how do you choose the places you slice for binning i see right right so it i said it uh is somewhat data adaptive and by that i meant that what we do is we take a sample of the data and that lets us sort of get a rough view of what it looks like and uh we sort that small sample on the driver node and then basically chop it up and that means that it's it'll likely adjust to the various ranges although it is a sample the samples taken from every node and it's basically taking sort of like a fixed fraction from every node aggregating to the driver yeah which is what i was mentioning about the algorithm not being deterministic but you're not having to worry about having weird data on onenote things like that yes another question to the aggregation pattern you mentioned that you have a folder workers communicating to the driver do you actually implement it this way or is it yes right that's a good question the question was yeah is it all the workers just communicate to the driver or is there some kind of tree and if you used spark 1.0 um it was basically just sending it to the driver there is a tree aggregate function which was implemented and is now being used in decision trees so the current decision tree actually does use this nice tree yeah yes um the starting example with like the email thing you talked about having like text features um is there a way to actually turn text features into sparse labeled points um because i tried to do that and it ran out of memory in the train phase because it was trying to turn it into a dense vector right so that's a good question the question is um and this is specific to decision trees is that right okay i so with knife base i would not have expected that to happen uh it should support sparsity but uh could definitely discuss particulars offline decision tree unfortunately does not yet uh really take advantage of sparsty in the labeled point you can give it a sparse vector but uh it's not gonna um actually take advantage of that yet uh definitely something which will be great to include um knife base i'm surprised and maybe it was yeah just like a slight misuse or something okay yeah it'd be great to discuss later any other questions do you guys work with like you know just trees or whatever so the question was yeah uh trees are great but yeah in practice you normally don't train just one tree and expect it to be a perfect model you do some kind of bagging or boosting um and the answer is that there's currently uh i'll i'll discuss it this a bit more later um they're like on the verge of bnn ml lib but um and there are existing projects on spark uh with code available for that um the current 1.1 release does not have ensembles but it's like on the verge yes uh if you're working down that path why concentrate on the performance of training a signature while you could probably get easier than realizing the training index yeah so the question is why focus on single tree performance when you could focus on like training a bunch of trees in parallel and i think that there it depends on which type of ensemble you're learning so for those not familiar there are generally two ways you're going to combine trees either bagging where you can basically train trees in parallel on sort of bootstrap or sub-samples of your data and then average the results or boosting which is very sequential train one tree sort of re-weight your data train another tree and so i think for bagging you're right um and while there are ways in which the current optimizations and decision trees are going to help with bagging um you could certainly do a lot of other sort of orthogonal optimizations with bagging for boosting where it's very sequential it's definitely going to be important yeah but definitely upcoming so in practice i just like to review we've mentioned max depth can be important to tune based on held out data model selection there's this max benz which is good to set low in general but increase if needed i did not mention the number of rdd partitions and for those familiar with spark this actually is something which differs from some other types of jobs on spark where it's often good to sort of over partition your data and have more partitions of your data than you have compute cores and the reason for that is if your jobs take different amounts of time over partitioning can actually sort of help balance that out and make sure all the workers are putting in equals roughly equal amounts of effort for machine learning and in particular decision trees tasks take about the same amount of time very uniform and so it actually is significantly faster often if you set rdd partitions equal to number of compute cores so mlib supports a lot of things classification with binary and as of uh spark 1.1 multi-class labels regression with continuous labels different types of features binary k category continuous uh various impurity measures and other settings that we've seen and also as of 1.1 python scala and java apis there's a lot of ongoing there are a lot of ongoing improvements uh i guess some of which i touched on earlier and going from spark 1.0 when decision trees were introduced to 1.1 um just to give you a rough flavor here again 16 node cluster fixed 3500 features here i'm plotting on the x-axis number of instances and the y-axis training time uh so the original implementation uh you know ran i guess reasonably quickly here actually uh for 1.1 it's being cut down to almost four or five times faster for these data sets uh likewise if we fix the number of instances um here we're varying on the x-axis number of features uh version 1.1 you know ran i guess reasonably quickly and now with one uh sorry 1.0 moving to 1.1 a lot faster so they're definitely ongoing performance improvements and even more on the way i'd like yes oh sorry uh mix of terminology so by 16 node ec2 cluster yeah i mean worker nodes um for two million instances i mean training instances or examples yeah rows in your data too many overloaded terms uh yeah so like we are just mentioning there are ensembles both random forest this bagging and boosting on their way there's a pr currently out there for random forests there's an existing implementation on spark from alpine labs called sequoia forests which actually is a great implementation and we're coordinating merging some of those optimizations into ml lib and boosting is under development there are model selection pipelines underway where currently under design but uh planned to be in mml libs soon and this is uh gonna make it a lot easier to do that sort of tuning of these parameters uh which we are doing by hand in the databricks cloud and some more internal optimizations so in terms of where to go from here a good place to start is the spark project page you can download it try it out locally or on a cluster their videos exercises documentation and of course you can start contributing via github you can also check out the databricks website where you can learn about databricks cloud i gave a quick demo of it earlier and some spark training resources yes um i'm quite interested about the performance test the graph of the performance test that you showed us just now could you please sure yes yes so what's that performance test conductor with with the assumption that that the type of ec2 instances are of this type what kind of ec2 is right so these and in this case yeah i ran these uh using our large ones and um they were the same type across all these tests um because those run on the same cluster um in terms if you're asking about like performance if you had a mix of types uh that's definitely interesting i personally unfortunately have not tried trees on a cluster like that yeah that was my next question would you potentially see a different result if you mix different types of easy tools right i think the main thing there would be that then what i said about jobs taking very uniform amounts of time would not be correct anymore and there it would make sense to over partition a bit uh in order to sort of even out um you know the slow instances and fast ones and do a bit more work on the fast ones yep so our factory um works well with in-memory computation yes so when you see much improvement if you use lesser lesser medium range in ec2 instances but you you would rather stick with hs1 because of the auditions one is a powerful instance anyway right so you're asking about like is it worth it to uh have like maybe mem right um so i think it may depend on the type of problem especially you know of course uh this has a particular set of constraints in terms of how much memory it needs how much communication i feel like it would sort of partly depend on the type of problem you're running and what its main bottleneck is certainly having memory optimized nodes can be very helpful yes what if those instances are scattered across regions i see you have some i think that in general for any platform that would be an expensive way to run things if you're communicating across the country i think it's uh you know i i believe it could be done i think it would be wise not to um great thanks yes the training times uh right so it's after loading the data and loading the data in general is a small fraction of the time um you know it depends on the algorithm you're running but with trees yeah um right yeah so does not include data loading great yeah so that about wraps it up about on time i think and uh basically you'd like to leave a summary up here and thank the many collaborators who have put a lot of effort into this project and in particular all these people and more have put effort into decision trees on spark like to think tony alexi and scala and barry ml again and yammer uh for hosting us here we'll certainly like to take other questions and i'd really like to thank you the audience for sitting through this without food pizza to munch on during it thanks live and spark uh support things like location sensitive caching and algorithms like that that are emerging as very powerful classification um so your the question i believe was what kind of support is there for for example locality sensitive hashing and um i'm actually not too uh i i'm not sure if it is there yet but i believe it's definitely something which like yeah i agree is important and uh you know we're always welcoming uh contributions to the project i don't know of an eta or anything on that right uh so right i'm not super qualified to talk about uh spark certification which the question was about one part of your question does it involve machine learning i know that it does touch on ml lib but i would definitely i guess for that i'd recommend going to the website i know there's info there but i'm not really involved with that yeah hi so i in the middle a little bit about building so right that's not in terms of single trees so the question was is there a platform feature for basically incremental model building your data is coming in as a stream say and you're updating your model as you go so it's definitely under active development i know that there are streaming implementations of certain algorithms as sort of a more general machine learning framework this thing i vaguely hinted at of machine learning pipelines and a design doc being on jira for a discussion um is definitely part of like sort of a big effort to update the api include sort of frameworks for it to make it more generally easy to do things like that um so i think it's sort of algorithm specific right now but hopefully we'll be more general yeah that is currently in there right so i haven't worked with this streaming part myself pretty new to databricks but um can i field a question do i if i may uh so uh this is shangri who has uh been on ml lib for a bit longer so we talked about streaming algorithms one is implementation i'm only streaming linear regression you can specify this stream of vectors and it will keep improving your model current model and make predictions on the fly and the other one we have is the streaming k-means and it's in discussion and we have initial implementation and i think it will be merged in the next release and basically you have a initial clustering model and then you can just for the incoming data you can update in your cluster and at the same time you try to apply a decay factor on existing nodes and then so you can always capture the current trend now we will try to add more streaming algorithm like something like a b testing and for the online model evaluation it's uh well a lot of good things are coming so i i feel like you have it up first and then i'll come to you so improvements from version 1.0 to 1.1 right performance of calculation is there like increments um going on for like memory fit footprint yeah so the question was what kind of improvements are there for memory footprint i think there are two areas where you could have improvements one is the sufficient statistics being smaller um and the other is the data set being smaller so first the statistics uh there have been actually and the current spark master is more efficient there than 1.1 where basically it's better about using adaptive number of bins for compressing the data set itself that's something which is currently in grform but hopefully we'll be in there before long yeah um communicating data though sorry i should add um there is some underlying compression which happens with spark which uh yeah definitely reduces that sometimes uh so i think i'd said okay right so as far as comparisons with other libraries um so there are uh definitely some i've run unfortunately i don't have plots here uh there will be a blog post on the databricks website before too long about decision trees and actually i take them back if you look back i believe there are talks listed there which do some comparisons with other libraries like maybe psychic learn um i think that there it's a bit difficult to do sort of distributed comparisons because i don't know of a lot of distributed implementations of decision trees yeah sorry so if you look up i believe from the spark summit um there was a talk on decision trees which included i believe it included some comparisons that'd be the easiest one to get to uh do uh so the question was support for cuda on gpu clusters um there is not as far as i know right now i agree like supporting gpus could be very valuable um currently yeah i i don't know of ongoing uh work with that i yeah i agree that'd be great to have sure sure have a question about architecture of interoperation so hips data is a company which makes a data frame and they have deploying implementation and they try to integrate a spark so you can basically use some other implementations and they say you have a good implementation already which is proven and production noted or implemented in the maldives you can delegate to that how do you think a male leaf would accumulate other reliabilities like mahout next date the other plants to essentially be an ensemble library or do you want to implement all algorithms right yes so they're definitely uh the question was yeah is i guess about sort of plugability and in spark and ml lib and yeah there definitely are ongoing efforts for that um right now if you wanted to say plug you know mahout into ml lib um i don't think that there's the infrastructure to do that you know in a one-liner um but yeah there are definitely it's a work in progress okay uh you mentioned that you're looking for computers it seems like we have big dictionary folks how can i mean the steps are basically um via uh sorry uh the jiras and via github where discussion uh bug reports features can be posted via jira and then you know if you have an implementation of a new feature or a fix or a test or something like that can be submitted via github and a pull request in terms of like getting started and like why would you do it i i think the main reason would be if you know a particular part of ml lib is useful to you personally and you need a feature it's not there it would be valuable or there's a issue with how something runs it could be faster and that would be like a great place to start if you're just looking for like starter things there are a number of items listed on jira which would be you know good to take a look at thanks i go into the is disability foreign forest and contribute the second question is the database cloud that you showed us to look at statistics from models that open to right so two questions first was um can you contribute to jira's uh for random forests and there there's sort of like one jira for random forests there uh basically to get the implementation in they're discussing it um would definitely be valuable although right now there's a pull request on github so actually like taking a look at the api and uh basically the functionality there uh would be a good start because that's sort of immediately going to get uh to spark once it's ready um after that i think there will be a lot of optimizations which could be done and there yeah jira would be a good place to start discussing so the other question databricks cloud can you look at it and model statistics and the like so right now databricks cloud um is sort of not generally available available for immediate use but you can definitely check out information about it on the databricks website and also sign up uh for beta testing and we're definitely hoping to ramp this up and it would be great to sign up and find more information and as it is um you know ready for public release then be definitely be notified oh sorry don't let me mistake um so i'm not involved with the yeah getting in beta testers um i know that you can sign up for beta testing on the website and i think this is basically um you know a good way to like get on the list and um depending on your needs and uh interactions yeah uh i but yeah definitely ongoing okay i tried implementing it all last night but wasn't able to okay are there other questions right so the question was like can you go beyond simple mapreduce and um i'd say roughly yes and that you can submit a job uh have it be start running submit another job so on and um for an a beginner user though i would say that it is good to frame it in terms of some of these familiar operations because there are is a lot of infrastructure in terms of just like ap friend very friendly apis uh for example for doing tree aggregate um but definitely as you get more advanced for example the graph processing uh library sort of lets you think of data processing on a graph rather than uh sort of this very flat mapreduce but other questions okay let's take the speaker thank you