sfspark.org: Berthold Reinwald, Apache SystemML
Recording: sfspark.org: Berthold Reinwald, Apache SystemML
you thanks for inviting us thanks to the SEC for hosting this program providing system and union so I'm going to talk about scalable machine learning with a purchase system ml show of hands how many have heard of system ml before it's about the other you don't count it's about ten percent let's say so it's small number all right so when I drove up here this afternoon my team kind of told me okay so matrix how my presentation will be measured for tomorrow is how many people will go to github and star system mo okay so i will take you to the URL either in the course of the presentation if you get really bored of the presentation or tonight when you go home or tomorrow morning first thing when you come in the office go to the github system and love and push the start button okay promise otherwise cut them off the pizza you know so honestly what is this the memo so in a nutshell system and model it gives you a language this language is not a programming language it's a language for the data scientist to implement machine learning algorithms okay machine learning algorithms that you can run on on a small iris data set or you run it on 10 billion rows and n 10 thousand columns okay and system ml we built a compiler okay with a cost-based optimizer that looks at whatever machine learning algorithm I after implemented okay we look at the data characteristics that you want to run it over okay as well as if you give us a single node or a cluster like a spark cluster to run on we take those three factors in and generate an optimal execution plan okay that runs the machine learning algorithm for you that's it okay we have different execution modes for a system mo it can run embedded as a library it can run as a standalone java application in in a single JVM in which case obviously you will not get a scale-out execution end what it runs in the cluster that we call hybrid okay hybrid means that you know if you look at their machine learning algorithm you have a whole bunch of operations in there some of them are tiny operations you don't want to run on the cluster it's overkill but so large operations you were to form out to to the cluster that's what we call hybrid okay we also have a whole bunch of API is in there so you can use it from Java you can use it from scholar you can use it from Python and I will take you through those examples so we're system a mode today we started with system and more project a few years ago last November officially it became an Apache Incubator project ok so the URL is right there and even before we put it out there be incorporated into a IBM big data product called big insights it's part of that it's part of pig insights we have several modules in there one of them being a data scientist module and that they designed this module has system and mold in it and of course it's also an ongoing research project at IBM almaden research center and we are connected to the products team you're connected to the SEC team we are connected to open source so we love all of you all right did everyone copy down to URL it's very simple system a melted Apache torque I'm going to check on the way out ok it's going to be a quiz so if you go to this gyro okay that's our website ok now i cannot see it here it has a brief description on it there's a couple of your important points there I don't know here's the github okay that's where our old code is you just go there the start button is right there okay just making sure that you guys see it so going download the code we spent initial system I'm al was a research project and no research is research we do a lot of things here the b2 a really lousy job in documenting it but since it became open source project a documentation it's really fantastic i really encourage you go there a lot of the presentation that i'm i'm going to show here is actually you know just on this documentation website here you see all the different modes how to run system ml yes links to the to the documentation of our language on how to implement machine learning algorithms and i cannot really see much from this angle it's an open source project okay i encourage you to go to community which one is his first owner the second one I cannot see it though somewhere there is an issue tracker in there which takes you to to the chi-rho server go and check it out there's a lot of work to find in there pick some implement it commits back to open source okay so why should you care there's a lot of reasons why you should care so so as I said before they've been looking at been working on system fo for for quite a few years now and it's not that on day one we said okay let's do this in a mo that's not how things happen so we talked to a lot of IBM customers we paid attention to the industry and we summarized all those kind of use cases on this one slide to some degree and you know I hope you can see it's not a single vertical industry that wants to do big data and provides use cases it really comes from all kinds of different industries ok and the problem definitions are very different ok but you have to look at a large number of thoughts and then you know try to summarize them and come up with a smallish number which is what we did here that you want to support ok just to mention a couple of examples like the first one from the insurance company they just came to us and said we want to try out your platform ok and how how easy is it for a data scientist or a developer to do a parallel model building ok so if they just said no here's in fact they came with an R script to us said now here's an our grid linear regression ok do it a hundred thousand times ok how do you paralyze it and we did that ok sistema mode can do that for you we have construct both into system immortal do that for you another one was an automotive company it did not have anything to do with emission control so don't worry about that but they came to us and said now we have so much data ok but we have a problem is a customer satisfaction ok so we built a classifier for them we started off with a few hundred feature bi deposit first then we realized some more features we added and eventually we ended up with 22,000 features the better accuracy and better model we were able to create ok can your system today around 22,000 feature variables logistic regression on it download system mo I showed you is about now ok another one air transportation you go to you know san francisco airport you can gratify the entire Airport into like three by three kind of grids and you connect to the Wi-Fi system they're also the routers they can kind of triangulate your location there okay we recorded into timestamp location and the mac address okay and then you can actually analyze the data and maybe pretty you know develop an arena model to predict how many MAC addresses will be at this location 12 hours 24 hours or six months from now okay depending on what accuracy you will develop and yet several other use cases they're from financial industry a lot of them bear about you know bivariate statistics to do correlation analysis yet from retail banking again about bivariate statistics mostly and other services companies we are approached by by a railroad company okay and they said we have 7,000 variables okay can you do PCA on it okay what an insurance company came you know you give you ten temple in rows okay can you do a generalized linear model on it okay so it's a problem spectrum it's very very different okay eventually be summarized at all okay and we said so big data analytics use case you know what is it really about we don't have a textbook definition here but when people talk about it to say well we want to do large number of model building ok imperil okay so things are embarrassing paralyzed able but you have to provide it infrastructure for it okay obvious points are a large number of rows okay they have a thousand rows today and then they deployed and the company grows and whatever eventually save temple in rows ok number of features okay especially if you're going to text applications very quickly you go up to know tens or hundreds of thousands of feature variables all right and you need to be able to run sparse data large data is sparse data okay your system has to support sparse data right large number of intermediates that's very often with the case that we encounter large number a large number of peers so that comes into the picture if of large number of feature variables and they want to do bivariate you know correlate analysis then you have to embarrassing polarizable now also spear that he wants to do the correlation analysis very quickly on it and very often be we said okay now here's an algorithm okay intense and said more but you know it doesn't really exactly do what we wanted to do how do you customize it so in you know probably eighty percent of all those cases like a textbook machine learning algorithms it did not quite cut it but you had to go in and modify it for that specific use case alright so let's get to work I'm not going to use one of those use cases i'm going to use a really old slide okay how can you tell that it's an old slide just checking whether people talk about Wow amazing okay it goes that a few years back but it still holds okay the application still holds which one yes they States to this really good Isaac so the example is the following okay you have twitter data you go and downloaded it you put each tweet in a bag of words kind of model you build a very large matrix up there on the the column level you have tokens on the vertically you have to treat okay you build ups is a document token kind of matrix now you want to find out you know what are people talking about in social media okay maybe you do a product lunch okay and you want to capture so sweet and figure out is it good is it bad what do they like would they not like or whatever so it turns into what people call a matrix factorization problem it takes this very large matrix okay and break it down into two lower ranked matrices which is the document topic matrix as well as a topic word kind of matrix oh it's causing wh I'm going to refer back to those variables so stay with me so and there's a lot of techniques on how to do our matrix factorization and but the point is really that the matrix is hopefully extremely large okay now if you go now okay and implement a matrix factorization algorithm and in this case you have you decided to go non-negative matrix factorization specifically the GN MF the Gaussian non-negative matrix factorization there's a lot of implementations for it ok the chopper implementation is not a spark implementation it's a MapReduce implementation ok but it's still can be used to mate to get to get the point across so Jean MF it's an old algorithm introduced by some AT&T folks about two thousand one or so okay and it's better a very nice paper about it in the nature magazine and the way they described is algorithm if you open up the paper it's really in terms of linear algebra kind of expressions okay they defined and spend a lot of time defining what they calls us to update rules right you you have just a large fee that he started off with okay and then you have some random initialization with W and H and then it's an iterative algorithm in the iterations you update agent W until you you know reach maximum number of iterations where you have some convergence criteria that you're happy with okay if you implement that one in the low level of programming language against MapReduce it may be scholars spark instead of 1500 it it's a thousand but still not in the same spirit as what the paper was written in in linear algebra using system ml and that's our implementation of tnm f in 10 lines of code that is high level you can express that entire algorithm ok now this paper came out in 201 and how many people of you are really data scientists right so there's quite a few numbers in here don't be shy so I yes so it's it's so that's our language ok and it has a lot of linear algebra linear algebra primitives in it and I will take you through it so it's a lot of like so our language is modeled after RF language so it's an order like syntax o % star % is matrix multiplication transpose w matrix multiply V cell vice division so and so forth okay that's really how the paper is actually written okay now as a data sign that you implement that you run it you look at the results you're unhappy the paper came out in 2001 and pretty much every other year some other researchers published another paper which improved on that one by you know modifying those those update rules you know by adding some regularization by adding some scaling by adding whatever okay now if you implemented your algorithm that way it's very easy to modify it okay a data scientist should be extremely happy if you did it in Java or in Scala it's his low level you may have to revisit a lot of code okay yes yes sure but you know a data scientist oh so there's a small number of data scientists if hopefully you're on my side okay would you really drop down to a choppa language to implement your machine learning algorithm or would you really start you know stay with let's say Python what are you know like a specialized language kind of thing scientists like caleb is not capable I would not I would never say that no no I don't want to say that but I think they prefer a higher-level language okay yeah I know you still want to have souls capabilities in there right your IDs are always wonderful no doubt about it but it's it's a different level of abstraction where data scientist thing yes so doing it that that level it gives you some benefits to in terms of performance that people talk about as well okay yes or does not scale okay but it isn't all like syntax right yeah I'm sorry sitting in it this syntax is a subset of our okay it's our own parser okay it's our own compiler at compile time and at runtime we don't put in any order no we cannot is that why I call it an are like syntax that was our goal exactly so the onboarding off it should be very straightforward yes yes yeah you can it and in fact we know in our in our test fit we take our snippets here okay and to do some result and you know correctness verifications we just run the same snippets through the or interpret hopefully it's the right thing and if as long as the results match we are happy so yes the questions so either in overhauled Alaska to review but if it's a question okay or we'll have a mic for the audience and we'll need to serialize the questions sure works best too many options okay I'll take it over okay I need to respond to to this one comment that was you can just write your article and put it into system mo that is not entirely true because our is such a beast okay that's been around for 30 years with 5,000 packages that we cannot run okay but if it's like our base okay and you know in terms of I oh there's some differences here the majority of it should work but it's typically not the case minor modifications are necessary sorry yes sir mls language would all work in correct subset except for minor read write operations okay and some other extensions that we put in but i can point us out as we go through the language guide yes with sparkle I I will get to that I'll keep it in mind yes system are goes back to li j i think it's a 70s 40 years ago which was the first relational database system right and which is very different from the our statistical package very good well actually the analogy is very appropriate and I actually should have brought up system are because system are it's a relational database system and it's like afterwards they started a sequel effort on it okay which was a big big push in the relational database industry and it's been around for 40 years and created a multi multi multi-billion dollar market okay my higher level goal here is for system ml in this language to becomes a sequel for machine learning okay because we see a big gap there okay any other questions the only on slide two or three okay now let's put it to work you're implemented this Gaussian non-negative matrix factorization that scales to tens of billions of tweets and now and in the vocab really goes up to like a hundred thousand columns or so okay now you want to run it ok so this slider should look familiar to some of you who does it look familiar to thank you very much at least two Wow three all right so similar slides are shown in for from spark community right to show really likes a true benefits of spark providing a unified programming environment in the program model which is wonderful okay that's why we decided to go with spark as well because it has no like you can suck in your tweets from take a hose okay you can use sparks equal to do some data exploration on it okay and then eventually you to the future ization of the tweets and putting into tokens and creating this document with matrix of it okay and then you get ready to to to call system ml okay and I will take you through those details very shortly on how to call from Scala that's a scholar program here system ml and eventually invokes his chin MF 30ml script here ok so here with the spokes equal you prepare a data frame ok and then it takes this data frame and registered as an input into your gmail script okay and you also wretch is the output which are is a WND the H variables and once you ran it okay you can go and get the data frames back w + h after the execution and do whatever you need to do with it okay so that's really like an end-to-end flow with system ml in it all right okay now what is really system ml so you've seen this one example gnm FTM oh so sis FML it allows you to implement machine learning algorithms and then expressed in this declarative high-level language with an o like syntax that we talked about at length already okay actually just to give you may be a little bit more flavor off the language here okay so here's our language guide for the next three hours we're gonna do a tutorial on the language guide so get ready for it oh the heads goes up alright so don't worry I just want to show off a couple of things or you have data types in it it's very limited so there's room for growth there but we have scalar data types and we have a matrix data type okay and then we have value types integer double boolean and string okay you have obviously expressions so there's a lot of linear algebra expressions in the language in there damir statement ok we have control structures in there like so typically loops as well as a new loop construct that we caused a powerful loop that allows you to do parallel model building as well as other you know each iteration assuming that there are no loop carrots dependencies from one iteration to the next iteration it makes it embarrassingly paralyzed able which is really wonderful and the primary use courage from put our model building for you know large correlation analysis and so forth yes sir so the array primitive is a busy is that in dimensions or two-dimensional yes though we don't have any repeats equation yes so the question was whether we have only two dimensional arrays and the eons is yes right now but you know eventually I think we even might have a cheer to have like a tensor kind of thing in there ok so for deep learning and things like that those things will become important right we have functioned here we have a variable scoping in there we have command line parameters in there so from the command line you can just pass in similar to like see in C++ and the auction so forth you can passing parameters and so forth and then we have a whole bunch of built-in functions there too struct matrices to to do matrix indexing you know left in right indexing left in the range indexing column row whatever slicing it Ising you need to do ok nope nope you don't have yes so this one here is our order like syntax ok but the same language concepts we have a Python flavor of that syntax as well that we call pie dmo and that one is always on the website yes sir so one comment I heard earlier from from our programmer like the gmf example that I saw you know that should be a no-brainer for our programmer to write ok for Python developer and python is very popular from you know kind of advanced analytics as well it's kind of a pain ok so okay you also have a no api so that that's our language definition that's just another example there and then we have to find a pis and the IP is our to you know take those team our scripts as you seen in the end to end example and just invoke them from scala and from python and from chora and from our kind of the whole algorithm and passing variables parsing data frames if necessary inc and get the results back okay that is we feel is extremely important for deployments right because you can develop your algorithm and then you can use it in building solutions in building tools or whatever people need to do okay the core of Systema Melton is the compiler okay and I will take you through some details of the capabilities of the compiler but the gist of it is really a cost-based optimizer okay cost-based optimizer that means so if you have matrix multiplication in here like so % r % and depending on whether X is a small matrix or large matrix or whether it's a tall matrix or a skinny matrix or it's short and wide orbit its dense or sparse you will do a different implementation of matrix multiplication okay and the cost-based optimizer understand sir characteristics understands the classic data characteristics across the characteristics and chooses the right runtime operator to get the job done yes sir we compile to our own internal operators okay we do not generate code hour runtime is implemented in Java yes okay it goes through two layers of the eggs and I will show you what those are Thank You miser is working at runtime it works so we call it a compiler but it's something in between a compiler and an interpreter okay so i like to call it a compiler because interpreter would assume that you do like what r does line by line compilation we don't like that okay because some more code you give us to look at the better of an optimization we can do all right such as dead code elimination rewriting you know there's a whole bunch of things that you are able to perform and i will show a couple of examples ok so we compiled down to our physical operators ok i couldn't show you a couple of those as well and then our goal was really you know performance as well as scale out so you want to run on single JVM as well as to you know a thousand owed sparkle cluster ok so that's a very high level picture of the architecture in system mo so we have our DML script ok you've seen a couple of examples already it goes through the entire language layer to take care of that project that may have our optic impaler with the optimized in it where we have all the tags and those optimizations the rewrites as well as a code generation and here we have our runtime the picture is unfortunately a little misleading ok so let me explain what we mean so we started off doing system ml on a produced ok that's what we use is so we generate our runtime was hosted in MapReduce jobs in the mappers interpret users ok but we also had always like a single node implementation because each one of the scripts it has like scalar bridge there is no need to go to map reduce or to spark just do it in a single JVM so we always compile down to what we call hybrid execution plans small operations single JVM which is typically the driver program anyways that you always have around and the large operation you just form out to the cluster yes heavily oh the question box better be take advantage of vectorized operations yes and no ok we have prototypes to to use a cheap you as a back-end but we have not put that one in open source yet but it's on our list to really polish it and put it there because there is definitely benefits there in terms of performance but even sorry I misunderstood your question first people when they first right side email program says they use our language as if they implement Java code which is a really bad thing to do though you can you can multiply two vectors ok and write it up nicely in a for loop ok don't do that just use our high-level operators if you write a follow-up you will get the performance of full loop okay just a bit clearer and the same rules apply for our so if you open any our textbook first page vectorize your operations ok ok so you have thought on and so that our hybrid execution plans and then they don't be we re paste our quantum operators to run against the sparco api yes so that's a very good point we had some the question is is our compiler able to identify badly written gmail code that's what you mean I assume right so we have some thoughts about it and there's a lot of badly written code around so we have things like dead code elimination like you know you do a calculation but you never use it for anything okay then then we just do this a dead code elimination we do things like that we also think in terms of like code motion meaning you have some code that you only need to do once but you for better for whatever reason it ends up in a while loop so we would like to move it out so things like that are possible yes I cannot bash my customers because system ml is meant for data scientists data scientists are great okay so okay if we go to the github don't forget the stop button but you will also see a list of algorithms there so that we put out okay and we try to show off the breadth of our language okay so we have a team a descriptor for descriptive statistics very extensively univariate it by very statistics we do classification so we have multinomial logistic regression multi-class experience naive bayes the decision trees and random forest we have K means clustering we cover the regression linear regression with several versions of you have quite extensive implementation of cheer lambs with many many different distribution functions and Link functions stepwise regression we have in them we have two versions of pca that is not shown here you have made the x-factor ization tnm f obviously we didn't put it on slides but i think you know how to write it now as well as ALS we have in there and you have to survival models there those are all for model building in addition to that we have the matching scoring script as well ok so you produce a model and then you shift it over to squirting for some of the models we even created p.m. ml representations officer models ok but not all of them and we also added transformation capabilities in a very limited way for no data scientists where they need to do like recoding of values it's known as factors in our we have dummy coding that you can easily specify pinning scaling as well as missing value imputation splines be half a spline implementation yes I think it's a ladder I think it's the latter the question was whether we gotta prove from the PM ml consortium I believe so but i'm not a PM ml expert so i think we just took publicly available specs and serialized it that way and there was one chava library that we used in order to serialize it out but i forget the name of that library now it's an open source library neural networks we do not have yet you are probably able to express it in TMO but it's it's a little bit bone probably that way and it may not give you the performance that you would expect until we have like a cheap you back end yes is it fairly easy to express one in it we have a grid search like framework so we have 4 hype a parameter tuning we did some simple examples here but it's not very extensive it's pretty much you know you're in charge of enumerating it and you put it in a powerful loop and a for loop and then you iterate over it but no specific support okay all right oh so there was a count of like five data scientists from now on everyone is a data scientist okay everyone gets a homework you have to sit down and write a DML script okay it's for linear regression so it's manageable so you have a like your dataset X and you have your label y ok and you need to come up with a model w okay you have this optimization formulation there and let's say you decide on a conjugate gradient algorithm okay I just want to take you through one examples just to demonstrate that sells gnm f or linear regression scrip days they don't fall out of the sky magically ok so it's really some magic behind it ok so here's an example on how this linear regression conjugate gradient could look like ok so you do a really cool initialization of your coefficients here you have a really bad model it's all 0 ok afterwards ok a little laggy you you come up with an initial direction of K where you want to go to and you compute that direction doing a transpose of X matrix multiplied with y ok and turn it to the negative of the neck that's a initial direction okay then you compute your convergence criteria okay by using one for instance and then it's an iterative algorithm you iterated over until you reach a maximum number of iterations or your convergence criterion hold okay in each iteration you compute a step size here okay using matrix vector and matrix matrix multiplications okay then you to update you you use the step size okay and performs the step which means you your update w okay after that you compute your next direction okay and you do that you know until you're done okay so that's a very simple linear regression conjugate gradient method okay after the while loop you have your repeaters your w okay and then you can do some some you know accuracy measures that we compute like your residuals and those kind of things or whatever you please to do and you're done okay so that's your linear regression algorithm okay now you want to deploy it you want to use it okay you're very happy with the implementation this MMO it gives you different ways of invoking it okay you can just invoke it as a java application by pointing it at your tmail script and you provide some input parameters for it so you point that you're all our data comes out of HDFS okay you pointed your x matrix you point at the y matrix and and the betas or the coefficients you want to write into the result which we call be okay so just remember that is just a little shell script here under the covers it's just inbox chava okay or the same script okay if you have a scale out if you have cluster readily we had to go you can use our Sparky amount of JavaScript which under the covers just invoke Spock submit okay and the same parameters here you just fit into spark submit and then it runs it on the cluster okay or if you want to run it on MapReduce okay then you can just among system and all as a child out in the loop and again it's the same list of parameters ok so those are essentially two different methods IBM also and our big data product we have an r package they're called big are okay that you can run in your through our instance okay and that one can connect to a dude Custer okay and you can just invoke so script as well but you know similar things should be possible in the future through Sparkle as well okay in terms of AP is with ml context you have seen cleavage examples here you can just invoke it through scholar okay that you can just run from a spark shell if you're a fan of that or from PI spark okay or just from a child I application if you if you feel like doing that no exactly it all goes the same compiler to the same optimization and then it should just exactly vogue the same plan yes okay here's another example if you want to invoke it from from the scholar shell here's your tmail script you just use our ml context that allows you to plug know the system member or compiler into scholar here you register your input you x and y and you register your output you can specify the command line parameters here and then you just invoke execute and at the end you get your data frame pack and you can use whatever scholar tools are available to look at it this this one is scholar video we do have functions yes okay if you want to wrap your TMO algorithm into an ml pipeline for some of the algorithms one of them being logistic regression and we are in the process of adding more you can also invoke it through the ML pipeline so here you would kind of input your wrapper class for the loot logistic regression da Mao script and then you can define a spark M our pipeline here by setting those three stages or tokenizer hashing as well as a logistic regression with your instantiate here and then you can just invoke pipeline that fit and you can execute the pipeline that should not be any news to many of you it's just to show off how nicely system ml fits into this entire spark ecosystem okay you guys want to look a little bit behind the curtain what's really happening in system Ella is everyone excited or kind of getting tired no more questions or what I think everyone is impressed okay so we talked about the API is quite extensively so the command line there is a ml context there's a pipeline you always have chain LC think of Jim I'll see as using our compiler and you embed it as a library in whatever your building okay so we have defined a Java API and think of it does everyone know jdbc okay jml see is the machine learning equivalent to jdbc okay so you can take your script okay you can actually compile it which means you you prepare it okay and then you again similar to the ML context you bind the variables to it as input and output and then you can just execute it many many times without all wendover recompiling it that's what KML see is that is extremely important it's less important for scoring because that is heavy lifting anyways but it's extremely important for less important for for training but more important for scoring so I ok that picture you have seen so we have more less so we have a pause in there and they may have two kinds of tags okay so so we have hops tags a high-level representation of all the operations that we have in a TMO script and we have a small number less than 10 kinds of hops okay we have like unary operators we have course the input and output like a read and write kind of thing is then you need operators we have cell wise binary operators that's a hop then we have like four binary aggregate which is typically matrix multiplications that's a kind of hop you have some hops that don't touch data but change the layout of it so transpose is one kind of this hard or well reshape that's another one of them okay those are independent of whatever back-end you choose okay once you decide on the back end meaning you tell us or run although MapReduce were rounded on chava only or run it on on on spark okay then we choose low-level operators okay and those are geared towards the back end you want to run against okay hour runtime is structured in the following way so typically you have a control program okay the output of the lots of compilation is is a sequence of instructions that you want to execute okay so let's get loaded into the control program okay and then you just happily execute those instructions some of those instructions are singing out instructions other instructions are what we call control program inserts those are single node or spark instructors or MapReduce instructions okay the spark instructions just go against the sparco API while the MapReduce instructions they kind of get executed in the map and reduce of MapReduce jobs okay the really cool thing about our run time architecture is this matrix block library so if you have very large matrices okay that it cannot hold in the senior JVM obviously we have a blog structured binary representation of souls and we have different kinds of blocks we have dense blocks sparse blogs even ultra sports prox because if a very large data and which is very sparse you know most of the cells are just 0 you know you want to not spend a lot of memory on those guys ok so our matrix block library actually is kind of a hybrid so for the distributor one we have like 1k by 1k kind of matrix blocks ok but the same matrix blocks can actually be deployed in Cena node where you have more memory right so but the operation of the same thing we just redefine our block size into whatever is necessary to hold in a signal JVM you also have a now control burma buffer pool they hold small data with an eviction policy against the local file system were just pulling the data from HDFS okay I'm going to skip that one here let's take you through an example on you know what those tags really look like ok so very quickly same slide as we've seen before just put a little bit more detail that I wanted to fill in so if tonight everyone goes and Donald's get up okay and goes through the source code repository you will be able to find a way around okay that's my goal so let's take sits one expression here you have x matrix multiply with that's a vector plus the scalar okay and then a cell wise multiplication with why we do not do line by line computation just to be clear but for illustration purposes i just take this one line okay so we have a parcel obviously and the we build something like an AST okay so we have some representation for it and then we break down the entire email program into what we call statement blocks those are mostly defined by either function vacations or control structures or things well a couple of other operations that we had to do it ok and then me to live variable analysis of which the Arabs go into the block and which one come out of it so you can do some clearing out of the buffer pool in things like that which are very important if you build a a compiler okay part of this language component is also what we call it a validate which does a semantic analysis so here we if the meta data is available then we are able to check dimensions to make sure that you know if you do a matrix multiplication of X with B plus s beef you need to make sure that the dimensions are compatible otherwise you know you just get the garbages results if your dimension line up once all of that is no accomplished then we generate a dag ok so the dark if you if look at it bottom up you read your be ok and you read your SP ok and you have two dimensions for it so that one is a 500 by one vector that one is just a scalar value ok then you do a binary plus on it's a binary plus feeds into this matrix multiply it with X the result of its feeding this is binary multiplication and you produce a cue okay the really cool thing here is those are our operators and we exactly understand number one the semantics obviously but also the implementation of it which means we can annotate those operators with memory estimates ok we know in order to perform this operation we need that much memory in order to do this large matrix multiplication you would need that much memory which accounts for the input ok or multiple inputs as well as an intermediate memory that you would need in order to implement our operate as well as for the output once you enter Tate also note with the memory estimates then you can actually do some operator selection which says well that one is a smaller operation I can just execute that on NCP or that one does not fit into my my driver program ok that when i'm going to let do by spark ok so with that simply heuristic there's many more happening in there by just try to keep it simple here we are able to generate a low-level a low-level dag ok with with the operators in it and they're very similar except for the decisions that we make in there like for instance this map multi are we going to do on spark ok once we have this lob tag ok then we just generate an instruction program for it and and get it executed in the control program yes we also have three rights in there ok which make it more complicated but semantically so you need to be critical you know if you add things or you you x so I've actually on the next slide I have a couple of those examples and so's operations people only execute on the hop stack ok so in that sense hop and log or isomorphic modest ok but we do rewrite at that level we don't do them here ok so I think we talked about most of those building blocks except for rewrites ok and we have two kinds of rewrite static rewrites and dynamic rewrites and I don't want to boy it too much here and just show you a couple of examples here like static rewrites some of them are really very simple but in touched by far not a complete list okay simple things like you know somebody writes down X plus X ok X plus X would tell you ok I need to do a binary addition of one input and the other input ok but our optic our compiler busy can do a little bit smarter like that ok and realize especially in a MapReduce we're in a spark setting you would release a I have to reheat X here and then I have to read X again ok so there's no need to do that ok so we just rewrite that one to two times X so just simply examples ok well x times X ok similar so you can just convert it to a power of it but you can also have some more much more like people write down transpose of trans plus of X you know obviously that's X don't do anything ok so rewrites are very easy to add to our system and you know we have some of them in there you know going to add more all the example are a little bit more involved like to simplify dag operation so here you have a matrix multiplication of x and y ok and afterwards you want to take the trace ok but you can rewrite that one and you can wear it it into this binary operation and asymptotic behavior matrix multiplication is an extremely expensive operation and being able to rewrite it into binary operations is a huge huge performance benefit that you will get ok other examples so sort of static rewrite why static because we perform those many have the dimension information available compile time now that is not always the case we also have what we call the dynamic rewrite and if you have an example like that one here so you have a matrix Y ok and you do matrix indexing on X a range of rows a droopy and see-through be ok if at execution time you realize that the dimensions of X and the dimensions why are the same ok get rid of this very expensive matrix indexing operation and just execute x equals y which is a no-op right and there's many many additional rewrites so maybe maybe one more we should take a look at I don't any favorites they're all really cool maybe that one here if you you know take the first column from why okay and assign it to X but if you realize that why only has one column in there ok just to execute of I so simple things like but now as you write down a large algorithm you there's a lot going on in the head you might not realize those obvious thing is so why not have to compile it those those things for you automatically so ah yes I actually skipped over that entirely so we have to think in here called recompile ok recompile means all compiler really shines if you have the statistics meaning we know the number of rows the degree of sparsity in the number of columns now for not for all operations you have that information available for instance if you do a contingency table ok it is data dependent so you don't know what comes out of the contingency table you don't know the number of rows at number of columns if we don't know we have to make a worst-case assumption and generate a very conservative plant that will not run out of memory ok which is all bad because very most of the cases contingency tables are small and less people do crazy things now what we do is we compute a contingency table after that we put kind of like a like a break there ok and kick off a dynamic recompile of the rest of the plan ok with the exact statistics and typically get a much better plan so that's what we mean by dynamic recompile okay so spark is great okay maybe rebased from mapreduce to spark give a whole bunch of things that really benefited system ml okay one of them being like this you know you have to or dd's and you can cash those are disease in memory okay but that does not always happen automatically so because they also have this notion called a lazy evaluation and that you know it's good and bad and never talk to that very shortly but for instance after the read and if you just read in a CSV file right so there's a lot of reblocking maybe recoding or whatever happening in there okay you only want to do that once ok so after all so also these very off me just to a checkpoint in there okay in order to make sure that you do sir parsing in the value creation only once okay and it's also extremely important to do that in front of the loops here for instance right because X is a read only variable which is always used in the body so you don't want to trigger over and over the computation of X again ok so we before the vial oops we create a checkpoint there okay to make sure that the RDD for X is cash the memory that's one of the big also spark is really great by allowing us to do this repartitioning there what that means is we can actually you know if you have a matrix end and a vector okay you can actually call align them and to make sure that you know you don't have to do a lot of shuffling there okay so we can actually inject what we call it a repartitioning out of why to make sure that the petition of Y and X line up in which case you can do like a lot of map site only operations as opposed to going through a shuttle so that's very important as well okay operated selection so we have a very simple heuristic in there we look at the driver memory whatever you give us is great if it's fits in there we perform at their otherwise be executed on on the spark close to those operations which sometimes may cause problems there so we also have the thing called the transitive spark execute type which means the following that if you have likes the sum of x times y okay x times y because x is is from large you do it on spark okay but the result of it is small okay so the sum of operation you could actually do in CP okay in which case you would actually have to take the result of x times y loaded into CP and then execute some operation but it's actually much more efficient to just push the sum although you could do it and see just push it onto the sparkle and piggyback it with the existing operations in there so that's what we mean by transitive physically operators we have no seven different ways to do matrix multiplication and the operator selection here takes care of which one depending on the data correct issues should be chosen we also have a whole bunch of fused physical operators there so those typically we put in into the system to get much better performance and that kind of comes through experience so we look at the large number of the mo script and if we kind of identify common patterns in them and then we go off and implement those fused operators fused operators means like weighted square loss or or some other operations it means that you want to exploit the sparsity okay and number one you want to avoid intermediate results okay so a bunch of operations such as like matrix chain optimization that's another example isn't the right you want to avoid those intermediates that word mean by fuse operators okay maybe very quickly our powerful optimizer so we have essentially three modes to execute a power for okay if if it all fits into CP then we just do a multi-threaded execution if it's too large okay but for the CP but you want to have let's say a higher degree of parallelism ten you want to maybe execute it on the cluster okay what I mean we both compiler locally powerful and then stand up our CP coded each one of the mappers were executors into the execution there if the code or the data does not fit into those executors because emily restrictions then we have to resort to a third kind of plan which because the local powerful with with jobs okay which is multi-threaded execution but it still executes a spark charts okay and there's a whole bunch more details maybe i'm just going to skip over those one important thing here which is really important if you want to run dimelo programs on small data just to create a spark context it may take up to 20 seconds okay but executing this the entire program might only take you know 500 milliseconds so you have to spend 20 seconds in road to do 20 min which just doesn't make sense so our compiler if possible it tries to detect whether there are any spark instructions in the program and all if not then we don't even create a spark context and just to us a signal to execution okay how much more time do I have Alex see here any time right too much tomorrow's I'm the only one standing in the corner man okay so maybe I'm going to skip that example here because it's a little involved the takeaway just in two or three sentences is falling you have one TMR script it's your favorite conjugate gradient linear regression example but it just executed on different data characteristics okay you're executed on eight gigabytes with ten columns or 80 gigabytes which is one hundred or a thousand or ten thousand columns the same program you compile against different data characteristics we create different execution plans okay and we get the best performance for each one of those data characteristics out of it okay and here's the proof so that takes about two minutes you can then via almost done alright so this is the same data characteristics eight gigabytes 8808 terabytes and that is a log scale here okay so why is that important the blue line is what we call you know control program and spark execution okay the point that I'm trying to convey is the blue line which is always the best line that's a system about performance okay running against a spark okay but then we also measured it against running exactly the same set up against the control program and then as opposed to spark jobs we create MapReduce jobs okay let's let the green or yellow and color blind and the red ones the middle one that's the spark so forget about the CP okay all the operations are always executed in a distributed mode which is spark okay so if you have a small data set eight gigabyte and you have drive a member of 20 gigabytes there's one important point to make it ok now I'm through the whole thing it's very important to avoid distributed operations if possible ok so pushing everything into spark is not a good idea doing singing of the execution benefits and the performative CP spark and CPM are is really the same because it is exactly similar on time no jobs are created at all so there's a 3.86 33.6 performance difference here which is quite significant ok now if you go to the next data set 80 gigabytes this is where spark really shines okay because that's Chloe's a sweet spot of it though this one here is system ml running some operations in control program and some of them in spark but compared to if you compare those two numbers right that one is using our DD caching that one is not using our DD cash that's already MapReduce and you have a big performance difference here okay but yet you get this performance difference here because the CP operations are still small its operation in ml program which you don't want to go against sproxil so CP ensberg gives you the best performance okay now if we go to eight hundred gigabytes eight hundred gigabytes is larger sends a distributed aggregated memory so spark does a really nice job trying to hold our d DS and spilling it if necessary okay but the performance gap between spark and MapReduce shrinks okay and if you go to like eight terabytes which is a data set to launch for your cluster anyways so everything is fully utilized okay it doesn't really matter it's the same plant that executes everywhere right yes that one is it's the same in that is just a you know different run with some variation in it it's the same run time that I could execute in one kb m that is multi-threaded multiple-course so the 16 course I'm gonna show with our answer to your question that number 21 and 24 think of it to be the same you first implemented on MapReduce replace it to sparkys so does it make sense to continue to mint and for now we do it's there going forward who knows more comparison would be to run the spark implementation let's a good point yes yes i would guess that is still some overhead there but it's probably minor yes that's good point the point the question was what is the difference between running in spark only as opposed to CP plus park yeah it's just to get a point across that the CP implementation is important that's that's all in reality you always want to run in that all the same I was the same bit on okay okay the point is cost based optimization is important ok I hope that is hybrid execution plans are important so don't do everything in spark but CP is important as well spark is great if the data fits into distributed aggregate memory but then you know as you grow it no spark does not fall flat on its ground but it actually gracefully know kind of degrades and it doesn't get worse in my videos ok which is a really good thing yes question with MapReduce I assume depending on your programming can run in several phases right Yes No multiple chops you might have to run exactly which is bad right so that's why you get that lousy performance here compared to their well here yeah because you know you run a MapReduce shop dump it out you come around in a loop oh I need to reread it back in to my little operation ok i'm done with the loop right it back out iterate so having the a memory or dd's for iterative algorithms is a must yes okay that's three questions question number one was how do we represent the matrices so we have a block structures the representation of the matrices the way we represent them in Sparky's we have Java parodies of a matrix block ID and as well as a binary block which is our own the representation okay upon we have some work on the way for compression as well yes yes and even if you have a large matrix some blocks can be sparse and other other blocks can be tense because typically it's not uniformly distributed yes good question so maybe that got lost a little bit so we need a little meta data okay and number of rows a number of columns is a minimum right because that allows you to validate as well right otherwise you do garbage matrix multiplications that don't line up so that's a minimum right and if you give us the sparsity as well that is great they have many more statistics yes yes yes but that is a different application right because now you get into indexing and that's like a point access ml it's mostly scans you look at all of it and I think there was a third parts as a blast library now we have our own matrix implementation but eventually especially for deep learning kind of things and hardware acceleration be we would resort to like khuda aur coo DNN or whatever is out there we have our own implementation that is cash concerts we try to we try to write the chava code in a way that it exploits the cache lines okay it's very conscious about it simdi operator be exploitive yes I forgot to be the item performance measurements against in tell em curl which is like number one on Intel at least and we are very close to it okay so they definitely are better because they a native implementation and we are trouble that's number one what is this but we are you're pretty good it so does kicks in and eventually it goes and creates assume the instructions yes yes Oh neither one so adding on to that question actually so what kind of so what can I java code are you writing and how are you verifying and it's actually using this innate Drummond's are you just timing it tiny you looking for someone no its timing okay okay okay sort of related but a little bit is how do you how do you profile the code on this to determine how much time is being spent in each operation you have some guidelines for profiling this to get the optimum performance out of it so we have a sofa individual java code we do a lot of like Chavo kind of profiling that's number one but the more important question is actually and we do have a problem therefore for profiling because now although spark is great it's a lazy relation kind of so we create our instructions right and you know we want to execute those instructions and the lazy evaluation if sometimes you does just don't know a bit under the covers it still readers some operation that is kind of very troubling for us to do proper profiling for it and we wished we had better too flament so one more question on these numbers if basically what that says is that the far end of the data table there is no German right right now it's eight terabytes on a thinking custom no it's fully utilized but fully you guys have cpu memory I uh how did you know that you couldn't cut back on the resources allocated in each of those configuration ah and what was the game what was the fusing so so a terabyte so typically like to just answer it the network now I think now I think in this case it was such a CPUs because it does a matrix multiplication that one is a cpu intensive case you but you can already hit those run times you needed that many cores enough yes yes it's probably fair to say that it paralyzes near near perfect and that each thread is only cereal every operations almost cereal yes yeah that should be I mean I'm not sure whether you really verified it but that's what I'm guessing I mean you're you're doing multiple tags yours what you're going for it here yes okay there's some lessons learned there no spark switching from MapReduce to spark it was a good move for us okay stateful distribute Akashic is extremely important for iterative machine learning algorithms in terms of memory efficiency going from MapReduce to spark you have to be much more careful to not screw up things okay because having the data in cash is great but it takes away from other memory lazy or deed evaluation is is your friend and your enemy okay it is your friend from for us it was our friend because we came from a producer and what had to do that we had to do at the lowest level a lot of piggybacking to piggyback a lot of our instructions on to a small number of MapReduce jobs okay that thing went away that entire component forces parking permit ish because spark does it for us so in that sense it's really a friend but for profiling and kind of things you have to be much more careful because you know you want to clean up some variables to free up stuff but if they serve some painting operation on it then you do the wrong things and things die okay and now for the clarity of Mo the Sol system Ahmad is really about declarative machine learning it was a really good proof point because we had our 20-plus machine learning or girls implemented okay and none of those scripts had to change okay all we had to do was switch out the back end and for our users nothing changed they got happier because it ran faster but no code had to be changed for them yes LaPook scholar park yeah like scalar park nope because so we looked at scalar park and things like that but they have their own restrictions okay as a typically perform very well okay but they're very particular in terms of data layout and as well as data placement you could probably do that and you know there is a whole bunch of libraries within IBM who actually do that for for powerpoints and so we have like an ESL library that goes in that direction we did not use it because tipperary those things they don't differentiate between tens and spice okay they only do it for four tens which is limited there are sparse implementation for cheap you see em for cheap use but I thought you were talking about clusters yes yes yes right sis officer for kukuda for instance for tens know is it's pretty good okay but what do you want the cpu tight OS yes I mean if somebody you know we could actually you know take our matrix block library okay and defines us ap is okay and and if people want to use it separately it should be possible it has a lot of fun yes we do yes we are open for business and if it go regenerate this set of challenges for the open source community yes you can read the summary slide that let's skip it so just to go with that so you know you can find on the spark you know after you push the start button you can go to the chi-rho server and actually look at our roadmap there so the roadmap include a whole bunch of things in terms of consumer ability so right now our API is they are very coarse in the sense of you can invoke the emmaus crypt okay but there is no fine grained language integration like like link you know that's always a very nice example there or creating some other dsl's hood tears are like the mahout scholars scholars or others that might be useful I'm know if people want to go that route you know doesn't have to be scholarly it could actually be Python as well python might actually be a better choice in Scala data scientists prefers python / scholar that's one aspect there's a whole bunch of work to do the items for for the compilot optimizer you know additional data types improved cost bonds our cost model for for the spark back and it's kind of limited right now it's more improvement they're fused operators are always great they give you a really fantastic performance data scientists are here so you go and implement your favorite machine learning algorithms in dmo and countries contribute them back BBF a couple of them in the works though canine behind the process to add and there's others that we want to add if you have your own favorite algorithm implemented in DML and put it out there okay time series is something that people should look at yes so like so I'm a strong believer in the claret if machines running because it really does his optimization under the covers for you okay it runs on on an iris dataset I keep it repeating myself up to 10 billion rows for more hood they don't have an optimized and okay and that will always you know that is one of the major contributions through system at all the question is whether that could work against streaming applications right so for training I mean that's kind of a little odd for scoring yes yes but for for for training typically you have you know multiple iterations / / training set right so it's kind of a confined thing but for scoring I you know definitely it should work right so so as your IOT events now kind of flow in okay you quickly want to square it to say you know this one is a hopper top you know if you like a classifier that work for those kind of applications are chained I will see with very low latency if it's very low overhead you know you composite email script only wanting to reuse it for many many spoons such should work okay any other questions