sfspark.org, Fred Reiss, Apache SystemML: State of the Project and Future Plans
Recording: sfspark.org, Fred Reiss, Apache SystemML: State of the Project and Future Plans
so hi i'm fred i'm the chief architect here at the spark technology center and tonight i'm going to be talking to you about what's new in the apache system ml project so a little bit about myself to start out I came here from IBM research where I worked for nine years prior to that I was a grad student at UC Berkeley working in databases and it's been a great time working at IBM and but it's really exciting to be working at the spark Technology Center now and the final thing is I am a system ml committer so I'm a member of this project and I'm going to talk about today so how many people here are familiar with the batchi system ml so some of the audience well for the rest of you let me answer the question that's probably for most at your mind right now which is a what is Apache system ml now the next slide I have a nice animation which is pre-canned and goes in exactly 30 seconds so okay here we go so in machine learning it's very easy to code up custom algorithms if you write them in a high-level language such as our Python but when you get that high-level length the high-level representation to run a scale you need to rewrite the algorithm and make it go on something like spark which requires rewriting algorithm takes a lot of time introduces errors so what the Apache system ml does here is it automates that process of going from the high-level representation of a custom or semi-custom machine learning algorithm directly to a fast result this is Apache research asari IVM research technology which is now inside an Apache project a patchy top-level project as of about two weeks ago so our websites at the bottom of this page system imelda Apache organ you can find that link again at the end of the talk now by the way I should note this is going to be a very compressed representation of what's going on in system ml because there is a lot going on but the slides will be available after the fact and we'll have some great links in the the talk if you want to find out more so that's the that's the elevator pitch for system ml is about taking high-level representations of algorithms and automatically adjusting them to the data characteristics to your cluster characteristics to make them scale a couple more details about system ml that we're going to need to know to understand some of the new stuff that I'm going to talk about today the way that you express algorithms to system analogy you use linear algebra and the way that you use linear algebra should be familiar to anyone in the audience who has familiar with coding up in R or PI sorry Python with numpy so here's a quick example of how you and how you would code up an algorithm I'm going to do linear regression from scratch in three lines of code so the least squares formulation of linear regression looks roughly like this if you write it down in terms of linear algebra you want to choose a vector W of coefficients that's going to minimize the difference between the product of your training data X and W subtract it out from your labels okay but minimize the l2 norm of that I should say which you can factor out into this second line okay if you take the gradient of that when that gradient is 0 this is going to be minimized which gives you this nice expression down here in linear algebra format a system of linear equations and you can if you're coding in R you can basically copy and paste this line of mathematics directly into your program and simplify it a bit to make it easier to read and you get three line script to do linear regression you can do the same thing in Python it might be a couple characters longer so this is a really really convenient way to take an algorithm out of a text book or a paper and quickly produce a prototype script and your prototype script will run great as long as the data is small but then the question is of course what happens when you try to run this over large data or over sparse data or over funny-shaped data well sometimes it works and sometimes it doesn't in the case of this particular example that first line up there is that the performance critical operation here X is your matrix of training data X is all of your training data in that first line we're left multiplying that matrix by its transpose so if you wrote down this code in R or in Python for numpy you would have to make it so that the entire training data fits in memory twice because you also need to store the transpose in dense format so like I said it works great in some cases but what effects is sparse well then you need to use switch over to using a sparse matrix library which adds some lines of code adds some complexity or worse what if the X is so large that even if you encoded in sparse format it's not going to fit onto one machine then you need to go back and just use this these three lines of code as a reference and re-implement the algorithm so this is kind of reimplemented algorithm changing the data representation these kinds of steps get very painful for data scientists who want to focus on the equations focus on the math and so this is the pain point the system ml addresses by automatically creating the parallel or sparse version of the program and adjusting that based on the properties of the data and the properties of the available hardware to execute the algorithm on so that is a very short introduction to the core of system ml I have a 30-minute version of this online on YouTube and I can also give a 90-minute version of this if any of you want me to come and talk it much greater length but I'm going to move on to new stuff and just before I do that let me bring up a couple points that are going to come up over and over again because the chief benefit of the system ml approaches is basically summarized here so because we're separating out that high level representation of the algorithm from the actual implementation and automating that transformation transformation based on characteristics of the data that means we can greatly simplify developing that algorithm you just write down the math it also means that we can change the algorithm you want to add some regularization some custom regularization to your linear regression you want to change the the loss function for factorizing your matrix you can do that by just changing one one equation you don't have to rewrite a comma parallel algorithm and 1/3 and this is going to be the dominant theme and a lot of the stuff towards the end of this talk as the system gets better as our optimizer gets better your algorithm doesn't need to change to take advantage of new functionalities so over time algorithms get faster and faster without modifying the code so keep that in mind it's going to come up later on ok so I think we've covered briefly what is Apache system ml obviously there's a lot more but I'll give you in just so I'll give you some pointers to find out more information later on but let's move on to the the meat of this talk which is what is new in Apache system ml so now I mentioned that we write algorithms down in system ml using high-level linear algebraic formulations it's just some quick representations of how you would actually write an algorithm down for some future reference on the next couple slides in system ml we have a domain-specific language we call that D ml when it has an R like syntax and there's another variant which is underneath the same language but has a Python like syntax that we call PI D ml here we have two implementations of Gaussian non-negative matrix factorization in the two dialects of the language the key thing to see here is this is a domain-specific language it has a whole bunch of linear algebra primitives it has a lot of useful built-in functions you can add new user defined functions to it but even though it looks like R or it looks like Python it's not our it's not Python it's a much narrower subset of functionality that our optimizer can tune and automatically paralyze so when you're you when you write an algorithm and using this language and you want to embed that algorithm in a larger pipeline or a larger application for scoring purposes you need to take that algorithm feed it through a system amount and get back a function that you can call from a host language like R or Python or a Scala or Java and we provide a series of api's to provide to give you exactly about functionality with system ml if you download it you'll get the Scala Java and Python API S and there's a new project called R for ML which is being incubated by the spark Technology Center that also provides in our front end to system ml now in this application interfacing space we have some new developments in the project the key one is a API that we call ml learn which allows you to take an algorithm a customer semi-custom algorithm implemented with system amount and wrap it up in an API that looks like scikit-learn or the related api from spark ml pipelines so we've reproduced in ml or we will with the next release pre-wrapped algorithms implemented in our domain specific language for some common regression and classification areas that you can take and modify or just use as example code to wrap your own custom algorithms and this supports since it's wrapped up in psychic learn compatible wrapper you can use all of the high perfer ammeter tuning and composition that you would normally do with a psychic learning algorithm here's a quick example of how pluggable it is this is the example code for doing linear regression from the scikit-learn website if we take this code and just modify two lines and of course include system ml's jar on your class path actually import system ml since this is Python then you can are then doing linear regression using a high level algorithm that's compiled and optimized by system ml and in fact if you have more data than fits on one machine system ml will run that algorithm spread across the cluster and just by changing two lines but you'll notice the parts in the bottom in red to actually fit a model to do prediction they don't change because it's the same API similar thing happens and when with ml learned if you want to use the ml pipeline's API that comes with SPARC here again we have the example code this time from the SPARC website for doing logistic regression with ml pipelines if we take this code and modify change to lines you can switch to a variant of logistic regression that is implemented in our high-low language DML and is optimized and customized for your data characteristics by sistema melon and like I said two lines everything else including how you fit the pipeline how you can compose these stages together it doesn't change so that's a very nice way to access the functionality system ml great way for I say a chief data scientist to package up an algorithm for more broad production use and in addition to allowing you to take these canned algorithms you can plug in a custom algorithm here and another thing that you can do with this API is you can plug in a machine learning a deep learning model and have system ml run that deep learning model for training or scoring purposes and the tool that we use to do that plugging in is called cafe 2d ml this is a new experimental tool still under active development but a reversion of this should be in the next release and it basically what it does is you run the cafe model file and the coefficients file through Cafe de ml along with some information about how you want to what do you want to do with this model do you want to train it do you want to score it if you're going to train it how are you going to train it and that generates the script in our language DML remember our language has lots of facilities for doing linear algebra and deep learning when you get down to it is just linear algebra so we can take very complex models up to the scale of like a ResNet and generate a script that will train that model and we run that through system and now as an added bonus the generated script will produce tensor tensor flow tensor board compatible output so you can actually use the tensor board tool to monitor its training progress and again it's a script written in terms of our high-level language so as our optimizer gets better that script gets faster the other thing is that script you might think since this generated code is going to be kind of incomprehensible but actually these scripts these generated scripts turn out to be quite easy to read because the code the cafe to demand DML generates in our high-level language two ml is in it makes use of a oh sorry here's an example before we move on of how you call that you just pass in your cafe files and then you get back essentially a pipeline stage that is the model so as I was saying this generated code that happens inside of here takes advantage of our NN library this is a library that was written primarily from by one of our developers here at the SPARC Technology Center that provides a whole lot of deep learning functionality entirely written in our high-level language DML because it's in that high-level language and we don't need to write provide like GPU implementations of individual operations we can add new functionality to this library really fast and this is what we've done in less than a year this is continuing to move forward quite rapidly you can see a link down there to the github for the core of that and this will be included in our next release so so here's some examples of creating a deep learning net using that library as you can see it's a lot like tensorflow except routines to build up the graphical structure and again since this is high level code underneath these functions are just expressions in linear algebra if you drill drill down to the source code since it's high level code this gets faster as we move as the system moves forward and in particular it's getting faster and faster as we add GPU support to our back-end so in recent builds of system ml we have been adding a GPU back-end to system and now nice a back-end because you see we have in system ml remember we take a high level representation of an algorithm we automatically turn that into a low-level plan that low-level execution plan runs on our runtime that runtime is actually a pluggable quantity so we have three different runtimes we have one for running on single node in SMP mode we have one for running just on SPARC and we want have a hybrid of the two that hybrid is our default mode where the optimizer chooses whether code executes on the master node or across the cluster in parallel so and we're using this pluggable API to add a new GPU accelerated back-end to system ml so that we can take advantage of GPUs in several different configurations either a single GPU on the single node execution multiple GPUs in one machine or GPUs spread across a spark cluster and the idea here again coming back to what I was saying earlier remember this slide from before is your code gets faster as the system improves so as we add more and more to our GPU back-end you basically GPU accelerate every piece of DML code that has been written without actually changing the code of the algorithm so here's a quick snapshot of our progress as you can see the single node single GPU version of this back-end is mostly complete at this point we're working on multi node multi-gpu and then in parallel also planning the fully distributed juice to use support but already with only a single GPU we get significant speed ups on the order of five to ten x4 deep learning use cases but also we're getting speed ups for non deep learning use cases because we just take this GPU acceleration and apply it to more traditional scripts so that that's a great example of as the system gets better your algorithms get better without needing to write a new line of code now in addition to putting offloading code to the GPU another thing that we're doing in support of deep learning but also to make older algorithms more traditional algorithms run fast is code generation aggressive code generation in our compiler this is an extension of something that we have already implemented in previous versions of system ml that we call fused operators now remember our linear regression example from before we have three lines of code that implements linear regression and a high-level language and we have that that key operation of multiplying the matrix of training data by its transpose if you do that in that operation in terms of the basic linear algebra primitives you'll need to transpose X and then multiply that transpose by the original matrix that's two operations they're both quite expensive actually if you combine those operations together you can compute that left that multiplied by left transpose much much faster than you could do the transpose followed by the multiply basically what you do is you take the dot products of all the combinations of columns in the matrix which is something you can do with very aggressive use of parallelism on a single machine and in a single parallel pass across multiple machines so fusing together operations into these complex kernels allows our optimizer to take the the intuitive representation of an expression and instead execute an equivalent but much faster version and this can make things actually asymptotically faster so here's an example of an expression that occurs a lot in equations that you'd be running when you're doing matrix factorization use cases this is best explained with the picture so imagine you have you're trying to factorize a matrix W which is a sparse matrix of your past observations into two factors U and s say for a recommender system use case well in the middle of the algorithm to do that factorization you'll probably be evaluating this expression what that expression does is it multiplies the two versions of the factors together to produce this u times s product and then multiplies that again multiplies that cell lies by your original values and then multiplies that again by one of the factors to get you an intermediate that can be used to compute the gradient of the other factor now the thing key thing to note here is typically this matrix of W your original weights from your observations is quite sparse but U and s the factors are typically dense so this matrix here is a very very large dense intermediate value typically much larger than the original training data and so if you compute this expression from the inner frame outward it's going to require a huge amount of memory and it's going to go very slow but what you can do is you can actually directly compute the output if you fuse all these operations together and that's exactly what we do insist in the mill the optimizer if it detects this kind of pattern will replace it by a single fused kernel that walks through the blocks of W computing the blocks of the output and this is asymptotically faster it's not just a constant amount it actually can be faster by an arbitrary amount depending on W sparsity now this is a very very powerful technique that can make algorithms run a thousand times faster but there is one caveat that the optimizer the component of system ml that's automatically parallelizing programs needs has only a fixed menu of these fused kernels and a fixed menu of transformations that will make a given expression match one of these kernels so if you have something that's not on the menu it's not going to get accelerated this way well to to remedy that and broaden the applicability of this technique what we have done is we now generate these fused kernels as Java code on the fly and so the way that we do that is we started with the directed acyclic graph of high-level operations this is our internal representation of a expression in system ml and we translate that down into an equivalent expression with some additional annotations that allow us to run some rules to transform the expression into a format with an order of operations amenable to code generation then we go and generate Java code on the fly to evaluate that expression multi-threaded or parallel and generate the same result but much much faster and once we've synthesized code we take we generate a new operator which implements the fused operation plug that back into the execution plan and move forward with compilation so with this generated approach if you look at individual equations how long it takes to evaluate the expression the value of expressions here here we have some little expressions at the top that we're actually evaluating but generated code which is the blue line here the blue bar here takes about this month is runs about as fast as our hand tuned handwritten primitives and when you fall back to the level of entire algorithms entire scripts we get similar amounts of speed ups in some cases and better speed-up in other cases so here we have the execution time for plus on non-negative matrix factorization this is one of these matrix factorization algorithms that has that expression I showed earlier or you have the large dense intermediate and so if you do not have operator fusion it takes a very long time to factorize a matrix if you've written down the algorithm and it's intuitive form but with used operators is much much faster as emphatically faster as I would say as I said because you're taking advantage of the sparsity of your input matrix with a generated code we have a small constant increase in running time because of the cost of actually compiling that generated Java code but you can see as the data size gets larger that increase becomes less and less significant here's our numbers from another algorithm computing l2s VMs and and you can see in this case actually at the larger data size our code generator is finding a new fused operator that we didn't have before that actually makes the algorithm run twice as fast so already we can see the benefits of having the optimizer generate new fused operators instead of choosing from a fixed menu so this is a really cool development it is in our mainline code it will be in the next release and there's actually a research paper insider this year and there should be another paper coming up soon which brings me to the next topic which is compressed linear algebra another topic that we've been writing papers and putting code into system ml for the basic idea here is that when we have a lot of scripts a lot of algorithms that we've written in our high-level language for not only so we can have a nice library of algorithms but also so we can stress test our system and while we were looking at across all these algorithms we noticed something that there's this very common pattern that occurs over and over again that have repeated read-only access to a large matrix of basically lightly transformed training data here's a concrete example this is linear regression by the conjugate gradient method it's an iterative algorithm it has this while loop on the outside it repeats until it converges and the performance critical part of gradient descent is typically computing the gradient in this case there's one input X which is your matrix of training data and everything else is of such a small size that it's basically insignificant performance wise so the key to the performance of this algorithm is making this line run fast which means doing repeated read-only access to X fed into these linear algebra operations and this pattern occurs in a lot of other algorithms over and over again so what we do here basically is we borrow some some insights and some techniques from the database world where compression is a well studied technique for making queries run faster with less memory since a lot of this data comes from a database or looks like it comes from a database your training data we can use a lot of techniques from database compression to make it smaller and by being careful in our choice of techniques and how we combine them together we can actually do operations linear algebra operations on the compressed data in the compressed domain which saves us from the chief drawback of what would happen if you tried to just compressed data with safe snappy which is that decompressing the data takes a lot longer than most of the operations you'd want to do over the data by operating the compressed domain we can make things very fast in fact end-to-end attend entire scripts you're looking up to 26 times improvements in end-to-end performance in our experiments so in micro benchmark results some operations when you compress the data they turn into metadata operations so here we have the example of X x squared basically squaring every individual value in a matrix if you compress that matrix in an encoding that separates out the distinct values in the matrix from the locations of those distinct values then while squaring every value is sum is a metadata operation you only need to touch the distinct values you don't need to touch those in their locations and you can do that much much faster here we have the time for that operation in the compressed in the uncompressed data and on data that's been compressed with our compression technique so you can operate on it in the compressed domain blue is the normalized time to do the operation over each of these machine learning data sets in the uncompressed domain in green is when you compress you'll notice though a certain lack of green bars that is because every green bar except for the first one has a size that rounds down to zero pixels if we switch to a log scale this is what it looks like so except for the Higgs data set which has a large block where we can't apply compression because of the way it's generated these other common machine learning data sets compress really well and we can square every single value in that data set three to four orders of magnitude faster than if we were doing it over a dense matrix or an appropriately sized sparse matrix for the sparse data so that's a really powerful result now obviously not every operation speeds up that much but even for operations that really really need to touch the entire matrix such as multiplying a matrix by a vector as you would if you were doing stochastic gradient descent we still can do those operations in the compressed domain in time comparable to how long they would run if they were over uncompressed data and as a result of all this we can take a script in our high-level language and execute it with the main input so we can fit like ten times as much data in memory or if we're reading it out of disk repeatedly read it from disk ten times as fast and when you take this and apply it to entire end-to-end scripts you get results like this so this is we took varying sizes of the amnesty data set which is a generated data set of handwritten digits and we ran the Algar the algorithm to do if I recall correctly support yes the trained support vector machines and we ran it with uncompressed data we ran it with compression turned on and the overall improvement in performance here was about five times and this is end to end this is an end to end result this includes the overhead of reading in the data uncompressed in both cases and when we're using compression the overhead of compressing the data on the fly so that's a very powerful result in some of our experiments we were getting for some algorithm and data combinations up to 26 times faster so yes there's question in the back the question was do we use caching of intermediate computations based on heuristics and I can give you a very long answer to that the short answer is yes and is and our system aggressively manages that cache data based on an analysis of the control flow in the script so very good question yes but I'm running short on time so I'm just going to wrap up here we can add some questions immediately after the talk so I talked about a couple of cool new things happening in the Apache system ml project number one we are the top level project Apache system ml now we have graduated from incubation we've made significant improvements to our API s for running high level scripts in our high-level language we've added support for deep learning both in terms of API is for running deep learning models or for training deep learning models and GPU support for training them faster and we've also got code generation which helps not only deep learning use cases but more traditional machine learning use cases and compressed linear algebra to make operations run order of magnitude nth and faster now if you download the latest version of system amount from our website which is 0.14 you'll get some of this functionality if you wait until this summer when we will be releasing system ml 1.0 you'll get all this stuff this would be our first release as a top-level Apache project you'll get the API improvements an improved version of deep learning code generation dynamic code generation of fused operators and compressed linear algebra is now turned on by default so that's a summary of the really exciting stuff going on right now in the Apache system ml project if you'd like to find out more information about this project our website is system ml Apache org and I have a couple of links here to more much more detailed information that you can download papers from these slides will be up online so you can get the links off of the slides and anyhow thank you very much for coming to my talk and I'll be happy to take any questions at this [Applause] so we have brave soul with a question I know I went very fast through that which can be a little intimidating but yes please try this one promising caller special I think it's a great high level way to introduce your business eyes Commissioner mr. Aris predominance of Pfizer y'all told you said Oakland is going to come in detail we've had we've been working with a select number of different groups inside and outside IBM to try out the system and everyone that we work with has been very helpful in tuning it the primary use at this point has been in IBM products but we're very eager to collaborate with outside adopters of the system and you can go to our website again and subscribe to the mailing list and see the traffic there and posts to the mailing list if you should happen to download our release and try it out I would encourage people to try out the project and contribute to it because we're really trying to build out that community additional questions if not [Applause] [Music]