scale.bythebay.io: Fred Reiss, Apache SystemML: State of the Project and Future Plans
Recording: scale.bythebay.io: Fred Reiss, Apache SystemML: State of the Project and Future Plans
so hello everyone my name is Fred and I'll be your presenter for today I am the chief architect at IBM's spark Technology Center down on 505 Howard Street and I've worked at IBM for a while I joined IBM from UC Berkeley in 2006 worked for IBM Research for nine years and since 2014 I've had my role at the SPARC Technology Center just in case any of you are wondering what's a spark Technology Center we're a open source lab at 505 Howard Street in the Watson West building we have a three-part mission we contribute to the Apache spark community we build core technology and make it more enterprise and cloud ready and we build data science skills and teach the enterprise about data science and what all that those three part missions generally break down to is writing a lot of code pushing a lot of changes in particular to Apache spark but also to related Apache projects like Apache arrow and the Jupiter ecosystem notebook ecosystem and as is relevant for this talk Apache system ml so let's talk for a moment about what is Apache system ml and here we're going to hit our first slide where there used to be an animation but I'll just talk you through what the animation was so in a nutshell Apache system ml is about going from the linear algebra formulation of an algorithm expressed in a high-level language like our or Python or with our latest API Scala with breeze like matrices you take that matrix and vector representation and instead of having to re-implement the algorithm to run on top of a system like Apache spark Apache system ml makes that algorithm run in parallel at scale by translating it into a low-level execution plan so before system ml if you have an algorithm writ that you've written in R in terms of matrices and vectors you have to rewrite it in terms of say rdd's or data frames and that means you write the algorithm twice which is expensive it's also error-prone because you write the algorithm twice with system ml we have this nice big blue box where the algorithm goes in one side the parallel execution plan that produces the result goes out the other and what's inside that big blue box is a lot of advanced technology that was originally IBM research technology that IBM donated to the Apache Software Foundation in 2014 to create Apache system ml which is now a full-fledged Apache top-level project so so I mentioned linear algebra a couple times in that exposition let me just take a brief digression into why linear algebra because linear algebra is really the the language of machine learning if you look at the paper representation of a machine learning algorithm it's almost always in terms of linear algebra even the new deep learning algorithms which are typically you think of them in terms of tensor algebra that turns into linear algebra as soon as you hit an actual implementation and here's a quick example of linear algebra in action in the context of kind of the most basic machine learning algorithm linear regression so if B if you're looking at linear regression with an l2 loss metric here's the formula at the top of the screen for your loss metric you want to solve for W your vector of coefficients that's going to minimize l2 loss the difference between your labels or your value your target for prediction and your your features multiplied by that W the vector of coefficients if you set the gradient of this equation to 0 gradient being the linear algebra equivalent of a derivative and solve for W you get this system of normal equations so we just need to invert this matrix and multiply it by X transpose Y and we get our solution and if you're writing in a high level language such as R or Python with numpy you can translate this equation directly into code or in a slightly more verbose form this kind of three lines of code so if linear regression and three lines of code and this will scale reasonably well up to that whatever fits on your laptop in particular these three lines of code if you just type to our will scale up to the point where your original matrix X of features fits into memory twice as a dense matrix and you can also fit that the result a in memory at the same time it works except when it doesn't though because if if your features are sparse all the sudden converting them to dense will make you run out of memory if if the features are just too big at the at the outset to fit into memory the whole thing is going to crash if you can't fit two copies of it it's going to crash so generally speaking when you take an algorithm like this it's super simple until the point where you need to scale and what system ml is all about is taking this super simple representation and making it scale by picking apart that algorithm looking at the pieces rearranging them constructing a parallel plan for executing those pieces and stitching them back together again giving you a result and key to this is being able to look at the properties of the data right because you this this even though this is just three lines of code you're going to run those three lines of code differently depending on what is the shape of X what if the sparsity of X what's the size of X and you need to generate different execution plans depending on those data properties so using data properties in a cost-based manner is another very important part of the system ml of algorithm so at a high level system ml is about taking that linear algebra representation of the algorithm and automatically generating an efficient execution plan and if you follow this approach you can simplify the development of new algorithms because you can code at a high level and not have to re-implement the algorithm again to make it scale this makes experimentation a lot easier and key to this talk is as system ml continues to move forward the same algorithm code gets faster and faster and faster we'll see an example of that at the end of this talk so we've got these three benefits faster reading code better in experimentation and faster faster faster so that's system ml in a very short nutshell but for this talk I'm going to focus not on what is a patchy system ml but what is new in Apache system ml and going to focus pretty much exclusively for the rest of this talk on stuff that we has been happening with in the last year year and a half and there are five things here on the list the first one I've already covered Apache system ml has graduated from incubation just this year we are now in a top-level Apache project as for the remaining four we have some significant improvements for our API s for expressing those algorithms we have a new technology called compressed linear algebra we're using code generation in new and exciting ways and we are starting to add some support for deep learning to the system let's move on to API improvements so to write down an algorithm in system ml you're going to use R or Python syntax most of the time so here we have an example of the subsets of R and Python that we support and this example is doing a matrix factorization use case in this case Gaussian non-negative matrix factorization which is an iterative algorithm you're iteratively solving for two factors H and W oh those names those letters come from the German names for the factors and those you multiply to those factors together they will approximate an original matrix which in this case is V and so you can see how you can just basically copy and paste equations into either of these syntaxes once you have your algorithm expressed as this you pass it through an API in a host language our Python Scala and our support of this host language is although our is going through a second third party project right now called alpha ml that we're working on folding into system ml so that's the those are the stable API it's for accessing system ml we've made some big improvements to the ml context which is kind of like a sequel context in JDBC that you use to pass in these algorithms to system ml but another cool thing that we've done which will be very interesting for any Scala people in the audience is we now have an experimental Scala DSL this is still something of a proof of concept we have a PR and a brain where this is implemented but it's not folded into mainline system ml but I have some lengths here which will be in the PDF in fact the PDF that I'm presenting right now that you can look to download offline or you can search for that work item number one three five three the basic idea here is again here we have matrix factorization in this case with a plus on noise model implemented in this spell of DSL and this is all Scala code that's going to be all native to your parent your larger Scala program we have this keyword up here system ml that's basically a function that takes into this argument a block of code and that's going to use a toolkit called Emma to walk the the parse tree walk the ast of that code and in generate our own internal representation of the program and then return a function that you can call later on or rather an object that you can call the run function on later on that will look at the runtime characteristics of the input data recompile that ast and generate an efficient parallel execution plan some really cool stuff done by a gentleman from Germany called Felix Schuler who was working at the SPARC technology center at the time now all these api's i've talked about have been for algorithm development and one thing one experience we've had with users of system ml is for every algorithm developer there's going to be five or ten people consuming that algorithm so with the recent releases of system ml we've worked a lot on new api is for consuming algorithms and in particular we have api's for compatibility with the pipeline model in psyche and SPARC so here's a quick code example of how this works on the left-hand side of the screen we have code that we've copied and pasted from a numpy example on the scikit-learn numpy based example on the psychic we're in learn website you don't need to read all those lines of code the key thing to note here is you change these two lines on the right and now this example is using system ml to do a linear regression we could do the same thing with spark you have a spark example from spark ml pipelines you change two lines of code and now you're using system ml to do a scalable logistic regression on top of the same spark installation that's doing all the data transformation of your pipeline so those are API improvements next on the list is a compressed linear algebra how are we doing for time by the way we're good okay this is deeper into the engine now we're getting into the core of system ml some of our newer optimizations compressed linear algebra is a concept that we have been working on for about two years now and the idea is we when we implemented a bunch of algorithms common algorithms for on top of system ml we noticed the pattern occurring over and over again in these iterative machine learning algorithms where you have repeated memory bound access to basically your lightly transformed training data as you iterate to try and find the best model here's an example linear regression same problem as we showed earlier but this time with the conjugate gradient method which is able to handle a larger variety of loss metrics in this case you're iterating over the input data X and constantly recomputing a new gradient so this pattern occurs over and over and over and over again and lots of different algorithms this repeated access to the large training data and making all of these algorithms scale is all about making that training data that you're going to be running over over and over again fit in memory it runs fastest if it fits in memory on your master node on one machine it runs almost as fast if you can fit it in aggregate cluster memory and as soon as you go out of aggregate cluster memory performance gets progressively worse you have to switch to an out of korah implementations so what we do in system ml is we use compression lossless compression to make that training data that we're generating over smaller so it's going to fit in memory of the master node for longer it's going to fit in memory for on the nodes of our spark cluster for longer and even when we get beyond this point where the compressed data fits in memory we can still stream it from disk a lot faster because it's compressed now if you just did this naively and gzipped your matrices it doesn't really work because the overhead of decompressing that data is more than the overhead of a lot of the operations you do directly over that data not only that but general-purpose compression techniques tend not to do well with data that came out of a database and a lot of data that you run machine learning on came out of a database so instead what we deal here is we borrow techniques from database compression and extend them with some new techniques that we've developed so we can perform we can compress the data very highly and we can do all these core linear algebra operations directly in the compressed domain so that gets us the much better compression and much better performance compared to just naively using gzip to the point where we can go much faster than running over uncompressed data so to give you an idea of how things can get faster with compression some operations that you would do on matrices and vectors are basically orders of magnitude asymptotic ly faster in compressed data because they turn into metadata operations here we have an example of the performance of squaring all the cells of a matrix very common tasks if you're computing a loss based on Squared's run over one two three four five six common machine learning data sets the green the blue bars are how long it takes to run over the uncompressed data the green bars are how long it takes to run when the data is compressed and you'll notice the absence of green bars that is because all the other green bars here round down to zero pixels high so you need to switch to a log scale to actually be able to see that difference and this is happening because if you're using a dictionary based compression technique to encode the data in the matrix squaring all the values in that matrix is just a matter of operating over the dictionary not the actual data as long as you have repeated values every repeated value can be squared at once now not every operation you would want to do in the context of a larger algorithm is going to have this property and of course because of anthos law whatever is the least accelerated operation is going to be the the gating factor here so we've gone through and done a comprehensive look at all the fundamental kernels we have in system ml and we have optimized all the important ones so that they run at scale on compressed data here's a particularly interesting one matrix vector multiplication so multiplying a major and matrix by a vector on the right this is a very challenging operation because it's varied a very data-intensive but as you can see we can get this running at comparable speed directly over compressed data compared with running it over the original uncompressed data when you take all these kernels and combine them together with our optimizer we can generate compression and execution plans that make end-to-end algorithms in this case training and l2 SVM run anywhere from five to twenty six times faster and this performance across by the way of a wide variety of data sizes this performance benefit again comes from the fact that we can fit the data in memory of the master node a lot longer we can fit the data in the memory of the worker nodes for longer when it's still spills out of the master node and when we go to and out of core implementation of the algorithm we're streaming data from SSD a lot faster because it's compressed and you'll note by the way that these numbers are including the overhead of compressing the data to make an apples-to-apples comparison we start both the compressed and uncompressed rounds with the data in uncompressed format on disk so that's compressed linear algebra in a nutshell the next new thing that we've added to the system ml project is code generation for the purpose of generating fused kernels so what's a fused kernel one thing I didn't mention when I talked about linear regression in three lines earlier on is that this this first line with that multiplying the matrix by its transpose on the left that's the key one for performance it's also one where the way you would do this naively if you just type that code into R is not at all optimal because you don't want to actually have to materialize that transpose if you're just going to throw it away it turns out you can compute the value of this expression in a single pass without materializing that intermediate result and system ml includes a fused kernel and will and use that fused kernel to make this operation go much much faster particularly in cases where X is stored across strength across multiple nodes and we have a number of these different patterns that were built into the system and the optimizer would pattern match against them here's a more complex example this pattern occurs a lot in matrix factorization use cases how where you have a matrix W that you are trying to approximate with the product of two factors U and s and in each iteration you are solving for one of the two factors in this case we are solving for u and at the important intermediate value to compute here is going to involve multiplying the existing factors together multiplying that cell wise by the original matrix you're trying to approximate and then multiplying again by the transpose of one of the factors if you did this naively you would be materializing this product here which is not a good idea because that is dense and W is sparse so this is typically orders of magnitude larger than any other data structure in this algorithm basically you're gonna fail even on trivial data sizes however you can do this entire operation in a single pass without materializing that intermediate if you have a fused kernel that does this entire expression in one pass basically you walk over the cells or the blocks of the output matrix computing those blocks and we pattern match in our optimizer on whenever we see this pattern we substitute in the kernel and that works quite well for a wide variety of algorithms the trouble is if you had like one additional additional variable included in this expression you can get out of the space that are pre-built kernel covers so what we're doing now in system ml is we dynamically generate these fused kernels the way that we do this is we start by analyzing the internal representation of that program that we have broken down into a directed acyclic graph that's a data flow graph we meant it with additional information for costing we generate potential options for what operators we're going to fuse into new fewest kernels use a cost model to determinate how fast those used kernels are going to run and how much memory they're going to require and then we make a choice and we generate new code from scratch for the fewest kernel compile that and link it back into the original dag of operators so some very cool stuff we have some more detailed technical documentation that I'll link to about exactly how this gets done but the upshot is if you include this in the system we can make end-to-end algorithms run in these numbers either just as fast or faster than with the original hand-tuned kernels and the Hampton kernels were tuned for these particular algorithms so our new code generation was able to find opportunities to go faster than what we had tuned by hand for the particular algorithms and you can imagine for new algorithms that we hadn't tuned those kernels for it's even better and there are actually newer numbers in under submission now currently embargoed because of that where every algorithm gets faster with the new code gen because there are always little fused kernels that we hadn't considered when we were doing this by hand so hopefully we'll be able to talk about that the next time I talk about this but look for that paper coming out hopefully in vldb next year so we talked about code generation compress linear algebra improvements to our API is becoming a top-level project now on to the last topic deep learning how much time do I have to this five minutes great so system ml started out as a project in IBM Research in 2009 and back then deep learning was not on very many people radar we were focused initially on more you could say traditional methods of deep learning making those scale making those customizable but in the last two years many members of the system ml team have been focusing on how do we make system ml extend too deep learning algorithms and the again the ideas as I said before we want to bring the benefits of this approach simple algorithm development system as fast your algorithm gets faster to deep learning now key to that is having a library of deep learning kernels and because we have linear algebra as our core API for accessing the functionality of the system we were able to build up a library in a very short amount of time basically Mike Dusenberry who's sitting right back there one of our committers I've put this library together in like a couple months of part-time work with a very extensive set of kernels and this went so fast because we have a high level representation of these kernels a 2d convolution which is many pages of code and C++ is about five lines when you're doing it with matrices on top of that for a pre-built models that you may have created with other toolkits we have an importer which can at this point import cafe and caris models and generate code in our domain specific language to train or score the model and we also have an extension to our pipeline API to be able to incorporate system ml running a deep learning model in the context of a larger data processing pipeline now that's that's the development of the algorithms to make them actually run faster well the code generation I just talked about is helpful in that but another thing that's helpful is GPU support being able to run in particular convolution operations on GPUs and we have a GPU accelerated back-end now there's take advantage of the plug ability that was designed into system ml from the start we design system ml to be able to handle both SMPS and scale out architectures like spark so we have a pluggable back-end we're adding three new pluggable backends for single node single GPIO single node multiple GPU and multi node GPU we have a lot of kernels that are already implemented for these backends and if you look at and step back at what the impact of these changes have been to our deep learning performance we're still not at the point where we can compete performance wise with systems that are purpose-built for deep learning but it's getting a lot closer if you look at a particular use case that some of my colleagues and research are working on involving inference for large scale image segmentation model starting a system ml 0.14 which came out earlier this year the performance running without GPUs has already gotten almost twice as fast to running the same code because of improvements to the optimizer code gen use of native Blas routines and with the GPU kernels we get an additional performance boost so we're overall about an order of magnitude faster now and this continues to move forward so talked about deep learning code generation compressed linear algebra API improvements and our being a top-level Apache project and if you want to try all of these new features out in addition to that core you know you're right in terms of linear algebra and outcomes a fast algorithm a spective system ml you can get system ml today from either system ml Apache org to download and install if you want to just try out a pre installed version iBM is hosting that on the IBM cloud you can go to data science ibm.com and just try it out without having to install SPARC I also have links here to some more detailed technical documentation about each of the topics that I've talked about today if you want to find out more yeah please take a picture there's all this will also be in the PDF so thank you very much for coming to my talk apologies for the delays here and then thanks to the system ml team because this has been really a team effort many people working over many years to produce this system [Applause] do we have time for questions you take a couple questions okay great any questions from the audience [Music] you so so the question is what happens when you add compression well that uses linear algebra to represent the intermediate data in the algorithm and that's a very important question with a very deep answer that we go into in a great deal of detail in our vldb paper the short answer is you need to work pretty hard to be able to compress matrix data fast and still get good compression and we did a lot of work on sampling techniques to be able to quickly pull out a sample the data determine how to compress a matrix and then compress it very quickly and we were able to get things to the point where in all of our end-to-end experiments we're including the compression overhead and we're still five ten twenty times faster yes now now one caveat here is this technique the compression is primarily for traditional iterative algorithms where your chief overhead is access to that training data that's stored in memory in deep learning we're still working out that story we have different ideas involving compression but those are going to be more about sparsity and that's ongoing work it's a great question thank you so thank you for the talk my questions about these memory module I'm serious if you're developing the foundation craft the question is are we doing a numerical district differentiation or automatic differentiation we implement the differentiation that the backward paths in our high-level language so that that library that neural network library I showed you that that Mike back there wrote includes the back prop as the computation of the derivatives there's no auto differentiation currently in the system but because we have that high-level language we can write down the gradient computation in a very compact manner now that means if you write if you add a custom operation you do need a custom derivative but at least you can write that in terms of linear algebra so great question thank you all right we've got time for thanks okay thank you very much