Scale By The Bay 2018: Matthew Tovbin, Making Spark ML Models Portable Know Your Options
Recording: Scale By The Bay 2018: Matthew Tovbin, Making Spark ML Models Portable Know Your Options
you thank you thanks for the introduction hi my name is Matthew I work at Salesforce I see a lot of familiar faces here thanks for coming do I talk so I would like to thank the all of you for coming over and actually thank the organizers amazing really amazing conference delighted to be presenting here so I do machine learning I do skyline spark at Salesforce and this is a little bit of a nice talk we'll be focusing on how to make your models portable but if you have questions around type safety spark and stuff you can talk to me after so where do we start let's start a little bit with a machinery and pipeline so this is pretty much typical machine learning flow that you would see from the books and then you can layer it out and kind of six stages more or less where you have your data produced by some process and then you you run some feature extraction which we're from which way like you extract the valuable pieces from your data you do some kind of feature engineering of imputations and maybe a useful statistics for your data either feature selection on top and this is pretty much the three pieces where you do the ETL right so this is where we do the kind of more classical ETL part of our data pipeline or machine or in padua in this case and then there is a learning phase in the learning phase where we we want to compete head-to-head different machine learning algorithms one to another we want to take logistic regression random forests with our XG boost whatever comes next tons of flow models who want to compete them and we want to do some evaluation on those models and eventually what we want we want to produce some useful scores and insights so this kind of represents the result of this pipeline this is one best model that we then eventually want to serve so this gets us to the to the scoring flow in the square flow it's not much different except that we don't know we don't now compete models against each other we already know the best model we've learned from the from a training set multiple training sets what's what the parameters are and now we have to the same ETL process because we need to keep the dimensionality of our data the same and apply the model right the same score calibration repeats insides and so forth so this is kind of give us the baseline for the stock what the machine learning pipeline is and then we produce scores and insights here the things that best model spans across those across those steps it's it's not like something just the algorithm part we actually learn a bunch of stuff during the ETL part as well so whenever we aggregate we actually might might learn some of the properties of the data which will which we then use to further our best models for example we started by inspecting our some of the text data but we figure out that it's not a regular text that is just just a categorical data just like a limited set of values and then we can actually treat them better as a categorical types instead of tokenizing them and applying different text text mining techniques which will be less efficient in this case so model actually again expands outside just the algorithm part it goes to the ETL part as well so how do we let them capsulate it in spark it's very natural to think about as a single application it's not necessarily the case whenever you deploy it in real life right we might have different processes that actually does some of the feature engineering for us offline but you can think about it as a single application something that you can kind of run it and think about as a single unit on your laptop or or on a cluster and a lot of people do that and I mean it's kind of very typical scenario it's not necessarily good or bad again it suits whatever suits your use case what happens in spark right is that we have a bunch of stages that we stitch together in those pipelines and those stages like might look for me like I just gonna memorize the names so they can fit into the same graph but they have much longer names and Java classes and in Python also we're in this case just like put some examples of okay we want to compute some feature distributions on on our data and then we want to decide now should we tokenize the text because we decided to text data with at organize it or want to pivot it so this is a decision that we will have to learn during the luring phase of our model and then we want encapsulated into the pipeline whenever we save it and then serve it so this is a this is one of those learning parameters that we want to keep keep around so whenever we train the model you know going back to the example of train right we now learned all those like I put this green green thing so those are things that we learned when we trained the model is that for therefore the for the imputation for example we learned what's the mean on our training set and and now we're going to be impute in this value for the mission instead of the mission values on our scoring data set same goes with the tokenizer for example maybe we learned what the most typical languages are and then we learned maybe some stop words that we want to exclude immediately from scoring data so all these things have to be included into the model and then and unserved so I actually like to think about this whole thing as a is more of a and like a functional twist a little bit right so we have we have a program that we write and then we train it on the data sets we applied on the data set which then produces another program that we can now apply the new data set and we can think about as a function right it's like a train function is a function that will produce a scoring function that will be applied on a on a data set of the same type which then will produce a data set of type s so I wish spark engineers would think about that well anyway maybe they thought but the reality is like we actually live in a mass much less beautiful world so SPARC is actually pretty messy to work with so if we can if we can somehow to get to that purity that would be great maybe like in several more iterations we'll get there interfaces grow based on the usage unlike the feedback so I guess it's in our hands to get there so what what's really the problem here would like to learn we would like to train a model on spark so that's like that's the first thing if you're if you're not crossing that part probably or you don't have enough data or you actually have to be very careful on that moment do you really need spark so if you decided you need spark the rest of the talk is for you if you don't and in a lot of actually a lot of answers are no we don't really need spark again there's a people that I really like to write on this buzz of yeah the big diet and stuff but a lot of things you can really really solve without it so if you decided that your company or team needs spark then eventually what you'll have to do you'll have to save your model somehow you have to train it on the data sort of multiple models and then juggle it to the production servant system so we need to load the model and then serve it efficiently depending on and depend on your runtime you will be run into if you don't spark this is again very natural because it's something that that's very easy to do out of the box where you want to run it on some other a runtime so examples might be you want to serve it on some application server in JVM where request comes in and you immediately score it and return back so yeah why not serve the spark right spark Azhar as a messy graph of dependencies and it's you know for a long time it's been stuck with an older Scala version it really only runs on Java 8 we can that's like a long list and Jason 4's problems and there's just a ton of things it happens with a lot of legacy code bases so it's not unique we have we see the same problems with the other large system such as Hadoop which are more much more modern now but before it was the same thing so I guess it will take a time for the ecosystem to evolve but so far I see a ton of projects where people you need this kind of stuff and then in the real-time part okay so what I wrote here is like a real real time scoring so you can do real time scoring with spark it's just not you cannot really do it in a sub millisecond of very few millisecond latency for the record data frames have you know have have over had reasonable overhead like a significant overhead of for the runtime and you will you will see it once you start working with it so so what what are the prerequisites here right we want to be going to produce a model that is a that is realizable and of course this realisable we want to make a model to be interpretable in some runtime and we're also going to assume something's right so for sub-millisecond when testing is actually most likely you won't be doing most of the joins and aggregations you can still do some some simple joins but I mean like one-to-many many-to-many joins those our cross products sometimes you actually do need cross products there has to be done in a training phase right you don't throw them at your score in time the exact ushion of them all has to be a thread safe and then can i as i mentioned SPARC is probably no go so this little scale on the bottom I putted so like how far can we go from spark so we can go from okay we have a spark runtime or we can have spark round time to just load the model and then we completely abandon it and we score it score using whatever JVM probably in this case where we can just use some of the spark dependencies for some of the reasonable sometimes you know we can just do JVM with something else if you're not in that camp you can try to go further and I will now explain how can you do that but first first let's see what how the house spark deals with serialization of models so sparkly finds this this interface is like I took one of examples I IDF IDF model that you have to extend ml writable trade in which you have to produce animal writer in UML in your write method ml writer have a safe implementation pretty much have only guess it doesn't have much limitations to it it just requires you to provide a it provides you with a path and then you have to figure out what to do with it you also have a spark context there most of the models that you haven't spark they actually do that I have the example yeah so what most of the models that they have in spark they just use spark session and then they store it using using parka files so not all of them some of them are some of them not so I kind of you find out once you start using them when when I speak also here's the model right so every single stage in spark is a model on its own so whenever you stitch them in a pipeline you actually have a bunch of those little little models saved on your path so this is like this is really not great and then the load very close right you have to your model have to provide a reader and then you have to you have to provide the reader function so this whole core that lives in spark every single stage you write in spark you have to write this stuff and if you extend you write your custom custom stages you also have to deal with this but what what gets the how do we get to the serene part so serene part is this right after you load it in spark currently what you can do you have this transform method for each method and on spark pipeline what it does it takes your data frame and then applies all your stages one by one sequentially on the data set completely unconstrained interface like you can pretty much do whatever you you want in there but most of them look like this they they apply UDF and you will you will find very quickly that this is very slow so like all those data frames optimizations they work fine with you know calmer like simplified corner operations they don't work with UDF's and then whenever you stitch your whenever your pipeline is more than three to five stages they can depend on the data size you already see the impact so this is like this is this is pretty serious stuff so we which makes it totally unusable for real time in the news cases so again to recap out of the books we can save and load up our key files and then we can use spark runtime to transform data over datasets so what what are the options we can we can actually how where can we go from here so there is a an abandoned part of spark it's called Emily local this is like a little library that has an implementation of a local linear algebra it doesn't support spark models yet and there is a year several year long discussions about how to improve it but really nobody touched it so I guess if you guys want to contribute to to open source that might be a good good way of doing it I'm not I don't think I have enough time to do that yet so what may be so there are some like I put some some spark gears here to to track so there are some ideas actually how to move this forward and maybe reuse some of the community contributions but so far nothing so another thing is the data bricks local offering so you might also have seen their announcements of that you can explore data bricks models from a notebook and I've seen some data scientists using this this works ok there are some there are some missing parts but there is a good there is no source code so you have to take a jar from maven and you know D compile it and so and they're not even supporting it so again hopefully it will become a moldable local eventually and all of the things I want to highlight this at the different approach where you know a group of companies is a including IBM they decide ok we have to build a standard and most of you already know about PMML Sokka was a de facto standard at some point now it's less and less popular because of its limitations so first of all XML and next to sell I mean they have strong benefits of course being typed and everything but they're really really painful to work with so it doesn't have it doesn't have control structures either so if those like Forks that I mentioned where you want to out should I pivot or should I talk in eyes you cannot really do at run time during scoring so here goes PFA so PFA while Lagos comes to FN and this kind improves on the PMML offering by by providing a bunch of conditional operators and variables and I found it actually very nice especially because it's a standard too so first first things first you have a very strong foundation of rules and then we have to actually several runtimes that that we can use and recently they announced there is an art fork library for spark which supports Muslim spark models I mean it's still still very early but usable so and then of course you can use some other third pie offerings such as mo Emily from combust ml where you can bundle your your pipeline into their format and then run it another option is of course write in your own writing your own around time in which you would use spark to read and write write and then and then you would you would use you use some kind of custom runtime you would write your custom runtime for the for the spark pipeline and here I want to just do a brief case study on the transmogrified library that I'm developing so this is automated machine learning for for spark and we took we took an approach of actually building a custom runtime for us it was important for it again to have a sub millisecond latency but we didn't mind that if it's one spark or or I mean sorry we didn't mind that it runs on JVM and has part dependencies since we already have them so what we did we actually implement in some ways classes to marshal the stages using the the the spark writers and then we offered we added a new function to transform any air between key value key value pairs so we wrapped Allspark models in our classes and then internally we use the Java reflection method mirror API which is surprisingly fast and people like to think about reflections is very slow and not performant the things that reflected method is slow but once you reflected it execution of the method is very fast so you can reflect it keep it as a whatever is a lazy vowel or vowel and then and then execute it on for per cell level extreme extremely fast so it's just really only probably twice as slow as a regular direct method invocation and this is how it looks like so we can have a model that wraps the rocks around spark model we reflect a bunch of internal methods and here you go have a type signature to do the same transformation that spark model has but now the difference is that we can use this transform function to execute it on just pure JVM without need of data frames so whatever is not wrapped we just use PFA we just use PFA engine to to execute it and and I just build a little performance like I ran a little benchmark here yeah I drink it on the 6 million rolls and at Everage but point zero three milliseconds per record which is like two factors of orders of magnitude faster than that what you get in spark or sometimes three so what have we learned right so artwork PFF very still oh I didn't mention that but they have some buggy stuff still it's pretty much the one-man show yet so expect expect some bumpy ride on this if there was a Scala - PFA whatever a compiler plugin or a macro I believe it should be a compiler plugin because that would that will actually increase the the adoption of PFA over all and would be much easier for people to use and then yeah custom code libraries you know I I would prefer to have a standard again so I'm going back to the start part I would love to have a standard that several teams several companies work together and we don't reinvent this we love custom runtimes and try to do things like that so I want to end with this little joke so how do does anyone recognize what this is so this is Jason cool it's very simple okay now how many libraries do have two parts Jason so yeah github says 4400 just on the gig happen at Google says I can even find them near me so did it doesn't parse my query well which is it most likely doesn't it it literally says that I can find them so instead of a summary for you so a little decision diagram if you're in deep learning which I didn't put on the presentation because only 20 minutes and I have 15 seconds left you go with a standard Colo and an X check it out again several companies actually gathered and trying to build a standard if you don't mind SPARC just yeah like whether look at it make your decisions based on that one more thing if you still wanna run spark there's another thing I I found a couple weeks ago so Microsoft open source their MML spark library and they have sparks serving what this thing does it actually allows you to bypass spark driver and a data frame and hit directly to the executors I mean you need to wire this whole thing but this can eventually bypass a spark and allows you to use the spark parallelism so if you're up to a challenge you can also try that one out so questions yes so yeah exactly mentioned this look why would you even stay with spark right there is still it some retail you want to do before scorns actually why we should yes yes of course there's a ton of ETL yeah if there is a tone of ETL that you have to do that's like it's actually this is the cool part of spark which you actually you can leverage the part lism and the the the aggregators across all those executors yeah yeah exactly you could yes a bleep so Emily please so it's actually pretty nice right but it's just fully file based and back from the file and it's again it's custom it's keep changing we have we have seen some breakage across versions so I don't think this should be as it should be less of an issue for standards those develop slower okay thank you very much [Applause] you