Down the Wabbit Hole

Scale By The Bay 2018: Chris McKinlay, Down the Wabbit Hole

Recording: Scale By The Bay 2018: Chris McKinlay, Down the Wabbit Hole

you yeah so my name is Chris McKinley I'm a principal engineer at formation and we mainly work on using reinforcement learning to optimize customer loyalty programs and so I'm gonna talk about some of the best practices we've adopted to write better machine learning code so what are best practices in machine learning there aren't really I don't think there's a strong consensus on it it's still like a very young field relatively speaking and it turns a lot even I think that criteria are up for debate I mean certainly correctness could be construed as a criteria but for many machine learning algorithms correctness is hard to even define properly performance is certainly a criteria but I don't think the most important one testability that's pretty compelling but I think it's it's probably downstream of a more general criterion which to me is is probably the most important thing so maintainability that's what I would like to see and we had a big issue earlier this year with maintainability of our code so we'd identified a fairly simple version of our reinforcement learning problem that we thought we could solve with contextual bandits and so we identified this library Valpo Abbott which had a pretty compelling out-of-the-box solution to the problem and our data scientists understood how to use it the problem was that we started building up a lot of code around it and the code was very very specific to VW and it started to leak into other adjacent services and this problem has been actually described before this is a paper many of you probably heard of or read and they outlined this problem referring to it as the glue code anti-pattern and they defined glue code as that the code that you build up to get data in and out of general-purpose machine learning frameworks and interestingly enough they have a recommended solution to the Glueck of the problem which is you should just reimplementation learning algorithm in your language of choice in your architecture choice which to me is like not great advice to give a startup especially if it's an algorithm of any complexity it would take us probably better part of a year or more to reimplementation VW and there's no guarantee at the end of that that the company would even be in the same place I mean applying reinforcement learning to customer relationships is like fraught with assumption churn so we need to be somewhat flexible to that so we didn't do that instead what we did was just built up some fairly generic interfaces that can abstract away the library and work with other libraries as well notably tensorflow and so I'm gonna spend most of this talk just explaining how we used some design patterns in Haskell and the type system to facilitate that and I'll talk a little bit about one particular use case running EMR jobs to train a stochastic gradient descent process okay so the I think the main pattern that we use a lot at least on on my team is called the handle pattern but a lot of patterns in Haskell are kind of blogosphere phenomenons it's not like there's any book for this so it's been referred to as the service pattern before - basically what it lets you do is encapsulate and hide state inside of handles explain what that means in a second provide functions to manipulate this directly rather than manipulating correctly and finally safely configure create and destroy handles so if you were to replace handle with object in function with method this would sound a lot like object-oriented programming and maybe in some other use cases it is in ours because the state we're manipulating is inside of C++ memory space it doesn't really look like that as you'll see I should say though that doesn't mean this is a great thing to do I think there's many ways to use Haskell that are much more principled I'm sure lots of you heard of monads functors applicatives functor algebras adjoint functors recursion schemes free monads they and all that stuff is really beautiful and great the problem with machine learning is you're generally stuck with a very stateful object written in another language in a lot of those things are really available to you okay so here's a simple example of a handle called a source handle which just represents a source of model files somewhere and it has three functions that you can use one to load a model meaning move a model file from some abstracted away location to a local directory probably a temporary directory on the machine and you might supply a model version if you don't you'll get the newest version you do and the versions there you'll get that version similarly saving and listing the versions available so we use this type of handle to abstract away the the layout of model directories and because we use tensorflow serving at work a lot which is a way to serve models not just tensorflow models it expects model directories to be laid out with like a model and then sub directories for each version and so on and so forth so the the code that generates these handles is responsible sort of maintaining the integrity of those versions in managing resource consumption so bandwidth streaming files from s3 to the disk and so forth that's a monomorphic handle like there's no type variables that's a little unusual a lot of our use cases involve polymorphic handles and in particular this is a handle of all spend a lot of the time and this talk talking about so this represents a model that's been loaded into memory and it's got five type parameters it's got an input for learning I and input for scoring J which may be different from I an output from the score and a error type and finally an M type which is monadic so the the call sights will put constraints on him like monad IO monad mask that type of thing and here's how you configure one so this is a type class to configure a model handle and actually properly speaking it's a type family so type families are a haskell extension language extension to GH c rather that we use that allows for some basic type level functions and by that I mean if C is some type that's implementing the class model config then C gets to specify for itself what type the learning input is the scoring and put as the output as the error is and and the finalizar is the thing that's going to destroy the handle ultimately and we'll provide a simple type alias model handle Prime that's going to sort of I guess you could say clinical call that type level function in abstract away all that the dependencies behind it a simple model handle prime of c m and because it takes up a lot less roomm i'm going to use that a lot and the rest of this talk and in addition to the types there's a two functions that need to be implemented i need to be able to create a model handle and i need to be able to destroy a model handle or delete one and here's where the constraints I mentioned earlier come in so in order to create a model handle your M has to implement monette IO which means I have to access input and output of the program and it has to also implement monette mask which is a way of handling asynchronous exceptions and because all of the actual types that implement this class are bindings to either VW or tensorflow or extra boost or whatnot there's generally asynchronous ASEC exceptions are of life you have to deal with its what those constraints are there and then you can see create model handle just takes the configuration and generates inside that monad a tuple of the handle and the thing that will ultimately destroy the handle and so you can see that being used here's a function from the API which is an instance of what some people call like the bracket pattern so what does it do well it takes in one here in action right so some function that consumes the model handle and uses it to generate some results so scores it on data for example and with model config is going to handle the safe creation of the handle destruction of the handle and then we'll run the action and give you the result back in the Monad and the way that it does that is using this this function called bracket which takes in a creator a destructor of the action and then runs the action and you can see in our case create model handles pretty straightforward delete model handle looks a little funny because there's this thing called a type application so this isn't a value per se it's it's a type because under the hood Haskell's basically system f this is sort of exploiting that and the reason it needs to be there is because if you remember delete model has all this type information around C in it but it doesn't actually eat see the value anywhere so we're gonna use this trick a bunch and then the composing with the second and the first or this is what spilling off the finalizer from the tuple that was generated and create handle and the the actual handle so this just takes the handle out of the tuple and then acts on the handle ok so let's see what one looks like here's an instance let's just configure a very simple univariate regression in tensorflow so y equals MX plus b and I can do that by mr. probably a new type just wrapping some generic tensor flow model config type which will handle all of the framework level configuration so what devices are we using how are we logging out of tensor flow that type of thing and simple is going to specify what it the types it wants to use in its configuration are and and because it's going to take when it's learning inputs of x and y this is a supervised learner it's going to take a tuple of float primes and it's going to output you know Y hat a single float prime or sorry when it's scoring it's going to consume a single float prime which is X and the reason their float primes where float prime just means tensor data a float is because I want this model to run inside of tensorflow and consume data from within tensor flow I don't want to pipe necessarily data into that and that's pretty common but I'll actually generate outputs and errors outside sense flow and then the way that I specify my create model handle is I'll build a special function that simple is going to have to provide called make handle and building simply means taking a build and running it inside a session and in tensor flow at least the most prevalent bindings build is just a state monad transformer on the actual graph state so build is just a pure function that lets you specify which nodes attached to which and that's it doesn't actually run a computation session runs the computation in session itself is just to reader T so it also notably provides the two constraints I care most about right so how would make handle actually work well it's fairly straightforward I know I'm going to need an x and a y for my inputs tensorflow has this like fairly see like way of specifying variable batch size and I definitely want the inputs to my to my sensor flip program to be variable I don't want to hard-code that stuff in the model definition itself that's not a good place for that so I'm gonna make sure that the you know sizes of the batches of data coming in or variable m and B are the two parameters that I need to optimize right so those are initialized variables and then the rest is fairly straightforward right I'm gonna write X MX plus B equals y hat see how far Y hat it is away from Y take the square of that and then I'll take Y hat and loss and I'll render those two things which just means specifying which devices etc those nodes would be rendered on and that's two-thirds of my model handle right there and then finally I need to actually specify what an optimization step looks like and because this model is fairly simple we could just use vanilla gradient descent if it were more complex we might want something like atom or something else but we probably wouldn't want to hard-code that in the model either that would probably have to get pulled out and putting to a configuration as well but once I have my step I can define my learn function and I ran out of space in this slide but creating the model handle from there is fairly straightforward just taking score prime learn prime and error prime attaching them to inputs and return in the handle that's nice configuring a VW handle looks much much different why is that well VW is similar to tensorflow in that it's written in C++ and it's basically designed around the notion of performing stochastic gradient descent online but after that it's completely different first of all there's no ability to design your own models it's all done through twiddling of command line and flags so this is a CLI set of instructions to run collaborative filtering algorithm on the classic data set called movie Linz ran out of slide here and everything that you would never need to pass it at the algorithm has to be passed through the CLI so all the hyper parameters around the learning the decay rate amount of l2 regularization the number of bits in the underlying array that's being hashed to even feature engineering so I'm going to take two different namespaces of features and I'm going to multiply them all together and limit the rank of the output matrix so a lot is is forced to happen through through the command line R this is more germane to our actual use case so as I mentioned earlier we use contextual bandits a lot there's a particular kind of contextual bandit called an action dependent function are actually two action dependent feature bandit and it in this particular case the set of commands is saying let's train a contextual bandit configured to explore using the epsilon greedy algorithm with probability of 0.1 and it's supplying a train set to that and it's asking it to output predictions as it goes so probably one in every 10 steps or something like that and write those out along with some logging and so this is the function that I want to spend most of the rest of the talk explaining to you how we production eyes but in order to do that I'm going to start at a fairly low level the F of high level 4vw so mm at the F of eye level all we really have is a pointer into V W memory which I'll call a session as well and we'll model it with just an opaque pointer alias so there's no there's no way to actually get any particular value out of this but I'm gonna wrap it inside a pointer so this is a pretty common trick for FFI code in Haskell and tensorflow does this too but the main difference is this is almost all I get from VW I mean I can get its notion of what it thinks that predict type should be which is basically an enum over ten or twelve things like string float int what it thinks the input should be it's internal representation of that CLI that that command line set of flags which is a untyped byte string and that's it that's all you get and instead of a tensor I'll have an example which again is a pointer to memory and I don't get much there either this is pretty much the full API right I can pass in a byte string of options representing the command line flags get my session I can kill my session if I have a session and I have a byte string this time representing an actual VW formatted example I can get access to that pointer and if I have a session I can destroy the pointer and then I can learn on the pointer which just means update the internal weights doing say one gradient step I can predict on the pointer which doesn't actually mean predict it just means make those updated weights back available to the example pointers that I can then do something with it right both of these have the same type signature and then finally I can get some basic statistics like the number of features seen and that type of thing out of the session so if I add two brackets to those things with session and with example I have just enough to be dangerous and so here's a simple function using those two so I'm going to take in my byte string of options I'm gonna take in something that looks like one of those learn or predict calls but could in fact be sequenced learn and then predict or predict and then learn some kind of extractor some way to read that pointer and get an output out of it and then I'm gonna pass it a list of untyped data inputs of some sort and then here's how I run it so I pass the options to my with session function and then I get back a session which I can then pass to with example for each date for each data in the in the list and then I can act on it and in practice what does that look like so here's here's some data VW uses the old lib SVM formats that's why it looks funny but basically this is a label and these are namespaces and in each namespace these are features and here's my command-line arguments and here's my action and here's my extractor and when I run that I get actually get a probability and that's because VW has inferred that what I really want to do here is run a logistic regression is and if I were to run this on the same input replicated five times you'd see the probabilities creeping closer and closer to one as I over fit the hell out of this one example so this is um obviously pretty limited and not what you'd want for a high-level API that the most notable problem is there's very little static assurance or compile time assurance of anything the byte string for the options is untyped the by string for the input is untyped you have no confirmation that what VW is expecting in terms of input type is what you're actually giving it in terms of input type so let's just quickly solve all of that and one way we can do that is by just noting that this looks a very sort of CPS or continuation passing style right I've got a bunch of unnamed lambdas nested inside one another so I can use what's called a continuation monad to model that carefully and all a continuation monad or monad transformer does is takes a function that looks like this and what this type signature says is if you give me an e to monad of O then I will give you a monad of O which if you're familiar with United limos basically the same as having an e modulo fuzzy reasoning and if I type that if I specialize that using some applications so the type I care about we get this which looks very very similar to with example right in fact the second half of with example looks just like the first half of my Conti constructor which means if I just compose them I can replace this part with something more general that's still polymorphic over the output type and lets me structure things more nicely so I can take that type and abstract it and now I have something I can work with I can place constraints on so this is just simply a type that represents what we had here except instead of forcing it to be a byte string I'm going to loosen that up so I can take more structured inputs other other types of input and the first thing I can do with that is differentiate between things that are serializing data in two example pointers but never do any output they never read stuff out of the pointer so that means if I have a function that does that it really should work with continuations for all output types it shouldn't care what the output type is because it never does anything with the example it's just on the input side right so that's nice and then maybe I'll take that the one other thing that we saw a couple times where I've got either learn or predict and then some kind of extractor and I want to run both of those so I can also lift that and do a VW and in this case the input type and the serialization type are the same and once I have those two if I go back to process lines it looks a little bit different instead of that nested sequence of continuations I just have one little monadic statement and I've got my serialized function on my live function and that's nice I don't know if that was worth all that effort but this isn't really the target audience process lines is like a throwaway piece of code what actually we get to do is construct more complex serializers and use the types of VW to impose constraints on that configuration when I generate model handles that's really what I'm after I mentioned a moment ago that our main use case was what's called action dependent features so that has a very complex serializer which were able to sort of nicely capture using continuation passing style and this is almost perverse but the way that action dependent features work is it actually takes two example pointers and a list of byte strings which needs to be non-empty and this represents really morally a single input a single example and what it's going to do is look at the head of the byte string serialize that into a pointer run learn on it get another pointer take the rest of the byte strings BS serialize all of them into pointers run learn on all those pointers but throw them all out until it gets to the end of the list then serialize a empty byte string and then return you want an e2 because of course and it's so it's nice to have some structured code to help you to reason about stuff like that and finally the last thing that we do is when we configure model I have a couple type parameters right and I can enforce that like the thing that learns never generates any output because it shouldn't write it should be a pure IO action I can enforce that the thing that's serializing examples into the learn function because recall recall my handle takes different learn from score types so I could enforce that that actually isn't touching the output and I can enforce that something that's actually going to generate output isn't doing anything else with it I mean it has its own serializer and so when I put all that together I can build a model config fairly nicely one thing to note is my e is actually the example type so this is how I differentiate between different kinds of bandit but all bandits all models have the same statistics so that's the error that that my model handle generates is always this large fairly large type called statistics and I think in the interest of time I will skip the details of the handle creation and go straight to using which is more fun so I'm going to talk a little bit about how we put together a harness that can run SGD jobs using constraint cones and the kind of thing and one thing we use a fair amount of work is it was locally referred to as a has pattern which comes from Lin's library so lenses are used a lot in haskell to access data and what a has food class does is it simply says s has a lens Prime between itself and foo which basically means it has a setter or sorry rather a getter it can get a flu out and if it has a foo we can update s's and there's a similar version for prisms and one thing that we use is a slightly weaker version which just includes the getter version so if I have an S that means I can somehow get a foo and I'm gonna need that in a moment and actually we're going to make it a little more general so that we're not hard-coded two Foos but any a so get a is simply function from s to a Morley so okay let's talk about our SGD harness so first off I'm gonna run SGD online so to speak which means like I'm gonna take a bunch of learning steps and then do a few holdout steps and then take more learning steps and then do more holdout steps so rather than doing all the holdout at the end which make the more used to and and that's a good fit for a couple reasons but it means that we're going to log the learning stuff separately from the holdouts stuff and put them together at the end so as our learning logger look like well it needs to know the number of examples had seen so far and the loss that the models accrued the training error and I'm gonna give it a mono ID which I won't talk about much but because the model is C++ code I need to rely on its error type from that handle to tell me how it's doing which means I need to have the model handle Prime and the input of that handle whatever it is and then I can pass it one of these L info learning info types and then because it's going to need to access either a tensorflow session or i/o + VW case it's going to give me back in M of L info and so once I have that log or type I can give you an example and you can see where these constraints are getting used so if I have or what's this type signature saying is basically saying for all configuration types and all monads such that C is in fact a configuration for some kind of model and the error type of that model is able to generate a loss because in the case of VW the error type is a very large statistics type it has lots of stuff in it right not just a loss in tensorflow case because we can really specify exactly what the model does I might be able to just make the error loss directly but this function doesn't care about that right as long as these constraints are satisfied it can do this job it says okay well tell me however boats you want me to be how often you want me to update and I'll I'll check if it's time to do that and what does do that mean well it simply means call the error function of the handle on the input get the error use the get lost constraint to update the loss and then log it somewhere probably need monad IO for that and then lift the result back into the M specified by the model and so I can start to put this together with there are stuff to run a pipeline of actions and one way to do that in Haskell is to use a conduit library which is sort of resource managed streams very similar to FS - if your Scala person and it revolves around this type conduit T which is it's a stream of inputs the stream of outputs some sort of monadic effect and then some sort of result which lives inside the monad and I'm just a a aliasing here so I can fit everything on the slide and so here's what learn looks like if I have my model configuration then and I have a model handle and a logger type then what can I do I can consume inputs and in this case inputs are just tuples so this is just the example number and the actual input type which of course depends on C itself and I can grab the input pipe it into the handle and learn on it lift the result into the conduit and it arrum look like that so it's just going to run a IO or M of unit action a whole bunch of times and keep returning the A's and then I can use my logger after it consumes the handle to scan over that input of A's and aggregate them into that l info type and to begin with it'll because it has a mono it'll just be the default so this is pretty nice you can also see that I have tons of constraints on C but C is a value doesn't appear anywhere okay let's do the same for scoring okay but scoring is a little bit different because I'm gonna do it outside of the model do it in Haskell and that's nice because it lets me enforce even playing field on different models right it'd be very hard to compare a tensorflow contextual bandit with a VW contextual bandit if I'm letting them both do their own holdout scoring so however in order to do that I need to specify some additional constraints so first of all I need some way to get a scoring input from learning input and I don't want to have to do that in data I don't want to be reformatting every tenth data type or every int data type when I run this job we if there's a model dependent way to actually just throw out the label from the learning and put somehow and pass just the features to this scoring input and I have to do that because I've I've got two different types for learning and scoring so in that simple tensorflow case we saw get might just be first right because I had a tuple and I want to drop one but in a more complex case this might be something really different so I'm going to encode that in what's called a constraint and in order to do that I need a language extension called constraint kinds which just simply lets you sort of manipulate constraints to functions so the second constraint I'll need is actually to do the assessment the holdout assessment and in order to do that I need to take the output of the model and figure out what action it shows it's got let's say a number of different things it could to select in order to do and because this is a reinforcement learning algorithm we don't necessarily know the the the world's response to that action if the model had chosen it but I can get the action still and from the input if I can get a special type called an ARP which just stands for action reward probability if I get that type then I can use an inverse propensity sampler to estimate what the cost would be hence the term IPS so this is a constraint that lets me run an inverse propensity sampler and when I add those constraints to my monad i/o constraint I can actually do holdout and so the way I do that is I just score the input so we're score just looks at the the data and then it uses the that get to extract the scoring input from what is actually a learning input right past the scoring and put to the score function on the handle so the types lineup lifts all of that into the conduit and I'll return that along with the original data so that I can update my holdout on that and so what does holdout update look like like this so I've dropped the holdout constraint cuz I don't need it anymore but I still need the IPS constraint and what I'm gonna do with a constraint is consume these these triples and I'm gonna run IPs on the input data and the output action from the model and then I'm just going to aggregate that using the the mono at instance for scoring info and that's it so when I put all that together and I'm not going to go too much into how IPS is implemented but it needs constraints in order to be able to actually compare the things once I put all that together I can define a logger that's just going to tell me okay how many times you want to actually score the thing vs. learn and give me two loggers one for learning and one for scoring I can put those into a fold so I'll issue zero ID to start so index starts at zero depending on the strategy I'll generate either zuv learning scoring i'll run hold out or learn and then i'll gather at the very end hold out our learn is just simply going to look at whether the either is a left or right and decide what to do and here's where that type application gets passed because we call learn and hold out don't actually eat see and then gather just takes those either x' and then aggregates the appropriate logger on the left or right and returns a a product and then when I have all that I can put it together in what I told you I was gonna tell you which is this with model config function right that API that we looked at on like slide 10 and recall that with model config has a monad mask constraint and none of the other code you just saw has any so that's that's why that's there and in order to lift everything in the monad mask I need something slightly technical called lift with from trans control but basically it's just going to for free give you a synchronous exception handling and all of that interior code which didn't actually have that constraint and it's going to consume that action function and all act does is run train fold which we just saw a get all of that and then it's going to check the very very last one and if there is a info there it will log it and if there isn't it'll just log a blank and in return and so this thing can run in a bunch of different places in a kubernetes cluster with access to tensorflow it could run in a Hadoop cluster so on and so forth so that's it I don't know one moral is I googled machine learning architecture I was preparing this talk and I love that the the very top hit is actually talking about liberating architects so I like I don't really think there's a lot of good literature on machine learning architecture and you should feel free to make up your own things in question authority here are the links I promised you and that's me thank you Chris great all the complexity of machine learning and all the dense obscurity of Haskell all in one thought a mile a minute yeah yeah so we we do have a few minutes for questions before running along to the next talk so are there any questions we never a beat against the sort of type class version of it so I can't really answer your question authoritative Lee but I would say the culture at my work is pretty anti type class unless it's like really necessary so we didn't go there yeah it's a great question so the reason is for some models it really just has to be so for certain reinforcement learning models the scoring isn't just adding a label to the training it's much more like adding probabilities of different things to all the different possibilities another reason is we want to make sure that models that are going to serve are not accidentally seeing labeled data and in bobble Abbott's case it will silently consume labels and update itself without telling you unless you're careful yeah cool thank you [Applause]