BDSBTB 2015: David Andrzejweski, Economical Machine Learning via Functional Programming

BDSBTB 2015: David Andrzejweski, Economical Machine Learning via Functional Programming

Recording: BDSBTB 2015: David Andrzejweski, Economical Machine Learning via Functional Programming

alright thanks so much for the introduction Alexei and thanks for having me here at scala by the bay this is really been a fascinating set of talks and conference so far and looking forward to some more interesting conversations after this stock so economical machine learning via functional programming first the pitch very quick just so you have some context of what sumo logic is and why I'm interested in machine learning in Scala in the cloud and the sort of thing assume my background is kind of machine learning research but at sumo logic we are operating a machine data intelligence platform that runs in AWS so you could send your log data for example that all of your Scala data processing code is undoubtedly emitting tons of logs send it up to our service and analyze it do lots of cool slicing and dicing I'm we're also working on some machine learning capabilities on top of that we were sort of an early-ish scala adapter using 277 back in 2010 so those ways ago and if you want to try it out there's free trial for less than 500 megabytes per day so the rest of this talk machine learning very useful very exciting I've seen a few talks and discussions around it already at this conference but there's a kind of a growing awareness that it brings a lot of potential additional engineering complexity in your system and what I'm going to be talking about in this is that functional programming techniques can kind of help manage the additional technical debt that you're going to be taking on from the very moment you decide to use machine learning for anything essentially so machine learning very exciting very fun will the robot to take our jobs will they annihilate humanity machine learning companies you know raising tons of money doing tons of stuff getting hired by Facebook and hired by google lots of clip art heads with gears in them little humanoid robots studying books you definitely know you're looking at some machine learning blog post or a company article if you see these so what is it though and this is a great explanation and and this particular fellow has won the go tell prize for being the co-inventor of boosting so we can take his his word for it that machine learning studies computer algorithms for learning to do stuff so what kind of stuff can machine learning do and how can we do it so there's been a ton of examples you're all kind of probably working in various domains where maybe you want to predict if somebody's clicking an ad you want to rank or recommend content you want to identify whether behavior happening in your system is malicious or not you want to automatically identify labels as what's you know automatically identify images by what's actually happening in the image associate text with images basically any of these kind of fuzzy tasks where there's not a super well-defined deterministic algorithm and you have some training data that's potentially a candidate for machine learning and how it's actually done you can sort of think about it that the 10,000 foot view of just breaking it down into three steps you're going to pick a model and so how are we going to represent the data how do we want to learn some function from inputs to outputs from images to text labels from ads two probabilities of clicks you select this function you have a whole bunch of training data hopefully some ordered input X output Y pairs and you feed that into some estimation inference or learning algorithm and out you get an estimated version of the target function you want to have and then finally there's the predict step where now you have some new data points and you have your estimated function and now you can predict some output Y some label whether or not somebody clicks something like this from the input so not actually that complex and one of the reasons that this is becoming more and more interesting and relevant is just these rise of things that are complementary to machine learning techniques which is more data Moore's law and more cloud so you actually can easily and kind of elastically afford to get the data to the CPU and do some machine learning on it and so actually you know there's been kind of a lot of buzz and not about deep learning but some of the more some of them some of the experts in that field if you talk with them they'll actually say the thing that's really changed in the last 20 years or so is actually just these things not so much that great new algorithmic breakthroughs or anything like this or suddenly new generation of brilliant scientist but rather just finally having enough cpu and data to throw at those kind of techniques and get some valuable results so this is great we're all very happy very exciting things machine learning money resources stuff fun great but there's a downside and this is from a really interesting talk that I'd highly recommend checking out the slides I'm not sure if there's a video from icml we're basically machine learning disrupt software engineering so a lot of the techniques that we sort of have for building reliable software or managing deployments or operating things really run into trouble with machine learning and one very popular way to think about this even in the non machine learning software engineering world is technical debt which loosely defined as these kind of things that pay off now but you're going to pay for it later there's going to make changes harder to make later in your code you've coupled something together you've added a dependency that you haven't really tracked nicely you've copied and pasted some code from one thing to another and just changed a few things instead of factoring it out into a nice separate method you've got some really heavy-duty states some global variables basically running through your whole code base all manner of kind of post-processing duct tape glue code everything like this this is kind of a fact of life and sometimes there's good reasons to do it and sometimes not so good reasons but machine learning gives you a whole new and terrifying universe of exciting ways to take on technical debt and have huge complexity problems in your code base and there's been a that's a nips workshop paper from some folks at Google and then that icml keynote I just mentioned and a PI data keynote from this year as well kind of this increasing trend of talks and papers about this kind of dark side of machine learning the software engineering challenges and complexity that you are going to be taking on when you include machine learning in your product service code base pipeline anything like this and some of it is it's just intrinsic to the nature of what you're doing you have these functions these predictors they're probabilistic they're noisy there you're training them on data sets you're applying them to other data sets how they're going to behave is not really nearly as guaranteed as hopefully a sorting algorithm it can be very difficult to test or debug is your algorithm bad is your optimization bad has the real world just changed any number of things could cause your machine learning code to not be performing as you would like and when you're changing it it's going to be very difficult to be sure that you're actually getting a genuine improvement in your model versus you've just over fit to new data set or you're now classifying these images correctly but there's a whole bunch of other images that you're not classify incorrectly anymore the boundaries between your components at architectural level can be eroded or you can have weird feedback loops you can imagine there's actually really nice an example in the bow too icml keynote of two different teams that are both working on different parts of the same page doing machine learning where their algorithms are sort of interacting with each other in a bizarre and unexpected way and so you can really there's an entire kind of cottage industry of very interesting stuff that just sort of shows how hard machine learning can make your software engineering problem and so that's I'm not going to dig more into it but we'll just say that you're taking on a lot of complexity when you add machine learning to your software so to take the technical debt analogy this is a loan repayment schedule so this is how many payments it's going to take to repay some principal assuming a fixed payment so let's say I'm going to take on a bunch of technical debt and I can only afford to pay 500 debt dollars per week how many weeks does it take me to pay off and as you go to the right you're only covering the interest and you're not even covering interest and goes to infinity you are never going to pay it off it's time for a total rewrite we're going to throw the code out and start over technical bankruptcy and at different points along the curve if you're looking at time to repayment as sort of your cost dimension you can see there's sort of different Marge cost of taking on additional technical debt principal so if you're already kind of in the danger zone you're going to be adding way more weeks of technical debt payments than and a further left on the curve and what kind of all of those other talks and papers would sort of tell you is that by using machine learning you've already scooted pretty far to the right on this curve and the marginal cost of complexity in all of your general software engineering is potentially a lot higher so what can we do about this so there's a really interesting paper that I've seen another kind of data science machine link toxin it's almost sort of a functional programming manifesto of sorts and it's out of the tar pit and they sort of try to break complexity and software engineering into essential verses at incidental or accidental and the essential complexity is what are you actually trying to do in the real problem at hand and the incidental complexity is just kind of the stuff that you sort of have to put up with as part of encoding your solution and software and so a sequel query in the relational calculator relational algebra that says it's declarative it's nice it just says what you want a really really complicated MapReduce job that you hand coded by Java in five thousand lines that's probably incidental complexity and there's been a lot of other really interesting talks at this conference in the Scala world about scalding and spark and a lot of these you can kind of view them as attempts to sort of shift the incidental complexity into generalized frameworks where people who really really focus on that can focus on it and let you as the framework or library user application developer focus on the essential complexity of your problem at hand and likewise functional programming kind of falling in this declarative world can also kind of do the same thing so I'm going to advocate that you can think about trying to control your complexity spend with these functional programming techniques and so avoiding mutable state minimizing the surface area of your custom logic being able to reason about what's actually happening in your functions and building your machine learning and data processing pipelines out of these small well tested well understood well typed functions and so what do I mean by functional programming there's two kind nice resources one particular focused on scala i would highly recommend checking out the functional programming in scala from manning and you know there's probably been a lot of other talks and discussions and in a scholar group at this conference probably assume some baseline knowledge but the idea is that we want to have pure functions that have no side effects and we can kind of substitute them easily and as a consequence a lot of things that sort of fall out of that are you want immutability you want to be able to treat functions as first class objects and you want to manipulate them with higher order functions and now before we dive more deeply into the functional programming content I have to you know say a disclaimer here that I'm just a simple machine learning guy and kind of coming to the functional programming world in Scala and sort of seeing how what I find there can help me solve the problems I want to solve to solve in machine learning context so if I mispronounce something or if I'm slightly wrong I'd love to learn and hear about it but just a disclaimer so this is just the super basics map again probably at a skylit conference you know you can map a function over some list or some container you get a nice output you can fold or reduce so this is just sort of at a very high level the flavor of the kind of stuff we're going to talk about so how to control the complexity first thing does you're the little helper functions that you decompose your machine learning pipeline do they even work and one big thing that even in Scala where you have this nice type system in all of this I still see you know more often than I would like is there's a two different functions for getting a data set and which one of them is probably going to be a source of some goofy bugs somewhere down the line right so clearly they're they're both kind of doing the same thing and from the signatures you can see it sort of makes sense but I could totally easily mangle the order of the first three parameters on that first function and the compiler will very happily compile it and run it and something weird will happen either in my tests or if I don't have the test coverage I'd like to have in production so we're possible Scala gives you a lot of tools for just sort of baking your types about your reasoning and knowledge into the actual code itself and if you flip the time range in the data set ID in the second example the compiler explodes and your code doesn't even compile fantastic so you fail fast using case classes liberally and there's something called unboxed tag types things like this so there's going to be links associated with these slides and I post them you can dig more into those and they'll give you a little bit more confidence then when you've kind of broken everything down into these nice little functions it's a lot easier to do unit testing and there's another really interesting PI data talk I'd recommend checking out about testing for data scientists and the idea here is that when you have whatever machine learning component even your machine learning code can and probably should be unit tested and what I mean here is the different subroutines the optimization steps the data processing steps just because it's all part of this noisy deterministic machine learning pipeline doesn't mean we don't want to test it and another very valuable resource here could be just a small collection of really simple really obvious examples that your classifier or your ranking system or your regression system should very obviously be able to get just a total sanity test and just bake those hard code those into your unit test suite and if you tweak your pipeline tweak your algorithm and now it's not predicting those anymore then that's going to be something you want to take a look at you can take this idea step further with property testing in particular the Scala check library so here you can you define properties that you expect to hold for your code and again at the function level this makes it a lot easier to kind of build up these property tests and the library will automatically generate random data to test whether these properties actually hold so in machine learning an example property might be bounded output this regression model should always output something a probability should always be between zero and one that's it and if your algorithm is not doing that if your model is outputting negative probabilities or infinite probabilities or something bizarre when given weird input these kind of automatic property testing suites are really useful way to determine if your function is going to behave like this which again in the machine learning context is potentially really helpful and you can take it even kind of a step further and use this property testing to actually determine if confidence intervals or pack style bounds of your machine learning models are holding as expected and to unwind some machine learning jargon here for a second pack style bound is probably approximately correct so in this equation here L is your loss or your error and P is probability so the probability that your error is greater than epsilon should be less than or equal to Delta and a lot of theoretical machine learning models have these kind of pack style bounds and guarantees in much more complex form than this I'm going to determining based on how many dimensions your data set has and all sorts of other factors and while these bounds might not necessarily there's been some kind of like debate in machine learning community but how valuable these bounds are in the wild on practice on live data sets if on synthetic data sets your estimation algorithm isn't even meeting these bounds then it's quite probable that you have a bug in your machine learning implementation and it's very easy to have that of course so what you can do is you can configure a in Scala check land a generator that will automatically generate synthetic data sets and then treat your estimation function as the function under test in a property check and you can sort of say generate a set of data sets and if I estimate overall my function over all these data sets the conditions specified in the pack style bounds should hold there should be very few data sets where my error from the estimator is larger than the epsilon that's guaranteed by the bound associated with my learning algorithm so this could potentially give you a lot more confidence in the correctness of your implementation of a learning algorithm so another kind of scallop functional programming nicety that you can use when you're constructing these data pipelines is the type class pattern or this ad hoc polymorphism so you can have a function that's just sort of hardwired to a particular type you can have the parametric polymorphism like Java generics and you sort of type parameter you can tag variants annotations on that but there's another interesting way to go where you actually sort of specify that there exists some implementation of a companion or helper object so sort of a canonical example from the Scala standard libraries ordering so the ordering in the standard library exists for a bunch of the basic data types and then sort nobody your data type does not have to subclass ordering or order it or something like this you can just bolt on this ordering companion object after the fact and this has some really nice advantages in design of kind of a loosely coupled system and it can be a lot nicer than wrapper class or subtyping so in machine learning land what is really nice here is made me to use it for utilities or libraries which are just not quite polymorphic they need some information about the type or some kind of helper companion object so feature vector so maybe you have some domain objects like their lucene documents or something could be anything and you want to specify that they can have feature vectors computed for them so a machine learning world the kind of canonical representation is often fixed length feature vector so instead of sub classing or wrapping this external data object you can just create something that knows how to compute a feature vector from it and now you've got a nice loose coupling between the thing itself flowing through your pipeline and the thing that knows how to compute the feature vector and that allows you to write really really general utilities and libraries so this classified doesn't have to be a classify of loosing documents it can be a classify of any type T as long as somewhere in my code base somebody knows how to compute a fixed length feature vector from it so another kind of example in is cross fold validation and so this is a really really really common machine learning pattern where we break up the data set each column here is the data set we break it up into four pieces for example then we train an algorithm on the three blue ones and we test it on the one red one and then we repeat the procedure with a different red sub data set under test each time and this can give us some really interesting information about how well our machine learning algorithm is performing and the raw plumbing for computing these cross folds breaking up the data set into these nice pieces itself could be super general and not really have to worry about what the data set items are themselves it can just be tea but if we want to do stratified cross fold validation where we're kind of satisfying that each individual fold has a nice mix of the labels so let's say there's kind of unbalanced data set it's eighty twenty eighty percent are not spam and twenty percent are spam and we want the folds to sort of respect that so we don't end up with a fold with no spam fold with all spam and things like this so now that super general cross fold validation utility is not going to work because it needs to know how to compute the label so rather than having to subclass everything we'd ever want to do this with some rapper or subtype that's label label we can just have this nice type class pattern and just say all this cross full stratified cross fold validation code cares about is that I can compute a label about it furthermore circling back to the property testing these type classes often admit like a really nice representation in terms of property testing so there's a interesting library called discipline for sort of making these easier to write the code at the bottom is some kind of basic Scala check but let's say that ordering back to ordering orderings in the Scala standard library and it just says compared to tease two things of type T which one is bigger but there's nothing in the type system that's going to prevent an intransitive or partial ordering or something with loops or cycles or something kind of goofy like that and so let's say that we want for whatever reason we want to be really sure that our ordering is total that it abase transitivity that if a is bigger than B & B is bigger than C then a better be bigger than see this is a very natural thing but it's not captured in the type system so what we can do is we can create a property test over whatever type we want that's going to automatically randomly generate a bunch of data and then validate that this transitivity property holds for our particular implementation of total ordering and so this kind of thing can be really useful when you decoupled your interfaces and functionalities in this type class pattern that gives you a much smaller and more focused thing that you can point the property testing machinery or other stuff again to have a little more confidence in what's going on so mono is is another kind of very useful functional programming scala pattern that we can use in the machine learning context so let's say that we have experimental results from our machine learning algorithm and this in terms of true positive false positive to negative false negative if you're evaluating machine learning classifier these are kind of the raw materials that all the other evaluation plots and metrics and everything are going to be computed from if you ever see an ROC curve or accuracy or false positive rate all precision recall all this stuff can be expressed in terms of these four numbers how many of the true positives do we identify and so on and so forth and so we frequently going to want to manipulate and combine these and the mono I'd pattern from kind of functional programming world gives us a generalized way to do that so instead of writing custom logic code attached to that data point we can have a separate type class that just says I know how to combine two of these experimental data sets and in this case for this and I also know what an empty one looks like so 0 and append so results 0 is 0 to positive 0 to negative so on and so forth and we append by just doing element-wise addition and actually I believe with the shapeless library this boilerplate II kind of mono I'd implementation could be automatically done but I didn't dig into that too far but so this now we just have a companion object that says giving to experimental prediction results I know how to combine them and I know what the empty one looks like and once you have this in the type class world and with implicit sand scalzi so this is actually the scalzi implementation of monoi door definition you can use something like this and what we have here is a list of experimental results mapped by I don't know test or control these two different keys that specify what algorithm we're testing and so we have a list of maps of experimental results and by just having that mono aid code in scope and some other implicit we can just automatically say it all up and it's going the Scala compiler and the implicit resolution machinery is going to say I know how to combine two results I know how to combine maps if I know how to combine their values and I know how to fold up a list and so in this kind of one liner at the bottom we can just combine arbitrarily nested data structures if we know all we had to do with supply this machinery that says I know how to compare combined these two guys so this pattern is also super useful for distributed computing and so Twitter algebra i highly recommend checking it out and also they have a bunch of property tests of these kind of things there's an interesting paper about the sort of efficiencies that you can gain and distributed computing so if you can phrase your data processing problem in your combination of data structures and data sets in this mono aid setting there's a lot of kind of general machinery and optimization people can do including this is a kind of just a fun paper from ICML two years ago where this professor was able to get cross-validation very kind of shocking number four hundred times faster than Mecca and so you remember cross validation from before we want to break up the data set into pieces and then trained on most of the pieces and test on the remaining piece and so if you were classifier itself you can express am on right over it you can do it with this prefix some trick so what we're going to do here is you train a classifier over the first data set the first and the second the first and the third the first second third fourth and then likewise with the suffixes and then any of the individual cross folds you want to run can now be expressed by just combining two of these train classifiers because we have they have the monoid property and what kind of classifiers have them on own property and everything like this I'd highly encourage you to check out the paper but a lot of sort of Beijing exponential family models based on method of moments sort of admit this representation which is really nice so a lot of cool stuff you can do once you sort of look at your data in this three this mono a lens so auditing computations with the writer monad so let's say we want to understand a multi class prediction and this was work originally done in our code base by a colleague of mine at sumo logic REI ajit and let's say that we have two classes so class 0 the mean is for they're both representing four dimensional vectors and let's say they're just iid Gaussian class 0 is minus 1 100 those are the means of each dimension class 1 is 1 minus 100 again these are the means of each dimension and we simply classify by taking the arguments of a vector inner product with some weight vector and so under the hood this is actually basically what a multi-class logistic regression or you know so multi-class SVM some representations so this is just kind of a very simple model for how we might do multi class prediction and let's say that for whatever reason we've learned wade's that actually have spurious weights in the third in the third and fourth dimensions so they think our learned weights are wrong in this case they think that there are nonzero weights associated with the final two dimensions for purposes of this example so the predicting again kind of written in this sort of type class style maybe we have we get the label we get the features and then we do predict by taking that arg max and there's predicted and so this is just sort of the very simple prediction code that might be somewhere in our code base this is an alternate version of the code written in the for do syntactic sugar style so it's really basically a bunch of nested mono monad flatmap calls and you can see that we've added two calls to the end which are admax tracking and add aunty tracking and what those are going to do is basically do the arguments vector inner product prediction but also keep track of which dimension made the largest contribution to that classification so which of the dimensions we were looking at had the largest inner product value or largest product value that basically rolled into the inner product what and to zoom it out of our little synthetic world and into an actual machine learning application you can imagine that it was a spam classifier that would tell you which word was the highest weight word in the decision to label that email as spam or not spam so this kind of feature Diagnostics is super important to debugging testing and understanding how your machine learning model is behaving and it's not something that's going generally in the published research paper or any of you know the kind of the fun stuff but it's super valuable when you're trying to understand a debug why is your model working or not working and what we get here is this kind of confusion matrix and so these max sig case classes it's basically true label predicted label max feature and you can see that at the bottom most of the data kind of clusters in those second and third features which is zero based are those last two features so the spurious features so we can by tracking that information through in this very kind of synthetic contrived example we're actually able to identify the spurious dimensions which were misleading our classifier in the cases where we did not make the correct prediction on this test data set so how did we do that again I said the the writer monad and I'm just going to do a simple drawing version of this let's say that we have three functions one that goes from nothing to blue circle and empty list one that goes from blue circle to green triangle and lists with a star in it and one that goes from green triangle to red square and a list with a smiley face so if we write it in this do notation and we kind of wave the magic right or monad wand over it what we'll get out of this whole thing is red square and a list with a star in a smiley face so this is the writer moan yet and what it basically is saying is there's that first value is the actual kind of value and the list is sort of the metadata that is associated with the computation and so this pattern allows us in totally type-safe and nice way to kind of audit or track computation as it flows through a function pipeline so we build up a pipeline of functions and we can just in a purely nice way track the metadata through it so this is potentially really valuable tool for adding this kind of feature auditing or other kind of things to a machine learning pipeline so the another example here is stateful traversal and so here we're going to assume a very simple model where we want to sample from a piece order autoregressive model this is kind of a very very simple time series model where the why the observation is the sum of the weighted previous observations going back p x plus some random noise term so this is a really simple time series model and one way that you could implement it is sort of drawn there where you have the wise up at the top and blue you have your fix this is a to order you have to a parameters that are fixed and then you have the red noise terms and so they all pump into some function f which basically looks just exactly like this mathematical function and then we dump that back in the Y now this is kind of a this is one way to do it but we can look and we can notice that we could just say that we only we want to make it into a function that has input and output separated so we want the input to be the previous window the noise term the weights the output is the new window and a new output and looking at this we can realize that the weights are fixed so we can kind of do currying or partial application and we can just fix those so now we have this and now we have a sequence of noise terms and we have window window plus noise in new window and output out so what we can do is we can map again we can partially apply and map this function over our noise terms and we get this so now we have window in window and predicted output out and this is an alternate representation of this synthetic data generating process that now has this particular functional form so we've kind of whittled down the original way we thought about doing it to this and what this will allow us to do is express it in terms of state monad again this is kind of loosely cribbed from the scholars e-library where is sort of a wrapper for a computation that goes from state to new state and output so we have a whole pile of those again it's nicely type checked it's functionally pure and we can easily kind of test and reason about it and the synthetic data generation we can imagine maybe it's going to be plugged into some property test or something like this so now that we've represented that we can do this so this what's this doing again we're going to convert each individual noise position into a state calculation takes window in window out I'll put out then we can do something called traverse sequence which don't not really going to dig into but it's another type class like mono ID or monad and you can read up on it in the scala c library as well but it basically allows us to commute the type signature so we have a list of noise terms that we've turned into a list of stateful computations and we can flip that and now we have one stateful computation that outputs a list of results so we've turned our data generating process inside out and then we nearly we just say this is the input window you have to come start the model with some kind of zeros or something like this this is the initial state and just run and we'll get out the entire generated sequence of data points and so this again if there's any kind of mistakes or anything like this the compiler will blow up in your face and it's very easy to kind of track the types through and we've sort of outsourced the plumbing for how our data generating process would work to all of these type classes like monad and sequence or Traverse so the actual logic of what we had to write the custom function logic for stateful synthetic data generation is reduced only to write a function that takes input context and outputs a new context and the output and once we phrased our data generating model in those terms we can hand off the rest to the scholars e-library and the type class machinery and all of that good stuff so the mono is in the monads and who cares that why do we do all this so these are really ubiquitous patterns in computing in general but especially in machine learning and data processing applications mono aids come up whenever we want to generalize combine some data structures and the monads so we only kind of went through the writer and state but there's a whole bunch of other cases as well they basically represent computation within some context and so these patterns you're probably they're living in code all over the place in some implicit form one way or another because it's just sort of part of processing data but if you make them explicit you can kind of reap these rewards by capturing it in the code in the type checker everything like that so if things are not lined up properly you don't you're not relying on kind of human reasoning that I know that this means this and this means this there's the interface the Monad interface and there's the plumbing to stitch it together and it just works or it doesn't you get all this wiring for free there's optimization opportunities so in the distributed case you know if you have something in a mono I'd like context people have thought for a very long time about how to compute sums essentially efficiently and quickly and easily and recompute how to do this in a distributed context so whenever you know just like kind of in the spark and scalding world whenever you're kind of separating out your essential business application level complexity and the machine learning models you want to learn or whatever from the infrastructure in the plumber and the plumbing and the wiring you can kind of let the plumbing and the wiring do what it does best and sort of identify opportunities for optimization and things like that also just from a design and engineering perspective machine learning code for all the reasons we've discussed earlier can often be potentially very complex very hard to communicate about there's a lot of specialized things going on and how you are trying to communicate with other engineers or data scientists or anybody about your design choices and what your code does and what's going on if you just say you know this is basically I have a mono eight over this or we're going to put it in a monad context like that assuming that they're also familiar with what you're talking about it's a very nice crisp clean common vocabulary for talking about these engineering patterns and design decisions that can kind of also help you limit the complexity of your machine learning and data pipeline code so that is the case that I'm making here is that machine learning potentially very complex you really really the marginal cost of additional complexity can be really high so it's potentially worth investing some time and effort into looking at where functional programming techniques can sort of help control the complexity spend and this is all the way from sort of basic correctness and working with small really well understood functions loose coupling among them and then these mano and monadic kind of design patterns so these all can be kind of used as tools to help make your code really easy to mix up and match up and try other things and iterate and experiment and understand that it actually works and so when you have some kind of production bug of some crazy data set that your classifier isn't handling properly you don't have to worry is it a bug in my processing code is the debugging my evaluation code is it about getting my feature code is it the world has really changed as a bug in my optimization routine is my statistical estimator not meeting the balance there are so many things that could go wrong that the more things that you can be reasonably pretty sure that the compiler would have caught the type system would have caught the property tests would have caught the unit tests would have cost or you've out sourced it to type class-based wiring in Scala Z or some other library you can really just focus and worry on the domain at hand and the actual essential complexities of your machine learning application which are probably quite numerous as well so that is everything I had to say thanks again for listening okay yep is it possible to flush this out in a book that's a really good question books are very hard to write I've heard there we go no what no promises yeah they're going up throw in the type system see what it can prove oh thanks