Fantastic ML apps and how to build them

scale.bythebay.io: Matthew Tovbin, Fantastic ML apps and how to build them

Recording: scale.bythebay.io: Matthew Tovbin, Fantastic ML apps and how to build them

so today I will talk about how to build fantastic machinery in applications all of you or some of you are building machinery in stacks at your company's same as we do at Salesforce okay I'll try to speak closer to the microphone so better so I would like to start from a quote the slowly seen that galaxies like dust is what most of space looks like the sentence is normal the richness of our whole neighborhood is the exception let's take a moment and appreciate our our habitant and we're actually really lucky that we have such a rich world around us especially as we have so much interactions and data and things happening as and we keep exploring so this this quote actually comes from the documentary I wished a while ago called powers of 10 and then this little documentary it's like a ten minute video that the travels from our our habitant like a starts from 10 to 0 and zooms out to the you to the observable universe and then zooms in back to the level of quark and it kind of gives a great perspective of how fast the world is gives and understanding what what actually the power of numbers and the size of the errors I like to draw an analogy similar analogy with the machine learning when we build machine learning applications we have a lot of levels of complexity that we have to go through that we have to cope with we start from data collection where you have to identify which data source is what kind of data we need to collect then how do we store it then we think about how we prepare this data for different machinery and use cases and eventually we get to the feature engineering aspect of it after that we have much more levels to go through right when we engineer feature want to select what are the best features that maybe make our models faster and performant we think about sampling we think about different algorithm implementations some of them are already there some of them are not and so on right for each algorithm we might have a lot of parameters to tune and we have to think about all of these little things and experiment with them this creates a lot of complexity for us this is by no means an exhaustive list so you can add more things there for example benchmarking or model quality how you guarantee it across different deployments when we roll out models so I would like to ask you like a question how long does it take for you guys to build machine learning application is it hours I don't see my chance here at days like maybe here weeks months so many more or you guys think it's it's more so on the average it takes months it's like this this picture is like a black hole it's a black hole that sucks all our time it takes all the time of all our teams and just sucks it in we start experiment and we start Union arguments parameters I'm not even talking about the deployments right cycle Hadoop parameters or spark parameters so how do we cope with this complexity is there like a magical formula or maybe some some magical method that that allows us to to deal with this this stack of problems that we have that so we can actually make it ours we actually want to shrink from months to hours we don't want to spend much time we want to experiment faster and move faster with with our without theories right because when we build machine replications we actually make a theory and I want either prove or disprove it by by proving the correctness of the model usually we disprove it and we move one but this experimentation has to happen faster so as a software engineer we all know that the task of the software development team is to engine the illusion of lissa team we want to make it simpler to to complete the work with complexity without without too much hassle so I'm as a software engineer kinda this is my main challenge when I build machine learning applications is how to make sure my my team is successful and my company is successful eventually so what does it mean right so how do we engineer this illusion we we build abstractions well build some kind of we just layer complexity in this boxes of abstraction like writes from left to right let's see let's look on this image right of bears in the woods like a rational painter you see a lot of details but what what the what we really want to see here is that okay some there's a beautiful morning in the forest and bears are just naturally behave the way they do and then we can enjoy this and then move into the right the level of abstraction increases and we might get think of our system as a black box right and it's not that useful to think about systems of black box because they might not be applicable well in different scenarios so the cope with complexity we need to find some proper level of abstraction the way we can easily argue between ourselves and within our teams so it actually boils down to something to a simpler problem well it's not that simple but it's at least I can explain it in one slide how do we find an appropriate level of abstraction to deal with those machine learning problems as I mentioned feature engineering featured feature selection and so on when we when we talk about abstraction right especially in software we're really talking about language so how do we express our thoughts when we talk to a machine when we write out the requirements for the application so language language syntax and semantics state they define the the degrees of freedom of the engineers of the teams the way we express ourselves so if we feel like look on the two two parts of the spectrum the higher degrees of freedom are will get basically lower abstraction think about when we just see your assembler it's it's extremely it's it's suitable for much more complex problems it's more difficult to use right and it's it's error-prone why because we have to we have to repeat ourselves every time we do some relatively simple operations like a we can barely reuse reuse some of the stuff we can we'll have to work to do around it with a higher abstraction such as maybe you can think of a maybe higher level dsl languages even sequel actually is a it's a good example right this might be less less flexible but it has much simpler syntax to remember the learning curve is lesson and we get a bell reuse of the functionality that we do so similar to machine learning right we think of a higher level concepts may be like tf-idf or one hot encoding or we talked about some algorithms so we're gonna do this decoration of linear regression this thing's we can write them down in one word we don't have to write the for loops every time so there is not a code I'd like to mention from Martin Adair ski that says functional program removes one important dimension of complexity to understand a program part of function you need no longer account for the possible histories of executions that can lead to that program part so this is very important to us because all not to us is I think to us as a human I guess though to understand that the importance of functional programming is that we can argue faster on on the library and application development as we go so if we if we use this ability to a functional program this property of functional programming and apply to build machine learning application I think we can get to a higher success here so in a functional approach right we have I kind of wanted to mention like top five probably perks that we get right everything becomes typesafe we don't see any side-effects when you look on the function would get very composability and it's the syntax is very concise fine grained control as well so to get so just in this example on the right this is an actual quote from our code base maybe I would like to extract some features from the URL and I would literally write a couple lines of code here okay I'll just create a URL I don't think of exceptions because there are no exceptions it's so properly safe and I only think about their input and output types of this function and then I can I can use it to map over values very easily so going forward right we also know that there is a another approach rights of the corrientes approach where we get three more perks of modularity quad reuse and polymorphism which were also extremely important when implementing data data reach functionality as we require for machine learning when we deal with different types of data and operate on different inputs so in this example on the right I also kind of explain what it means right so I'm just showing some polymorphic properties of a tokenized function let's say I have a sequence of some values and I would like to tokenize so depending on the type that will tokenize differently in this case base64 encoded string will be first for example decoded and then tokenized and maybe for phone numbers the phone number will be parsed correctly and if it's an incorrect phone number I might get some other same default that won't make my machine during pipeline break so which actually leads us to the next slide which which contains like a little why Scala yeah so we actually like scale and we use it all around our stack because it combines all those things I mentioned before as functional approach an alternate approach it's a strongly typed language and it allows us to build very concise concise language to cope with the complexity of machinery floats it's mostly fun unless you hit some dark corners which we try to avoid and then it's it's just a joy language to use and of course its default to spark which allows us to which allows us to scale our pipelines without rewriting the code base after experimentations like P our data signs actually experiment in Scotland's Park directly they don't they don't do our models and then we rewrite them it's not what we're doing so we actually went ahead and we developed a small library to allow us to to cope with this complexity and like make it much nicer to deal with the feature engineering feature selection model selection and so on so it's an automat library that we call Optimus Prime provides us very declarative and intuitive syntax to work with machine learning what I like the most is like it gives us very higher accuracy with with a significant reduction in development time when I mean development time it's actually prototyping phase proving disproven theories and also deployment of production all those cycles we try to cover with with doing one framework so just just to give you like an example of what what actually opieop rise contains right so it's not it might not it might look as simple but it's actually underneath we had to build we had to be a lot of types so this is like one example of the of our type Araki that the part of Optimus Prime that allows us to to deal with the complexity so for example we we decided level that we wrap all the value types into the feature type and then we identified three major types out of its like a text American collection and we went forward with that despite that the layer key might look complex so that took us a while to figure out exactly the the connections and it's actually doesn't contain everything it's not exactly up to date but but it allows us to express ourselves better I will show now some examples of that so which which leads us to this slide where where I'm just showing the how we get the type safety for value and feature operations when we deal with machinery in flows especially for spark so we're currently aiming for spark use cases but you don't have to don't think about spark I think about maybe think about pandas like pen the data frames which are some metrics of data and columns by by rows and so say I want to write a function tokenize for for text it's literal simple as this so I would maybe take texts and then I would map map over it and do some split and create a text list out of it when I operate on on the not on the value but now for example on the on a whole column of the data right I don't know how many rows I have I might I'm getting myself into a higher-order type such as feature right so we're going on one level up but we operate the same way as we do for for the value types so I would do I would create a type of this like feature builder the text with some extract function and then say I want tokenize I would do title dot map and then pass the tokenized function so it actually gives us very good flexibility to express ourselves as well as unit test all of the code that we built because because the code doesn't really changes from the value to feature operations it's all the same little functions just need to correctly combine them and then when we build a workflow for example in this case which is just a set of stages of transformers or estimators that operate on the data it's also very simple for us so let me show you actual actual example this is the whole code you have to write for example if you would go and and write on a predictive application that would predict a book price let's imagine for a moment that you like a dataset of books and you have four fields in there author title description price and I would like a predicted price based on this three three features such as author title and description so I would go ahead and define the features from my data set and then I might do some feature engineering on them I would combine title and description together tokenize by removing some punctuation and then naturally I would might do some TF an IDF to get some to get some popular tokens out of it then I combine it all together into vector type right because we all have to get to some vector type to run any any model init regression model and after that I spin spark so for example the two lines of spins Fergus can essential for spark whoever familiar this this is what you have to do and then I would create a model selector that would go through possible model combinations and hyper parameter T in Union phase and will eventually speed up a good model for me right so it was like this train function at the end that's what says okay now go go run this thing there is no mention of spark constraints or cluster sizes or anything that that is kind of quite a burden to write when you develop spark code for example or maybe when you develop in some other frameworks such as tensor flow we also need to remember to do some caching and like make sure you some of the data is correctly cache so it will perform there is no mention of the flow here it's actually just that just a logic that you want to express so there is like one magical short here that I want to explore a little bit further and second of vectorize it's like what does it even mean to vectorize say I take tf-idf data and author data which is a pick list in this case right pictures the second set of values what does it even mean to vectorize what's what's the default behavior so we built this factorize because we know the domain of machine learning and we took our best experience best effort to try implementing and this is what a lot of teams do as well so just kind of to simplify the machine learning flows what what vectorize might do so depending on the type it will apply a correct behavior for that type which most of our most of the machine learning engineers do right if we say if we see a feature of type email maybe we want to do top K domains from that email or if we see a phone number we want to do phone number validation and then extract the country code because it gives us better predictive power rather than just a simple phone number value or if it's an age such as like a numerical feature right we usually packetize it so why not do it automatically and this is what we actually did we went ahead and and we automated a lot of those little pieces that that kind of were repetitive across across different applications all over again so this is kind of things that you you probably should keep in mind is that your languages your abstraction needs to be able to tell our aid in those does this functionality easier for your frameworks that you developed because then you can just move faster with your application development and for example for zip code I would like extract and in common because it's by no means a full list right so much methods we can add into this framework to make sure that it's that it becomes even more useful so similarly maybe in the cycads right I don't think how I don't think it's too easy for example to extend Sai kit functionality which i think is a fair of their library mainly and for me as a software genius it's very important to make sure that we can add this functionality easy okay so after this vectorize creates features for us we actually need to do some automatic feature selection so we want analyze the the features this is also usually what what happens in the proper machine during application we define features we'll create them and then we one analyze them compute some statistics and correlations between them ensure that that features feature values fall into appropriate ranges and is this feature a liquor for example that leak any information about the label so it makes our model a weak model so here's an example how you'll do it for example Optimus Prime I would simply write price to check which will apply the st. I'd check in for my for the price label against the features so price the check with a feature vector features and I will maybe send some some instructions further for the for the code to do that such as like sample limited min marks correlation and so on another step that goes after feature selection is the is the model selection right we want to pick the some algorithm that that suits to our problem and then run through the different hyper parameters for this algorithm and make sure that we pick the best model against the the test set I don't know how many we familiar with spark but it's not not that simple for example to do there and for us it's actually pretty trivial also you can define it you know in a quite compact way I would define a list of list of algorithms I wanna run through and for each algorithm I would set multiple parameters to run through and then as a developer I don't really care how how it works I just want to write it down so this level of abstraction gives us this ability and so this is actual slide from one of our applications of different different algorithms that we ran and it actually it matters a lot which algorithm you run against which data set so here we compare it like a random forest with gradient boost of trees and like native bears for example which is like performed horribly on this data set and usually people just pick one model that they confident with they kind of that they develop intuition well they kind of think they develop intuition because we're really horrible to develop intuition again especially against statistical data I would assure you most likely wrong when you think you developed anything on this so you should delegate this precise work to compute to you know to machine right so this is why you would you should definitely build a proper model selector for your frameworks I would like to show you a demo obviously guess what it's gonna be about Titanic data set it's one of the most popular data sets out there so can you see this yeah okay I will try to zoom in so we have we've built a little command-line tool for Optimus Prime that allows us to the bootstrap bootstrap projects faster because it's color it's not super nice especially for first-time users right stuff like Python you just open your editor inside typing for Scout need to build like a project build and like put files in the proper folder and so on so I built a little generator but it's more it's more than this products project structure it's also we're kind of trying to take in a data set let's see we will look on the web on the passenger CSV file so this is like a list of passengers on Titanic with their features I would see their name and their age and which Kevin they embarked and and their gender I would like to take in this data set and just generate some say machine or an application that will do a prediction did this person survived or not so what I have to do is I will run the generator so it takes it takes an input a CSV file it takes the schema which just Maps which column which name it has has an ID I define the idea okay ID is a passenger and then I'm gonna call my project Titanic 1 so it just asked me ok what kind of problem is this it's probably it's a binary classification right so if I can survive it or not so that's it so I get I get my my project I'll go to the project I'll probably look here so get to the app and I'll just run just a regular sparks of it I'll remove the quiet part so we'll see it compiling so what's gonna happen now it will compile the code and I'll just just a spark Submit and we'll run it will run and train my model against the same data set I just give it give it let's see what's gonna happen it's gonna probably take a minute so it's gonna run to find some features run some models compare maybe pick a random forest energy situation and eventually okay done and eventually what it's doing it's actually writes out some results for me so it's rolled out that's okay area under curve was 85 which is pretty decent just out of the box and then the model type was RFC this is random forest classifier and then I might scroll up and see what does this model look like and which features were picked in writes like a survived some vectorizer named sex cabin ticket embarked on a lot so this is this is literally took me a couple of minutes right let's let's look on what what this code looks like there's an open this project didn't work maybe because I have to reopen it yeah I'll just close it what see you'll see okay here we go Oh in like this I don't know what happened here you know maybe I'll just reopen this project demos never go well as you would expect them to you might know that how to refresh this looks looks good to me but it's apparently not I'll just do that's a good try yeah I actually didn't open it so it's a it's a it's a fresh project right we just okay let's let's do it from the beginning because I actually want to show you some syntax coolness yeah it's pretty fast to generate so why wouldn't they regenerate it again okay [Music] okay maybe this what was missing okay something I think it's better now yeah and it's doing some syntax highlighting should do the guests of presentational yeah didn't exactly work I'm a kid compiled my might generate so close because we use our here right the passenger type comes from a virile needs to be created by the plugin so this is what this is what Mike what Jen created for me for each feature I have a handle which is a vowel it's typed and then after we have this this features here I'll just keep scrolling right so what this is basically all the features of my data set where is this being used it's probably not Titanic app so this is just like a spark app and here I have some readers for the data and what happens is it takes all the features and that's the vectorize and then maybe i have this label here so it saves a lot of time for us to just all this hassle to write this code but also now i can operate on the features for example say you have a name right so I'll say ok I'll do some no tokenized name dot let's say I want to do map I don't know what what time is this and say I won't even create a text maybe I'll tow all the text here and maybe it's a too complex for this IntelliJ to handle I don't know what the hell anyway believe me it works most of the time that's why I don't like showing demos it's just never almost never but what I wanted to show is that you know it kind of pops up the operations you can do any features so let's see which which type is this oh it's a categorical feature that's why okay I see now let's let's look let's take another one let's take age let's say let's do age da so okay so it's identified the type of the field as a categorical so now let's say I transformed h2 via text now and I want to do some some stuff on the on the text so I can do for example talking eyes and I maybe want to do like out of the tech language true and it will it will detect the language of the text and we'll try to to tokenize it for example so this return features is exactly useful usable as other features and I can pass it into the vector at the same pipeline will work so or maybe I want to do like TF right TF tf-idf on this and it's also very very simple to write and this one will return me a vector type and moving down right we'll have some safety check in on the features and then the second model selection here okay get back to this presentation okay so how well does it work actually at the cells for scale it works pretty well most of the models that we currently deploy are completely hands-free they were all auto-generated or partially auto-generated maybe a little bit handy and most of them are hands-free they all train them like in complete automatic fashion we currently serve close to half billion predictions per day and you can read about it and must you know different block block resources such as like TechCrunch and stuff so what's what's your home take away from this it's like a little recipe for you try to define appropriate level of abstraction for your machine during pipelines for your libraries of choice if it's cover not great you can maybe reach out to me and we'll we'll share some knowledge and some library formulas prime because we're currently doing open source pilot so do that if you if you interested in spark and scour and use types definitely use types to express it like that that's why they are built for so they're really good for expressing any work for the data and the machinery try to automate because it it saves your time and will save time for your teams and it will definitely increase the time that it takes four months two hours so for further exploration there was a talk yesterday by Carl's coach and Yan about the real-time ml pipeline so you probably want to work on the recording of that it's gonna talk about how we do real-time machine learning and then there will be another talk today later today from Thomas Gerber on how we use spark there is a function in algebra in Scala today also by blood pattern chef it's pretty interesting twist on how we how we do a linear algebra and a functional way there will be a talk or factual applying this little library that I just showed you for 4k specification and Einstein builder and there will be some talk about DevOps if you're interested about that how we how can we build an infrastructure to support this we're obviously hiring and thank you of question you know yeah we have time for questions if you like [Music] so yeah seems like [Music] and yeah yeah so yeah the great question is the first question in terms of using spark versus Scala and like the diffraction friction between teams on using different languages so we don't have that because we kind of savor on you Scala and that's it like everyone has to it's like it's a compromise right so everyone suffers a little bit so you Scallon period despite the how nice python might be O'Connor so that's why we add features to scour frameworks and spark which are missing maybe two the second question is that yes we wrap around ml liebe lot because we don't want to rewrite say the tf-idf transformer that or work Tyvek or linear regression we're currently exploring ways of doing scoring without spark and we actually very successful on that if you want to look there is some data breaks advancement on that also it's called I think DBM a local so you can look at inspiration of that from there yeah but the idea is basically to export your model in some in some form and that can be picked up but in different language which is Jason it's like simplest way jason with a bunch of arguments and then you need to write an interpreter to perform the scoring of your model elsewhere there is another one I can recommend you're looking onto PFA it's like a portable format for analytics that's one is a little bit more complicated it's like a separate DSL in llamó slash Jason that allows you to express transformers oh it's not maybe it's gonna be open sourced I own onset actually last year we actually actively work into open source it since it's a it's a largest project that we have our open source we had to involve a lot of teams internally like a prod sack and so on so it's happening we're currently open for internal pilot so if you like interested reach out to me and then we can share the quad video before we open source but we plan to do it yeah yeah [Music] [Applause]