ai.bythebay: Eduardo Ariño de la Rubia, Developing and Deploying Models at Scale — without Spark!
you [Music] solo my name is Eduardo ateneo de la rubia and I'm going to be talking about developing and deploying models at scale without spark and I'm also gonna be talking about some other things with some feedback so real quick I wanted to humanize myself before I dehumanize everyone else so this isn't part of my lovely wife she's the one in the relationship with the doctorate I do not have one even though I'm the chief data scientist of a company so noodle that one those are my three dogs I really wish I likely would have asked me to come talk about dogs and how awesome dog training is but he didn't so here we are talking about building malls with spark and on the left is my very first computers a sinclair ZX spectrum which had 16 roaring kilobytes of RAM literally everything a bird getting data scientist 1982 could possibly want just a little bit more about myself a little bit more self-promotion so I've been doing this for a really long time I wish I would have known it was going to be called data science because that's a way cooler title than like nerd who likes math which was what i called myself for about the first 15 years of my career i got my first account on a supercomputer in 96 I hired my first machine learning engineer in 1999 because I was really convinced that if you took customer buying data and you like rammed it through machine learning you could do interesting things with it I was right obviously didn't work out because I'm here talking in front of you and I have a job so like I obviously like didn't cash out on it like I should have but turned out okay um since then I've done a lot of things lots of real measurement science and manufacturing and logistics and robotics ended up started using Domino in 2013 and became the chief data scientist in 2015 I was literally just a user and advisor for the first couple of years before are you giving me the timing cues okay excellent real quick i'll try not to do much product pitch but it's literally my job so i have to do some of it so Domino it's an open platform for data science a real target is like if you've got a team of data scientists and you want more and you need to scale what your data science team does we do that real well you get to use the code you already like in the language as you like I think we would have failed if we tried to invent a new statistical programming language because there's already brilliant people doing that you get to get data however you get it today and you get to use like the best tools like our Studios really fantastic ID for our or like h2o is the definitive machine learning library like you should get to use those if you're a data scientist and we had a whole bunch of stuff to it like compute automation and Environment Management and I'll go a little bit into what that is you'll just see in my live demo because that's what I was asked to do is a live dinner so I'm going to build and deploy a machine learning model i'm going to be using h2o dot deep learning is the first thing that I'll do and on an x1 instance and I'm going to build an employee next one model on 10 million rows with 128 course hopefully that wasn't what happened already here today but uh so real quick this is one of my favorite slides of the data science process it was written by a guy named dr. tulog paske you should look him up his work is just fantastic and stellar and quantitative we're going to pretend like understand collect explore vision and transform are just easy we're just going to go ahead and pretend like those are not the hard part of data science and artificial intelligence they are they're really really hard and that's the part where you get into arguments and that's actually why I'm doing a master's degree in negotiation and conflict resolution because that's literally constant negotiation and conflict resolution we're going to focus on the modeling component and one specific thing I want to say is that this is the large model that on the bottom right is a thing called crisp DM just by a show of hands who here is familiar with Chris p.m. ok a few hands it's the cross industry standard process for data mining which data mining gross data mining turns out like the same set of problems were solved a long time ago and they invented this cross-industry process we should if you're doing data science e AI e type work you should probably be following it um so we're going to build something and this one's going to look like we're going to pretend like building an AI model building a model is the hard part we're going to use up 128 course we're going to use up to tera bytes of ram and we're doing on one machine and we're to leverage a little bit more as a large work it's you know you probably don't have big data like just sampling across all of you here like there's probably like four or five of you that actually have big data but data sets are increasing in size about twenty percent year-over-year that's based on a KD nuggets pole while in the same exact period of time where data sets were increasing in this for analytical data sets we're increasing twenty percent year-over-year you might have ram available inside of like single instance machines was actually increasing fifty sixty or seventy percent each year I mean I mean it's not so hard to figure it out you used to be able to get a box with 60 and get the gigs of RAM on AWS and now you can get a box with two terabytes on rahem on AWS that's a significant increase you know there's a before I go into my demo there's a great quote you know if you are applying what field you know which would you rather use to strong oxen or a thousand twenty four chickens and this was see more praise famous quote and I was funny because like during the late 90s and early 2000s we all felt really really smug because like we were like oh no it turns out that you want to use a thousand twenty four chickens uh well you know maybe Seymour was on to something because give me a choice I'd actually use 120 a toxin that pretty much an infinite number of chickens so just a couple of things we're also going to deploy a model in my demo so I'm going to literally from the just build using hto steep learning on the airline data set I'm going to build a safe learning neural network and then I'm going to deploy it as a restful api and it kind of cracks me up I keep hearing people be like you can't use our in production and I'm like oh you can't have you like try they're like well no and then it's like well okay so we're hiring a bunch of Python people I low because you know we want to use Python and immediately they're like but I've heard that python is too slow for real production anyways and then that's not true either then you then you hear that people are like oh it's really hard to port models to like Scala or Fortran or see or whatever we actually run in production it really doesn't have to be that way you can just you can just port the model so let me get to my live demo real quick let's see if this works you'll see my screen okay so let me make the font big enough that like humans can see it okay is that that's good enough all right so we're going to go so this is so this is Domino Domino is a web-based platform inside of here whenever you land inside of a project you can think of it's sort of like your contextual notebook each one of these executions happen in a fully isolated container and it's nice because you can like see the code that executed and like you can reason about it just look at the file itself I have this file called trained r which is actually taken directly from the large repo just small modifications basically we're going to in it we're actually only going to use 50 gigs of ram so we're actually not even using that much RAM you know I'm just going to go ahead and execute this to spawn this thing off all right so while I'm spotting that off let's look at the file real quick right so import some files do a little bit of normalization train a deep learning model go ahead you know get some performance metrics because you want to know how fast you know what I mean what the performance is go ahead delete the old model save a new model create a diagnostic list while this is going on hopefully I caught it in time I'm going to go ahead and copy this SSH command over to this terminal and then ssh over here and this is pretty cool that'll work for me right now I mean that's like a hundred and twenty eight cores that are processing away like right now again this is like there isn't a weird internode communication it's not super duper slow i didn't have to worry about loading data inside of our dd's i didn't have to worry about like that my EMR cluster like get screwed up because like my Kerberos ki got stomped on I'm just I'm literally just using like open source are and like the open source h2o library and I'm leveraging a hundred twenty eight cores to train a deep learning neural network model and like what's really cool I don't know if you can see this right here each ear has been executing for 25 seconds but the main threads that has an hour and 10 minutes of processing already assigned to it right because of the way process trees work inside of Linux the parent process gets all the time for all the child processes so this is just happening right here which would which is pretty cool huh if you look at a lot of analyses you know the average size of the average spark like job is like 60 gigs 60 to 100 gigs if you do the math you can fit 20 of those on an x1 instance and like why are you paying the weird cost of spinning up instances and RD DS and dealing with like spark and scale and you're like AI people would rather just be using Python or they just rather be using our unless you have like a really really compelling use case for why you need to pay the weird penalty um i really suggest you didn't so that's going to be done here in just a second oh yeah so it's got about you know it's only how 2 minutes and 30 seconds feels a lot shorter when you're preparing a talk and when you're standing in front of all y'all waiting for it to train so this will be done here in a second and what it's going to do is it's going to actually spit out a model and I think I showed you in the code where that's going to be I'll just go back to said here's where I just remove any old models and here's where I save the model one thing that's nice about Domino is that Domino has fully automated version control so literally every time you say every time you modify the file system we keep the deltas so even if in my code i'm deleting the model at any point in time I could just they're actually I'll just go back and show you like here's what it looked like a three o'clock these are all the times I was testing this to make sure it would work right so I could go back in time to any of these and I can literally see the entire project state at that time so that's a nice thing about having everything run inside of a single centralized system of record so okay so it's done it trained it I could look at the code right I can see any Diagnostics here's the auc i can i went ahead and plotted it hto does a nice job making like their models nice leap laudable right so like it just kind of all works with our you don't have to like worry about any weird like spark plotting libraries that I don't even know if those exist so let's go ahead and like publish a model so in Domino we just go to models and so I already pushed a predict delayed flights model so this is our new model management system this is actually literally the first time that has been shown to anyone who isn't a domino employee um so would we basically let you do is we let you take any code here I'll just publish a new version real quick so here I'm using the airline model project that's the one I was doing in it's like trained this in front of people I am terrified right now and let's see so I don't need the data set right I literally just need the model file and a file called a PIR and it remembers that there's a function called API I should I show you that book ed chill out for a second so to show you what it takes to actually publish it is really straightforward so I created a file called a PIR and it just like call the h2o library loads deep liar initializes sets no progress bar actually funny printing out a progress bar takes a surprising amount of time and then I just declared a function so there's a halt our code right and I just said there's a function that takes a list called data it has these parameters in it I create a new h2o data frame I you know I predict I turned it into an our data frame and return the prediction like there's no resti stuff in here there's no like magic in here this is just like straight up good old-fashioned our code and so here I tell it it's in the file called API door it's a function called API and publish so it's building it's starting and it should be I like this as build building I'm going to make fun of them for that so it succeeded so what Domino does is it actually like creates now a docker image it is like a serialized docker image so that like let's say that you're a bank and let's say that your models like we're denying loans to people of color and awful lot and like a regulator shows up and they're like hey can you prove that like this like your model isn't racist that's hard probably what'll isn't racist like it's actually like kind of a challenging thing and yet I think that as data scientists like we all need to do that I think we all should be really really focused on proving that our models are not like sexist or like misogynistic so uh one of the things nice thing the Domino lets you do is it lets you go you know this is regulator mr. regulator I can go back to the project I can go to the exact image and show them the code that trained it so that's all there so it's instant model succeeded let me go ahead and deploy the version sorry so I'm going to go ahead and deploy it just to show you all there's all kinds of nice functionality we've added to auto scale so we can set like the hardware tier set the number of pods we this new model management framework leather cooper Nettie's either of y'all apart about Cooper Nettie's but it's just a really really nice way of like managing fleets of docker containers I have no idea what threads per pod does or anything like that but i'm sure the tech people know so here we are we have this model and i'm going to go ahead and call it so that close my x term I did what a goofball all right so i created a little perl scripts because i'm very very old and I still write pearl and so it's just a perl script it goes through the same airline csv file that i trained and just goes ahead and calls on curl right so our model go ahead and close that our model just takes data as a JSON parameter so I'm going to go ahead and call that so clear / gonna call fairline all right there we go so it you can tell you know every time it updates you know that's the number of milliseconds it took to respond I could deploy a new version and it wouldn't drop any any it wouldn't drop any connections so this is like our in production anyone who's ever been like oh you can't like put are in production you can put our in production it works fine you just need to like build a DevOps harness around it and like you can build your own dead locks harness or you can use Domino there's like lots of really fantastic options out there uh so yeah so let me see so I have a few more minutes but that's basically the need of my talk the meat of my talk was supposed to be deploying a large model without using spark so what have we done we've trained a deep learning girl network we're using a box that has 128 cores we have trained did we've deployed it in production we're hitting it with a REST API I didn't eat spark I didn't need any any languages other than languages that data scientists know and love so that's the first part of my talk fortunately I prepared more talk because otherwise you all would have nothing to do for the next 15 minutes so hey there's an AI conference right AI school but what if it's coming for you have you heard of this guy Randy Olson has anybody here had heard of Randy Olson okay hey I know you hi so Randy Olson is is fantastic he you've ever heard subreddit data is beautiful he's one of the moderators of data is beautiful and he's created a piece of software called pee pot teapot is a automated machine learning tool that being said so I actually given a talk at strat about automated machine learning tools HD OS coming out with one that I'm going to be I'm just super hyped about this is just the one that was currently released that I could show you right now so what teapot does is you give it a data set you give it a target column and it actually like genetically evolved an entire machine learning pipeline for you you don't really need a data scientist or anything like that so I'm going to go ahead and just do that so let me go ahead and spin this up I'm going to go I'm going to rerun this because it's important to actually like show y'all stuff running instead of promise that it did ran at some point so here it is it's spinning up inside of Domino I want to go ahead and secure shelter the box this one is not as exciting as h2o it doesn't use Korres nearly as effectively I mean it does all right right you like see some usage but then like the usage you'll drop off it's not as good but it still does a pretty good job it's honestly it's hard to beat how well h2o does this particular job so what's happening right now is a t-pod let's actually look at the code it's just this much code so basically you create a teapot classified you import the t-pod classifiers you read your data set and for this I'm using real small data set because I want it to be done in a minute and a half so I'm literally going to evolve an entire machine learning pipeline from data and then deploy it as a restful web service in about two minutes right so it's just a small data sets the Pima Indians diabetes datasets kind of one of the classic UCI data sets I'm saying hey this is the column that I want do a test strain split and create some some kind of classifier I don't know what kind of classifier it is you know you can use 120 cores there's gonna be 10 generations of them you know and go ahead and actually like fit it and then from there actually going to print the score and it's going to export the pipeline so I'm not doing any machine learning right I'm not actually like doing any feature engineering I'm not doing any of those things I'm literally just saying hey teapot here's a column here's data set here's an x one instance that has 128 cores in two terabytes of ram just take a while and evolve a machine learning pipeline for me automatically this doesn't mean that like data set the data scientists are going to be like out of a job tomorrow but this is something that's important to be aware of the bar is moving one of the things that I'm going to talk about you no not at this talk but in a future talk is that the bar for automated machine learning keeps moving when i started doing machine learning you just like downloaded some c code of some like dudes ftp repo and you like compiled it and hope to turn into an art to network and then we like God carrot and that was amazing because like it did all the resample for you and then we got h2o which is incredible because it does like this in credit search for you and it like finds the absolute best hyper parameters and like now there's these things that are coming out like teapot or like the auto ml stuff that like the bar is moving and I'll talk about that here in just a second so here we are it's you know so it's calculating the different generations this is what I love about like live demos in the previous one you'll notice it took a mitad in 24 seconds fifty-two seconds a min and 51 seconds here we are two minutes and 30 seconds and not quite done yet so my timing was a little off mr. okay excellent so it figured out that for this data set it turned out that a you know linear support vector machine with you know hinge loss function with l2 with like this was the one minute founded was the best one and it actually like generated it as code for me like here it is like here's the code it like it does the strain test split it exports the pipeline it fits it and then I just wrote a little wrapper than add to this a little bit of code this API features so let's go ahead and publish this so I can go to models again there the teapot model versions I'm going to go ahead and publish a new version this one I just pushed some and the AI did it for me and yeah I do want the data in this case because it trains the model upon deployment it teapot just generated the pipeline here is the pipeline and I go ahead and publish and building starting again it's making the docker image and we can even look at the audit logs we can see that a new version was created I'm the one who created it build is successful let me go ahead and deploy the version and all right well that is deploying I'm going to go that's not a good window to show this one all right clear reset shouldn't I call it so i have another script and a perl script because I'm old again teapot through more than it does the same thing only this time it calls the URL for the teapot web service and there we go in this particular case it's much much faster if you remember the previous one was a couple of hundred milliseconds per score deep learning neural networks are much more complicated thing than a support vector machine but here you are like you know I like I as a data scientist I didn't build this model I didn't train this model I just uploaded some data into a thing I like pushed a couple of buttons and it like built the model like like built a classifier it like deploy did it for me and now here it is and like anyone could access it so yeah so I'm rapidly automating myself out of a job which is pretty cool so I've got a few minutes left I was going to take this opportunity to talk to you about something about completely different things hey so there's an AI conference and so my wife has a doctrine in in education and particularly in curriculum design and we get into like humongous like loving arguments at my household about whether a I can learn anything of education the answer is yes I've been it's been explained to me that yes we can and there was three points she wanted me to make is that people working in a I need to understand how assessment really works and we need to understand that there's two types of assessment there's a formative assessment and there's a summative assessment when you're creating machine learning models artificial intelligence you want to make sure that you're doing formative assessment during the learning process the kinds of questions that you're asking our fast questions you can answer real quickly that you can sort of go through your hypothesis space incredibly quickly so that you know that you're learning methodology and your training methodologies pointed in the right direction and then at the end of your deploy you're going to want to do a summative assessment right that's when you actually like you compare it against some standard or benchmark if you're not doing both of these things inside of your sort of data science practice or artificial intelligent practice then you're leaving some you're leaving some learning on the table and you know I often see that data scientists may I researchers don't do this they primarily just do summative analysis and just look at like diagnostic statistics and call it a day the other thing I was going to say is Bloom's taxonomy is pretty interesting computer science has been stuck for the longest time at remember you know that's what we focused on we focused on actually like having data structures that made it easy to remember a thing or an even simpler examples you know the entire field of information retrieval the entire reason you all are here I'm guessing at AI by the base because you want your machines to start moving up Bloom's taxonomy right you want them to be able to understand apply analyze evaluate and if you've looked at some of the amazing things that are coming out of the field of deep learning even create right we used to think that creativity was sort of like this like bastion of like human-like kind that like we would never be able to create machines that did it for us and yet like we're on that precipice right now like that's happening so keep you know keep this in mind both in the questions that you form yourself in the hypothesis space that you search when you're building an AI as well as what is the hypothesis space that you're asking your AI itself to answer and finally yes I curriculum turns out it's a thing turns out that whenever you're training your your AI on a data set you're actually providing it a sort of like almost like a unstructured curriculum that data is the curriculum and that's the explicit curriculum that is which is written the right so in that data set you're sort of pushing forth like the ideas that like the data generating process that is reality in the sensors that you had sampled and wrote it down that is what you're training but that's not all that there is right there's also the implicit curriculum that is you know that bad which is unwritten this is the thing that's not gonna be able to be replicated by others you're not even me to be able to prove that it exists this is you know the canonical example is that if if all you're all your cops are racist then all the data sets say that black people you know commit more crime hey you just got racist cops and your data sets themselves are like this mirror on the bias and that's like the implicit curriculum as you know I think we have a responsibility in this community to be aware that the implicit curriculum is larger than the explicit curriculum and that we always have to keep a skeptical eye towards it then there's a null curriculum that one's easy that's the things you're explicitly not choosing to test but if there's one thing that you know a lifetime in this space has taught me is that the null curriculum quickly becomes your explicit in your implicit curriculum as you realize that like the questions that you want to ask go higher and higher and Bloom's taxonomy so hopefully that wasn't too terrible ah we're hiring if you know any Skala engineers like we've got lots of really cool interesting problems and we're hiring them like crazy uh yeah our entire stack is written in skala everything you saw him Skala so hit this website and lastly hey thank you very much and I hope you enjoyed that Cheers [Applause] I'll do my best to provide a salient answer I just wanted asking maybe I wasn't listening I apologize um you and shown us this example particularly when this in airline data set can you speak a little bit more about what that data set actually was yeah the early data set is like a canonical fantastic data set it's the ASA's 2009 data challenge you might remember correctly so it's a it's a cool data set because it has it you know has numeric features it has a categorical features it's a real real world data set and what is it's basically just like the takeoffs and landings of like millions of planes and it's you know it's it's a it's a big enough data set that you can't like non-trivial II just sort of like blast it all into a thing and get a reasonable answer but it's not so massive a data set that you can't like do some clever stuff and do like a demo where we like we train on all ten million rows like right here in a two and a half minutes using h2o so it's like it's a fantastic dataset like if I would have shown you iris like half your eyes would have rolled out of your head you really owe the IRS data set it's just a really good data set for figuring out on time departures and that sort of thing cool any other questions all right well thank you very much you guys cheers you [Music] you