scala.bythebay.io: Chris Fregly, One-Click Deploy Spark ML + TensorFlow AI Models
Recording: scala.bythebay.io: Chris Fregly, One-Click Deploy Spark ML + TensorFlow AI Models
I you alright this is going to be all demo base okay so there's I think there's three speakers at this time slot so I have the first 20 minutes and I don't really remember the title but I think like just here is that from a notebook and i'm using Jupiter in this case I typically use Zeppelin just because I it supports Scala pretty well I've been working quite a bit with more and more data science groups and they all have this mental barrier with Zeppelin and Scala and so I've been kind of shifting and it was a scala conference of course but and i'll show some Scala code but as the back end but from like the actual user interface I tend to get more engagement from people when i use tools that they use so real quick about me Chris fregley have this company pipeline I oh we're building we're focused more on extending pipelines out to production so like a lot of people think of their ml pipelines kind of ending right at the right like training step so we're going to train with in rightly Jupiter using spark but will also then right like take that convert it to pee FML so there's two options we can convert to PM ml and then ship the PM ml out to a rest endpoint that can read the PM ml and take inputs from rest so I've got all that wired up i'm using netflix open source for that I used to work at Netflix Oh have a lot of experience there I used to work also for data bricks that was my job after Netflix so I I got to see you know sort of both sides of these pipelines and I'm trying to take both of that experience and you know combine it and create these high-performance servers serving up so there's p.m. ml will also if we have time I'll show cogeneration happening where we're actually going to take the same spark model and generate Java code and ship that Java code out the source code and then yes on the fly those servers will rightly compile the code and then swap it in and then all new predictions will start to run through the new code so that's actually the focus for q1 of 2017 for my company is going to be still supporting p.m. ml right because there's a lot of companies using it but actually generating native code generating Seco generating hourly GPU enabled code so I've got a tensor flow demo as well but we're not going to have time to do that so but yeah just know if you got a pipeline dot IO you'll see all this and this is all like totally open source just to make that clear I sometimes forget to say that people think this is a vendor thing yeah we don't make money I burn a lot of money but yeah we don't actually make any money so i just recently as a mockery of my old company i'm into mockeries came up with this kind of fun demo called data sticks it's kind of like data bricks you can get to Jupiter notebook Zeppelin you go give you your spark clusters you can use air flow presto we can look at the kubernetes cluster this is all Cooper Nettie's based it's all dr. Cooper Nettie's so this is all reusable pipeline iOS deliverables are right like docker images that that you guys can use I just got back from the kubernetes conference this week in Seattle sort of the next level of packaging for Cooper Nettie's are called helm charts and it's if you think of it just like package management right that but for specifically for Cooper Nettie so haven't gotten to that level yet but because I just learned about him this week all right so here's an example let's just get right to it so again the context here in fact let me show you guys the cluster so I actually have three different clusters I've got sort of a training cluster see if this works I've got a training cluster that that is where r at like Jupiter and these tools live that's separate from the serving cluster and yeah I've got a surprise for you guys which is I'm going to deploy to both google and to Riley amazon yeah so that was something else to I noticed when I was yeah I switched to google at the beginning of the year just because the prices were cheaper and I didn't really have any reason to use any of the amazon services i just needed correctly VMS and like containers and i'm gonna show to that when we start to actually load i'm going to run a load test against both Amazon and Google something I didn't expect but yeah google for the same price thus the same instance eight-core 50 gig is twice as fast when it's actually scoring so I didn't expect that this is not I don't work for Google I don't really care but at some point this was talked about quite a bit at this cuban Nettie's conference this week you can actually in real time you know shift traffic over depending on spot prices and things like that that's kind of you know the goal I don't have that in here now but instead of looking at the number of like requests per second going through like you can add a second metric that's keeping you know price per request right and then you can shift over and yeah so cost savings are like throughout the day kind of thing it's kind of fun to think about it's it's a bit of coding and a bit of feedback loop some things but ok so again all this if you go to if you go to pipeline i/o or go two days sticks calm that there's a fork me over here you can fork and get to the main project what you'll see is this this project is actually broken up into many different projects and these all basically map-21 tool which is then one docker image right so here's yeah this is Jupiter hub things like that okay so is that clear what what I'm trying to do here going to deploy a model and then we're going to serve it up the closest competitor really is things like cloud ml by Google or the azure ml these services where you use their tooling to rightly generate the model and then you you know click something and say deploy it as a rest endpoint right that's really the segment of the market that we're focused on of course we do support all the pipeline's leading up to it and out of the box were supporting spark and tensorflow those are the two that we see the most we produce and consume p FML yeah yeah I was with you at the sensor flow thing with ya what's up man okay yes and the project that we use is the JP FML project and they it's this guy v lu yes out of estonia i was just up there actually spoke at the Estonia meet up so yeah I got to chat a lot of those guys so I'm trying to work with him on some of the licensing it's you know covered under estonia law and he's you know anyway but yeah here's there's a tiny loophole with his license we can get around so we can talk about that but yeah he's doing a really good job I'll show the performance I mean it's 2 3 millisecond scoring's for the decision tree that that we're going to use here so all right so I don't yet have nice pretty buttons and things like that to deploy i keep getting I keep trying to do that but just haven't really got around to it here's a quick glimpse at the cluster so i'm going to show you guys this pretty cool tool here it's called we've works this is kind of visualize so this is specifically this is just the prediction cluster and this is on amazon so this is cooper Nettie's the amazon version i think i was having problems with the gcp version here but this yeah this is right like an open source tool it's called we've works you can actually from here hop right into the docker containers which is really cool so you can actually go in here if you're trying to like debug something or figure out you can click this just to get the logs or or you can actually hop right in you can pause you can look at you know the environment variables it's actually showing top and things like that homie Zuma has you know some some top stats for the different processes I write like try to stick to one process / Riley docker container kind of a docker best practice but yeah this tool really really helps out quite a bit ok so let's go back here so this is just to kind of explore the surroundings going to switch over here this script gets a server to Google we can look at what's running over there the end result is us we're going to be using this particular dashboard I'm not there's no load coming into it now so but yeah there's a like docker container that I'm going to deploy that's going to start up a load test here in a sec but first let's actually build the model okay you guys see that I'm pulling this data set I just put out there i created an s3 bucket called datapalooza it's got the popular our census data set takes a bit to you know load spark and that kind of thing but yeah this is the code here so I'm using our formula right like again just to appeal to some of the our heads out there we're going to predict income from all these other attributes and we'll see those here in a sec when it pulls in the data and we're going to build a tree we're going to build a pipeline so write like this is the whole key to what we're doing here is building a full p.m. ml pipeline so not not just the decision tree itself but right yes any feature like transformations that would be happening any normalization standardization the JP FML spark ml package let me just show you guys this because it actually is pretty integral so the way that we're currently relying on p FML I I have not used that one yet but yes that yeah that's totally another option as well so this is yeah this guy blew it if you just look at the what this is I guess like an organization he's got a lot of stuff here yeah so he can take your scikit-learn and then convert it to p FML our guys got XG boost support this is the one that we're using today because of spark just to give you guys a quick glimpse these are all the features so there's kind of this folklore going around the you know p FML where like I because this project has been there for the last few years people have kind of you know popped in and out of it and at the time it was not full-featured but I pretty much haven't had any problems all the models that we build this is pretty much everything within spark the feature package within spark these are all supported here's the classification here's the regression you know clustering things like that the thing that is glaringly missing here is recommendations because that's how right like a lot of us view you know machine learning yeah that's okay typically when you're generating recommendations your populating tables are populating caches that's all Netflix works they do a big offline and for every user that's the key in this database then they write you know the top 500 movies that they're going to show the next time the person logs in so that there really isn't much benefit to write League representing that you're you would just basically be building up a database in p.m. ml which is kind of ridiculous so for that actually there's another path we use that's part of this relic data sticks thing here where we actually populate Redis and then have distributed regice up there and which is very similar right like again so how Netflix does it and then we have the rest services that are in front of it that that call in to Redis okay so I can't mean to talk to you actually so we will talk okay I five minutes so let's get to this looks like did a collector lot of data okay so here we actually build the model there's this cool method called to debug string if you guys don't know about this this is actually going to print out the tree in a in yeah nice pretty printed form so you can actually see the split points and things like that kind of gives you some human gras cable way to think of this stuff here we're going to convert it to pee FML here's where we're actually going to do so there's two ways to like deploy this P FML you can take your existing cluster and push it out so there's a rest endpoint that would normally be hidden but here it's public where we're going to swap in the PM ml that gets generated so that's so here's the pretty printed model its decision tree classification model package or the class name within spark here's the the split points these feature numbers mean something you know stored yes internally by spark the like p.m. ml knows about these when it gets generated so let's generate the PM ml it should print it out there's our favorite we got the XML right like yes everyone hates XML and here let's deploy it and what did I get here to do G open okay so yeah so that would normally deploy it but it's already so the second option is we would commit it to let's see if I can see this should work up that didn't work I didn't test these two right before okay the so the second option is you actually write it out to disk and then commit it I changed a whole ton of stuff here still fun I thought you would commit it and then once it goes into github it would trigger a docker build and which would slurp in this new model and then deploy it and it's set up to deploy as a canary right like alongside so if you have a thousand current servers running and then now you're going to deploy the canary and the the piece that we're working on is the tooling to give data scientists from the actual right leg notebook itself to actually see the production canary and see how its performing right like not just system metrics but also performance metric or prediction like performance metrics right like alongside the other nodes and then from that point then they could scale it out so we could scale out here so this is actually this is working out some of this stuff's a little bit janky here but let me start the load test and we'll flip over we should start seeing some load pop up here so these are our live so yeah this is basically where things would have come in this docker container would have been deployed this is just one node right now so here's google on the bottom and here is this is amazon on top this is the performance right now there's just once there's one server out there we can scale that up here in a sec but yeah if you notice i just want to show you guys single node performance here let me so we got google on the left amazon on the yeah we have amazon on the Left Google on the right you could see the times here okay so it's a bit similar right now but about 3 millisecond for the 90th 11 millisecond so but yeah we're pushing right like almost 2,000 like requests per second and actually this is limited by the load test because it's actually flattened out there so and then with one command we could scale it up scale it out is it time okay it's time and we could actually see on our nice little weave scope thing here we could see containers this should scale up here in a sec we'll see see four of them come online i believe or five total maybe so yeah that's it that's at the bar for everybody