SF BA ML: Rajat Arya, Repurposing Existing Deep Learning Models

Recording: SF BA ML: Rajat Arya, Repurposing Existing Deep Learning Models

here we go so I'm rashad I don't have a machine-learning background I'm pretty new to data science my backgrounds distributed systems so bear with me all the hard questions I'm going to punt to someone else so I work for dado I spent a lot of time prior to data with AWS and with Microsoft i joined dado as a first employee so I've been around for a while I spent the first year and a half or so at data working on our production tools I'll show you one of those a little bit and the last six months I switched over to this product role where I spent a lot of time with customers helping them integrate our products into their applications so tonight I want you guys it's after work it's late in the evening I don't want anyone falling asleep so please interrupt me with questions i can make my taco as long or as short as possible so please make this fun me just talking about myself is not going to be that much fun except for me so telling you a little bit about dado who are we well we're based out of Seattle we are a I've only got two slides to tell you about data so I'm not going to tell you that much but that's roughly all of us there about 35 of us now and it's always nice I'm in San Francisco it's always nice when San Francisco weather in July matches seattle weather in july it's not that common but today happens to be one of those days so i'm quite happy so what do we do it data why do we exist it's kind of a simple thing we exist because we believe all apps today have to be intelligent and what do I mean by intelligent if you go back ten years the problems that existed in the world we're about getting all your data into one place so there were a lot of technical solutions to get data into one place do you think about the last five years it's been about traditional analytics and what I mean by traditional analytics are things like a dashboard that says thirty percent of your users are going to turn thirty percent of your users turn every after a month of using your product so you now know that in aggregate but how do you do something about it you get into a room as a group of people look at the dashboard and go oh crap thirty percent of our users turn after 30 days we should do something about that what do we do well let's come up with a strategy let's do something and and then you would take acts and that's not an intelligent app so an intelligent app would do something else intelligent app would say with thirty percent likelihood tony is going to churn oh he was over there he's gone now um hey he's way back there now he turned look at that um thirty percent likelihood tony is going to turn this week but I know I have five offers I could send him and offer number three has the highest likelihood of retaining it and so I'm going to send him that offer so if you think about the difference an intelligent app integrates predictions into the user experience of the app so of course you guys all know this machine learning is kind of the key to making that happen and there's tons of applications and use cases for machine learning so that's why we exist this is the world that we see today and this is the world we want to support so we built a platform and our platform is makes it easy to develop deploy and manage predictive or intelligent applications our goal is that our tools enable you as data scientists and developers to build things faster and helps you create new applications or explore new ideas so what's our platform look like it's got three products we're going to get through this in 35 seconds product number one GraphLab create hopefully you all have this on your on your laptops you pip install this just like you do any other python package an imp GraphLab create we have scalable data structures the s frame and s graph these both scale out of course oh you're not limited by system memory and they're both Columbia are representations for your data making them really fast for analytics operations and lots of really fun compression techniques as well on top of them we have our machine learning models I'm going to kind of walk through deep learning a bit today though not technically I'm just going to kind of tell you about it and our models are again they scale because they're built on these data structures of scale and we offer an SDK to extend our platform so go to production we have two products I'm not going to go into them in a great depth but predictive services allows you to take your model and put a service-oriented architecture and front of it giving you all the tools to manage and operate that that cluster and data distributed it's about taking what you've done on one machine using GraphLab create and running it on a cluster of machines whether that's on premise or in the cloud so that's it on about dado obviously i'd love to i can answer lots more questions about the company but i thought we talked about images instead so i want to get really quickly to writing code because i'm i'm hoping thats what you guys want to see but before I do that I do want to motivate with one quick demo and hopefully quick time is going to cooperate so how many of you have seen this demo before 11 person yeah don't give it away so so what are we doing so what you're looking at here is a really quick demo application that was written using data predictive services GraphLab create and specifically our deep learning model or and so we implemented and correct me when I get this all wrong but we implemented Alex net the the geoff hinton imagenet the the image net model the convolutional neural net imagenet model and took us a little while and that's part of what i'm going to talk about today's deep learning is it is a tough thing to get into and it's a tough thing to to master so even when you took an existing paper it still takes took us a few weeks to to get the model trained in the same way to give us those same results we then took that train model and deployed it as a predictive service to a three node cluster in AWS that's the cluster that is is running and then we put this front-end app on top of it that's just it basically javascript and HTML that's hosted on Heroku so the idea is when you click on a picture in this case I clicked on this dog we sent that picture off to the predictive service to get back the predicted labels not sure if you're familiar with the task of the the image net data set it's a million images and a thousand labels and so the idea is can we predict the label so you guys know what type of dog this is and from our training set this is a German Shepherd and what you see here is the the label along with the confidence of that label so let's try another example and so what type of dog is this do you guys know I didn't know until I ran this demo it's a movie era the flung my French is pretty lousy my high school French teacher would be upset and a non we go so what are we doing we're labeling or were classifying images we're giving them a label a predicted label so I don't know what do you guys think this is so many so much whispering so little saying it out loud alright we're just going to say mask and our training set ok so these sorts of demos can be pretty can so let's search for something instead on getty images so what you guys want to search for haha picture i am there's no people in this in the image net data set so it won't identify people but let's see if i can search for fast car and I have to apologize this demo often will get any image cooperated tonight great so which one of these are going to try let's try this one so I click on the image we sent it off and we got back our predicted labels sports car it's not so bad you want to try something else guys what do you think what's that cat like like that I'm very curious what Getty Images has four cats I've not actually tried that Oh see I spoke too soon Getty Images was awesome for fast car but apparently hates me now yeah really that's so when all else fails we're going to restart the demo we're going to try again there we go now we got some cats which one you guys want how about this one in a box let's see what it figures out here with the kitty image is thinking right over the phase of the cat beagle oh that's good that's good but load notice how low the confidence is it's not which one the one in the tie oh oh my this could be this could be horrible this could be Harl Egyptian all right okay it's not so bad all right so we're doing imageclass did we get a tie in their bow tie um oh my gosh you guys oh no where'd it go come back all right clearly we could do this for like oh that's so awesome oh well I think I'm done here I can't top that for the for the rest of the night so I wanted to so let me go back to my slides but I wanted to show you this is kind of what you see all the time with deep learning you you take some images whoa what happened here whoa wow that didn't show up right let's try that again Wow not that the slides that awesome so if it doesn't work we'll just move right along better so obviously there's lots of work that's been done in text there's a lot of interesting stuff Alexei mentioned some of it that you can start to do with images and I wanted to show you a little bit about that tonight essentially an afternoon that I was hacking around and i want to show you sort of the outcome of that so these are some great use cases so what did we just do what's the demo I just showed you this is a if you if you like you know million foot views of things you take an image of a cat you put it into this magic fat black box and you get back a label that says cat so if that's machine learning what's the workflow look like it's just all completely remedial you guys know this but I'm going to go through it really fast so for supervised learning start with labeled data split it you you create a model you run the model against validation set probably not good enough go back and adjust hyper parameters and off you go so I jumped to my demo early but that's fine so this is what we just did right you saw this whole process happening where we took the app that are sorry you didn't see this happening because the model was already trained but you then saw what how you would use that sort of application to make predictions for images so so this is as far as I'm going to go into deep learning this is a picture of the image net paper of the model of the layers of the convolutional neural net and that's as much as I know about deep learning and so I apologize if you guys came expecting a lot of insight on how the model works I can tell you think I know there's terms like dropout layer and pooling layer and and I at one point I think I even understood part of that but I've long since forgotten it I want to do I wanted to try something that would be a lot more fun so I wanted to push the easy button and not have to learn about deep learning at all I want to take an existing deep learning model that's already been trained and extract features from it much like you would extract features from other models and then apply them into a different domain sometimes this is called transfer learning so that's what I'm doing that's what the rest of tonight's about that's not what you came here for I'm sorry maybe you can take a sticker so transfer learning I already talked about this if you train a model use it for one thing and then you want to apply it to another task that's exactly what we're going to do with with this existing train deep learning model so how does this work back on our workflow our supervised machine learning workflow where you start with label data again this time you're going to extract features from your neural net this is going to give each each image in your neural net and going to represent it in a reduced dimensionality space as a vector of floats we're then going to use those floats we're going to split them into a train and test set because now we have this vector representation for each image we're going to train a simpler model let's say a well we'll use a nearest neighbors model but you could create a classifier again in a different context or something else and now you can validate and this probably works and i'm going to show you it's kind of disturbing how well this works so you deploy it and off you go to dollar signs so I promise potentially profit in the title so there it is of course I mean it's in a slide how could how could I be faking that so how does feature extraction work the slide looked awesome so I took it from my colleague Sean I think the main takeaway for me from this slide is it to extract features at the lair prior to it to labeling so if you think about neural net as a set of layers there's at the I guess at the at the bottom or at the top I'm not sure how you want to orient it there's the step of actually saying okay this this if you fall into this bucket you are a cat so we want to to extract features prior to that because we don't care about the labeling of cats we want to just learn what the model learned where we want to be able to extract what the model learned prior to that labeling that final step for labeling this make sense okay so that's it the rest is going to be about code and so we're going to go from cats two houses because I on a friday afternoon at work I wanted to see could I do this for house pictures so that's what I did we're going to switch over to ipython notebook I hope you guys are ok with that and this is all code so I'm going to zoom in a little can you guys know how about this can you read this yes no should I go further this is fine it's good I got a thumbs up way in the back to all right so I got to tell you what I did and I don't think I violated redfin's terms of service but I'm going to tell you what I did hopefully they don't mind too much so they don't want you to scrape the site but I wanted to get house pictures so I didn't scrape the site I did exactly this I was like well let me look at houses in Seattle and I'm sorry guys that these prices don't all start with a million dollars I know it'll be unfamiliar to you guys down here that there's actually housing you can buy that's less than a million bucks but hopefully you'll bear with me it does exist you just have to come to Seattle so say I pick this neighborhood in Seattle that has 290 listings which is exactly what I what I did scroll down on this this UI and you can download all of the listings and these and what you get is a CSV file that I then used and so I didn't scrape the site but let me show you the other half of what i did so i downloaded these listings and I did this for a few neighborhoods in and around seattle and i'll show you how many I ended up with but if you click on a property on listing you'll see that we've got an image here so let's make this image big and then let's go in and find the image URL so we'll open this in a new tab maybe you guys can't quite see that yeah you can't see that I'll do this i will paste this back over here okay can you guys you guys can see this though right I notice this kind of interesting thing so I'm now connected to the CDN directly this is a photo it says big photo I notice these three numbers line up with these three numbers and these six numbers are the mls listing so the ID for the the overall house listing so I was like well this can't possibly be true obviously they must be something more sophisticated than this in order to keep me from getting all their images and they do something slightly more tricky this underscores 0 I believe represents like if the real estate agent uploads pictures multiple times these Rev so these become one two three but for my purpose I was okay with just getting the original set of pictures that were uploaded so I went with this as a presumption and I went and so I took all my downloaded CSVs and I'm in GraphLab create by the way here so from here on in i'm in GraphLab create which is a python api over our see cost plus engine and i went to load i downloaded a bunch of csv files and i went to load them all and you can see i did this back on june fifth at 9pm 8 p.m. 9 8 9pm i think that might have been UTC I don't I don't know what time zone that was so off I went and I downloaded I took those CSVs and i ingested them and GraphLab create took a look at what I got and s frame is our tabular data structure so it went and parse the CSV and this is what it came up with there's lots of info in here but before I show you the column names let's just look up the number of rows I got which should be the number of so 5660 I can look at the top do a dot head on it and see what I get basically sale type home type address city state zip list price look at that guy's less than a million less than a million 2.9 million more than a million and a bow whole bunch other stuff bedrooms baths location square footage lot size your bill parking is it on the market days on the market bunch of other interesting things I didn't care about original list price and the big magic listing ID which is that mls number i was talking about which let me go and get some images for each listing so i did that before I did that because I I was doing a bunch of searching between different neighborhoods in Seattle I'm sure that I got duplicates so I wanted to collapse my duplicates so I did a couple of group by and adjoin so I'm in 5660 and then I wrote some pretty heinous Python code mostly because I was writing it to download these images so taking so given a URL go and download that picture if it doesn't already exist on the file system so i could cash these and then i wrote this little helper function that would go and reconstruct the big CDN URL though i noticed it's different today you guys notice it was HTTPS something or other and this says HTTP media done maybe they changed something who knows so and then I went off and downloaded and so I wanted to verify that for a particular mls listing I would go and get all of the pictures for that listing so download images returns the number i downloaded so 24 pictures or this might be the ID i think this is the ID and so i don't know if you guys are familiar with redfin but you can upload 25 pictures for each house so far so good yeah wheres great we're getting some data this was the the goal of the first hour two hours of my activity I wanted to not actually run these functions i just wrote it's a simple applied function so it's an arbitrary lambda or anonymous function in Python that i'm applying to a column of the s frame and since the s frame is immutable i'm assigning my result back to a new column that's created so off this went i downloaded a 60 7758 images so not too shabby for you know a couple hours worth of work on my this laptop this one you're looking at and i was running out of disk space so at the same time i realized i have to go and clean up some stuff have to ditch my music collection and all sorts of other stuff that was sitting on my machine so so far we've we've downloaded images and we have listings and so that's sort of the the raw data so the first step of trying to find similar house pictures is just getting some pictures so we did that so now we're off to it's actually using them so you guys can still read this right yeah I'm not going to run everything that I did mostly because some of the steps do you take a long time on a laptop they'd be super fast on a machine with a GPU but my laptop doesn't have one so I'm processing images like three images a second system with a GPU can do 200 even an older model nvidia cuda GPU card can do 200 220 images a second so so ipython notebook again i'm going to import GraphLab i'm going to tell GraphLab that hey when you're visualizing stuff just do it in line in my notebook so the first thing I got to do is I want to call I want to load all those images that I download it so if you guys were paying really close attention and the previous notebook I downloaded everything to a cash to a directory on my file system called images we can go take a look at it if you'd like it's a little big so it might take a little while but there you go so you can see these are all the this is the mls listing and the number of the image and the version and i always got version 0 so those are all my images and so I've got a I need to load them and I'm loading them into GraphLab create I want to create an s frame out of these images so GraphLab creates supports text numbers complex document structures like JSON and XML has column types along with images and so I can so I'm loading in images I'm saying that I you know figure out the format it's a JPEG or PNG you guys the the system can figure that out and recurse the the directory structure so if I had lots of subdirectories inside that I wouldn't have to run this more than once then this table that got created I want to add a row number to it so that i have so i have an ID for that for that image now the image net model if you're familiar with it it worked on images the image net data set is 256 256 x 256 x 3 meaning 256 x 256 each image has dimensions 256 x 256 and then three colors so RGB values so so we need to do the same one of the things about neural net models is you must your model if you want to use that model it you have to make your images be exactly the same the same dimensions have the same have the same number of dimensions so i'm using this resize functionality this is one of the things that is quite slow unless you have a GPU so I'm resizing all of the images I had and I'm assigning them and I'm creating a new column for the resized image and then I'm renaming the the columns because I don't want it to be called resize I want it to be called image and I want the original one that was called image to be called a ridge and then I want to go and get the mls listing out of the file name so I'm essentially again applying a lambda with a Python path based name to go pull the first half and the filename off because I'm going to use this later and then I want to save this s frame so everything I'm doing I want to save it back to disk the main reason oh go ahead I certainly can so the question was our s frames local or can they be distributed and the answer is the best way to describe best frames is they they start local and they can grow to distributed and you'll hear a lot more about that if you come to our conference on Monday and so but yes the way to think of this is that this is an out of core data frame so it's writing to your local disk now your local disk could be HDFS it could be s3 so you can choose where you'd like s frames to write things out but it itself is not a distributed file system so if you have HDFS or another distributed file system then you can save using that where you can also we support a you can stripe you're so s frames are clubbed are so each column is written to a separate file like most data warehouses like most calumny our storage systems and as such when you can specify a series of file paths where you'd like your s frames to be stored and this way you can get striping across multiple disks or helping you get to if you have distributed file system so you can leverage that so far so good yeah any other questions no okay so I saved this s frame and now I didn't really want to make you guys wait for all that to happen so I'm just going to load the 1i already saved and so I did I'll do it again just so you guys can see that it's instantaneous because again it's already saved on disk we're just loading the indices effectively so because you have 40 1531 images in my nest frame and when you look at the the head or the top 10 rows you see I've got the path of the file the original image dimensions the resized image dimensions and I've already extracted the features because that takes a long time as well but that's the next thing I need to do so the very next thing I need to do is go and grab that deep learning model so that I can take the train model and extract essentially tell that model to extract features from images that it has it has not seen so instead of asking it to predict I'm asking it to extract features from these images so you can download this model we've we've made our image net model that ran for 45 iterations available and so you can go and grab it so I already did and takes a little while to loads it's a pretty big model but once it's downloaded you can load it really quickly so I've loaded it let's take a look at it you can see like I said 1 million images 1.28 million images I'm not really I don't care about my target column because i'm not using the model to classify i'm just using the model think of it as a feature pipeline so i'm essentially wanting to run through the feature pipeline and take the output of that pipeline prior to labeling so now I did that so now I have the model so let me explain why you want to extract features so if you look at even a 256 pixels just 256 x 256 image x 3 you have about 200,000 dimensions so that's a lot of numbers to represent a small picture but when I extract features from my neural net I take that same image and now I'm able to get a fingerprint for it that's only 4096 or 4k dimensions so this is a much more manageable number so again you get dimensionality reduction when you when you extract features so how do you do that it's as simple as pre-trained model or whatever the model we just loaded thought extract features pass in the column that has the features that has the images you'd like to extract features from this will return you an SRA which I am then assigning to a new column of my s frame then I save it again so I want to show you what that looks like this took a while I think on my machine this took a while a few hours I don't think I i think i was i think this took about two two and a half hours so be prepared for that and what you see is now I if I look at one of my images so I'm looking at the first image and I'm looking at the features column I now have a vector an array of floats that represent this image it's not interpretable at all but it is a lot simpler to it is a lot smaller in terms of data than the image itself so I've got this so now I have extracted features so now I have this numeric vector of numbers that represent each image so I'm going to throw that into a nearest neighbors model so that i can and nearest neighbors is going to compute distances between these vectors and that in but with that with that representation the model will then help me query for the nearest images or nearest items from any given item does that make sense yeah okay so we're gonna do that we're going to run this one go ahead yeah it's going to tell us it's Euclidean and it's and we're using brute force so this is all just out of the box like I can I can change my distance function if I'd like and I can use ball tree instead of brute force and when I poked around with that it didn't make anything any better than I could tell so it could be because i only have forty one thousand examples you know it's not but so here we go ten seconds I've trained my my neural net now with brute force a lot of the heavy lifting happens at the time you query not at the time you're training the model this is essentially just putting all pairs together you still have to iterate through them to find nearest neighbors so let's find nearest neighbors so first things first let's let's bring up a sample image so what I did here is remember i had the images table the images s frame so i want to get one row out of that and so i'm using python slicing so 250 is inclusive I want essentially the 250th row and I don't actually couldn't I just do this it's totally fair what's let's oh tell me more about why I didn't need the land of function oh yeah well done guys I totally hadn't thought of that yeah okay so what happens if I do this I think I get back a dictionary which which is fine but if I do that do I get the same thing back I do not it's not a list yes the so I'm getting back this I think is a dictionary for one row and this is a Python dictionary yep this is a dictionary but when i go for slicing I I get back in s frame that's why because I wanted it I one in s frame so good it's a good question so the question was a Howard features selected out of the image and I don't have the the whale describe it is all pixels are interpreted and what the model what the image net model the image net model does a good job of identifying colors shapes and textures within those images so that's actually one of the things that's really amazing about the image net model is that it generalizes incredibly well so it was trained on very natural images the million images are very natural setting images it's not to say that it can work everywhere but I'm pretty surprised the number of applications that how jet how generalizable that model is so in like I'm looking at house pictures there is I doubt a house picture in the image net data set and yet you'll see how we do does that answer your question any other questions this one here that I'm showing this this row this is an s frame a graph lab create s frame of the dado esperan oh yeah so the idea this is why I'm slicing because I want to get back though I only care about one row I wanted to come back as an s frame go ahead nope we haven't touched the image net model other than to use it to take each of our images it's a great question so the question was are you did the nearest neighbors thing influence or manipulate the image net model the answer is the image net model we sort of aren't using it as a model we're not asking it to predict we're actually we're just telling it to use it as a data engineering pipeline so given a raw image chew up that image and spit out a vector of floats that represent that image given what the model has learned what the model has learned in part so we're never actually asking a model to predict we're only using it to try and absorb what it's interpreted from from its training so this is why it's the easy button guys it's a you know we're basically we don't have to know anything about deep learning and we can start to make something kind of kind of neat so what i'm doing here is a i'm getting that one row and then i'm going to show you dot show the original image so now i'm showing the one the so this is a picture i randomly picked out of that data set was like ah this looks interesting it's a kitchen let's see what's most similar from our data set to a picture of something that i think is I mean there's a kitchen right you guys are great with me like there's nothing in the data set that indicates it's a kitchen other than it's just an image and when we look at it it looks like a kitchen so we're going to go and ask our nearest neighbors model we're going so this was the nearest neighbors model we train now we're asking it at to predict so we're saying query this image and get me the 10 nearest neighbors to it and tell me what you're doing as you're doing it so off we go this is by the way all this is running on my laptop I should have made that really clear up front so everything i'm doing here is on my laptop and when i'm doing something not on my laptop i will let you know and this is by the way no fancy laptop at all this is like an eight gig macbook pro of very average proportions so eight seconds and we've gotten our ten nearest item so you can see for the query label which is labeled zero these are the reference label so these are the row numbers of the images that are close so obviously it returns itself because that has distance zero and then you can see everything else that's pretty close starts at 41 and moves its way up to 43 so you guys want to see see what we got so before I show them to you just remind you this is what we started with what are the 10 nearest things to that so that's what we started with this one the MA what we got back is the 10 nearest pictures look like this I'll just scroll through them again so you guys I can't get them to show up all together so I have to scroll slowly so they are ya they should be yeah looks like it yep it looks like the same kitchen but it maybe it's not i mean i don't know all these high-end kitchens look the same to me i have opposed to describe it they all have the these high ceilings and the the lighting that drops and the granite countertop and the yeah it does have identical light fixtures now one thought is maybe that's the same neighborhood i didn't i didn't go investigate I could actually I'm are these what's that it depends what neighborhood you're in you could definitely get this if you if you step away from Seattle and go 20 miles east for less than a million so this was kind of exciting to me I don't know if you guys find this exciting but I know you guys know as much as I you guys now have heard everything that I know about deep learning which is very little and yet i'm able to take advantage of a train deep learning model and actually do something that looks kind of interesting so this this was really exciting for me and my immediate reaction after i saw this was i got really lucky there's no way i just got lucky with this picture of kitchens that return kitchens like this was a this was a fluke so like any dev I think would I was like all right I'm gonna go get similar images for every one every twenty-fifth image out of the 41,000 and I'm going to store them as a dictionary in Python and then I'm just gonna that's what I'm going to do and then I'm going to go and look through some of those to see if I have any was this a fluke or did this really happen there's also pretty late at night by the time I got to this point so I was like I'm just going to leave this running go to bed next morning see see what I got so what I did is I wrote my little tiny for loop which correct me if my Python code looks awful it could where I'm given given an X I'm going to get that row as an s frame going to query I only asked for the five most similar images from this to run this faster I don't care about the seeing what it does anymore and then what actually is a pretty expensive step is this joint because of the image type and the way the nearest neighbors the nearest neighbors models just returning the IDS so I need to now join those that table that I got back with my images table and then pull out the the 10 most simple so that's that's what my joint is doing and then I'm sorting it by distance that I'm going to have those in in a in sorted order and then I'm exciting this to a dictionary and then that then I essentially can go through and show that dictionary this was what so did I ran I'm not going to show you all this because it's as you can tell kind of meaningless but the important thing I got out of it the next morning is I went and started poking through these numbers and started writing down you know labeling some of these to see if there were any good and some of them actually turned out to be pretty neat so then I was like okay well that was fun now I've got something it seems kind of interesting so the next thing I wanted to do was I was sick of waiting for that join that join was taking like a second for each each image on my on my laptop so I thought I'm gonna let's just put this up as a service I actually don't need to do the joint because I can reconstruct the URLs because I already know all i need is that mls listing number and the image number and i can just go and pull the urls directly and so why don't I do that and so I wrote a little function to help me do it and this is where i'm using data of predictive services i'm taking my trained model wrapping it in Python code and then deploying that as a service so now i can just query the service with a an mls listing and an image and get back the 10 most similar images so i wrote a little lookup function that just was a smaller s frame with just the ID and mls number cuz i need this to construct the URL and then i wrote this little Python function that became this is my service function so think of this as my service endpoint so again the idea is I'm taking a listing number and the URL and I'm going to get the 10 closest ten nearest images or ten most similar images and return them and return the URLs for them so that the client could then can then go and fetch those URLs directly and render them so I'm doing all the same stuff I did before I need to download the image so that I can then resize the image and then extract features on that image then I'm going to query the nearest neighbors model this time I made k equal to 11 because I don't want to return the image that was the input I want to get the 10 closest not the nine closest and the original and then I'm going to reconstruct these URLs doing some string manipulation in my in Python and referencing this s frame to look up the reference label because I again need to get back to the ID it's kind of a detail there and I'm going to return this as a JSON block where I'm returning a dictionary so that a Python dictionary that gets converted back to JSON and each entry in that dictionary shows me the rank of the image and the URL for it so that's my service and let's run it for a random listing and image oh right ipython doesn't know about my my function yet let's try it again so I'm just calling this function locally just so that I'm testing it this is you know what you what everyone would do so here we go my function ran and I got back rank 1 2 3 4 it turns out this was a horrible format I did a terrible job of making this API easy to interact with but so be it and I got back URLs so now let's so now I wanted to take this function and deploy it as a service so this is data predictive services I've got some boilerplate stuff on how you get started here but you set some AWS credentials because we're launching an ec2 you say what region you want to go to where you want your log files to get written the instances instance type you want to run and the number of them you want to have in your cluster and then it's one command to kind of launch the cluster this one I named Yelp funny which I was using for a different demo and so this is how you would launch your own predictive service I don't want to wait for AWS that takes a while to spin up you know three four instances could take ten minutes I don't wanna make you guys wait so we'll just load an existing predictive service and that's as easy as passing in the the place the s3 location where the the state path we're all the state for the predictive service is there and I can add to add the endpoint I just I just give it or to add this function the get similar images function which we defined up here and make this a service it's as easy as deployment add give it a name and then the function and the name here represents your the rest endpoint so this becomes part of the rest this this is the part of the rest api for this model or for this service so does easy is that we call apply changes I'm not running these two real time because this model is a little big and so it's it takes about 10 minutes or so to upload but this is the first time something's going off of my machine so I told you guys I'd let you know when something happens off my machine so when I call apply changes that's when we're actually taking on that function along with all of its dependencies or all the things that encapsulates and we're packaging them all up using a super awesome version of pickle that we extended and then pushing them off the s3 and then notifying the nodes in the cluster that and at this point there was only one node in the cluster though I think I've added a couple more the that there's a new model or there's new a new service to load and that's and so the service can get loaded so now let's query so that you guys can see what this looks like so again I picked a rant an MLS listing out of random and then I picked a random image so now i'm making a REST API call i'm using the Python client that's baked into GraphLab create but you could use we have a Python client java client JavaScript client you could write your own and curl it's a really really simple rest api so all I'm doing is making this rest api call and getting results so you can see I when I do this I'm connected to the load balancer that's sitting in front of a predictive service and then i connect it to one of the machines well since since you guys are all here I might as well let's look at what my predictive service has so I'm just looking at my predictive service it's got an API key this is a this is if you want to make query requests you need this and then these are all of the services that are deployed along with what version the service is at so how am i doing on time that's hard are you guys okay okay because it really we're at the point where I'm just going to show you a bunch of awesome examples and so that that's where we're at and so and we could try and find some more house pictures I was looking for like mansion pictures the last night so maybe we can try and find some of those so I saw it so here's my output that now running as a service so I query again I get results instantaneously predictive services come with a built-in distributed cache so queries that you see multiple times are cached the results are cached but this is not really fun to look at this doesn't demo well at all so let's write a little function to help us actually view the images and this is how you know you had a bad output format when this is the amount of work you have to do to get to something that you can see so I'm unpacking and I don't really want to go through this other than write a better service function that doesn't return things so cryptically and you don't have to go through the shenanigans of trying to make it so you can see something but now that I have this show results function I can pass it a set of results and we should be able to see so these are the images so now I've kind of got all the machinery I need right now I've got the service deployed so I don't have to spend the time to do the joint in order to see what images are most similar and I've got the functions to help me visualize the results or show the results inside ipython notebook so my next steps if I was to keep going with this I actually want to there's more features to add like I could I didn't add any text features yet at all certainly I could especially things like square footage number of bedrooms baths like those are those aren't really text those would be categorical but there are definitely text features i could add as well that I'd want to play with there aren't captions as well it'd be great to see if I can leverage them I don't think redfin makes those available so I'd have to talk to them probably but since they're in Seattle maybe they'd want to talk to me and so so and so there's more that I can do but this was really fun to just try and get started so I thought I'd end with just some of my results so some of the some of the the fun examples so i wrote a couple more helper functions here just to make it even easier on me so I wrote this get RX function and then i wrote this other function so all i have to do is pass in the index value from my my list here so let's start with bathroom I think bathroom is kind of a fun 1025 let's do that 1025 this is kind of what happens when you're trying to build a UI inside ipython notebook we're here we go here's my original image so this is my picture of a bathroom so what are the most similar images to this aha so I so just to go through this i set the idx the index the index this idx value gets passed into get args which returns me a dictionary which has now taken the ID from the ID value it's gone and gotten the MLS value and then it's gotten the last three digits of the MLS and pulled off reconstructed the URL from where that image came so I'm doing gymnastics to go backwards from where I started so that i can now pass this to the function to the service because that's those the rest endpoint makes a lot more sense to be given an MLS listing and a URL then to be given anything that references the training set so just this way I can apply it to two images that have the service can be four images that the training that the training set never saw so I need to in order to use it for this example I need to kind of unwind what I've done and then I go and call the the API the service I get back my results that ugly JSON and then I I called that show on the original image for that so because i'm showing you the original so results is just that JSON block that's really ugly i'm going to show you that next but first i want to show you the original image Sears where we started now i'm going to call I'm going to call that little show results function that I wrote to go and download those the the images from the JSON block I got back from the service and render them so for this image my ten most similar images or yeah ten most so again there's nothing in my model nothing in my system that says that this is a bathroom so let's see what I got I got bathrooms and they're pretty varied they're not a this is what was interesting to me lighting is very different they don't all look the same they look pretty different they're weird angles you know house pictures if you've looked at them a lot of the wide-angle lenses are used some don't some do different so that was interesting that was one example let's try another one yes let's try let's do this outside house 1450 let's do that one and show this so the original i'm going to show you three more and then i'm going to give you the plug for the conference and then be here for any questions so this is the original so it looks like an outside picture of a pretty nice house by the way i don't know if this is true in San Francisco but in the northwest you'll find this equal amount of photoshopping done on all real estate pictures so the sky is always blue like this amazing blue everything's really lush and green and now it is green but it's never that bright green that you can only get from Photoshop and so here you can tell and so again nothing about the image there's I you know I'm labeling it saying it's an outside house picture thing but this is what the model this these are the ten most similar images you can immediately tell how these are all doctrine I mean this guy really looks like that so let me do three more examples and then and then I'll switch back to my slides 2250 what will my notes here good 3d renderings of modern house exteriors so there we go it's a 3d rendering of a new construction I guess this must be the name of the style of house and then my ten most similar well this is another one from that same developer I'm sure interesting say it again this one I don't think of that photoshopped I think it's more that the model I think the similarity is in the way that these pillars and the shape of the entrance I think that's what's getting picked up is that this whites the the white pillars this one looks pretty lousy I'll admit I don't know where this one came from Oh could be yeah could be the white sky too that's a really good point since these are rendered images so that one I thought was interesting let's do another big house and then I've got one more where we'll do actually before this let's do let's do boats boats was fun I like this one so apparently and I didn't know this a a lot of people that have I guess properties that are on water will include pictures of like the water that they have access to if your house has its own dock or you your property has access to a doc you might include that in your your your mls listing well it turns out that so does everyone else who has that sort of view so this was kind of neat that I didn't expect to see this in MLS listings clearly I'm not looking at the right price point to find houses that have their own private dock but I thought that was cool let's do let's do this one and so here we go this is it I mean this is a rendering but it looks so close to a photoshopped actual picture too but you can see this modern house two doors saw tudor-style i think is what it's called with these pillars in front and this patio what did we get like I was a little disturbed by how it guy all that makes you a little sad that like all these expensive houses look the same but but there you go they all kind of do look incredibly similar and so I'll end here in terms of demos but again I just did this you can see most of this code I wrote was just messy Python to clean up my own service but the actual part of getting my images processed and finding similar images that was really really simple switch back to PowerPoint so i can give you guys a discount code for our conference so our conference is on monday and tuesday wow that is getting cut off that looks awful we will have a thousand plus data scientists not data scientists and we've got some great speakers if you haven't seen our speaker list you really should check it out you're going to get hands-on with data with our products and though there's a cattle competition that you'll get to hear a lot more about the website for the conference is conf datacom and it's in san francisco by the way I forgot to mention that it sits around the corner and the the marriott marquis this is the the discount code please don't tweet this this will get you fifty percent off the whole shebang I've got your camera out i'm so afraid you're going to tweet it it's totally okay okay i'm totally kidding relax uh-huh so monday and tuesdays our conference you're going to get to hear a lot of a lot more interesting stuff Alexei got me to talk a little bit about some of the neat stuff you're going to hear about there's a lot of other neat things that are coming as well and I should I should end with one more thing every couple weeks we're down here we don't have an office in San Francisco but we try and do office hours so every other Thursday we're in San Francisco one of our partners is galvanized and they've graciously given a space in their in their building which is beautiful if you haven't been there before so you'll see me you'll see a couple other folks from dato we come down every couple weeks and we're here on Thursdays the next the next time we'll be here is next week on Thursday go ahead that's right the whole thing oh you want me to go back or do this fifty percent off the whole thing so yeah it's I already think our conference is incredibly uh I think we're the we're pretty cheap too for 600 bucks to get two days a full two day conference especially in San Francisco but to get that at half off is it's pretty nice I have to warn you guys we will sell out if we didn't already so we when I left when I left end of day yesterday we had over 900 people and I think we can only fit a thousand in the in the space the fire marshal will be mad if we sell more than a thousand so that's it go create something hopefully you're going to do that with dado and that's all I've got questions how did I originally trained the image that model yeah I'll tell you how we trained it the best way I can show you that is by going to our blog p otra tether back one of our machine learning engineers is the one that you wrote this great blog post so we're going to go to it so bringing deep learning the grocery store that was a fun project that he did but before that and I'll zoom in here so you can see this a little bit better he kind of walks through the entire process along with sort of walking through how we trained the image net data set we also have you haven't been today TOCOM yet we have this learned section that has this gallery of ipython notebooks that kind of show end-to-end examples of things you can build using our product and one of those j j gu one of our co-founders worked with p otra as well so this is how we train the image net model so we did this on on a machine that has a GPU and used GraphLab create for it and you can see exactly how we did it it did take a while I forget exactly how long i think the model told us so if we go back to the model i think it said something like 29,000 seconds oh it was doing the demo I showed you earlier that where it's guessing where it's predicting labels for images that is the image net model deployed I should have made that more clear other questions to get the answer of how long it took there we go 28 no 288,000 seconds to train now we didn't have the most powerful machine to do this on but and we didn't use the latest Buddah the latest Nvidia card but that's you know that's why you don't want to have to train your own you want to make that a last resort go ahead yes you can yeah that's what this blog post walks you through how to train your own this notebook walks you through and this blog post walks you through how we trained the image net data set but what we really changes a convolution convolutional neural net using our deep learning toolkit or deep learning model you can certainly use AWS the GPU instances are great multi-gpu support you should come to our conference on Monday you'll get to hear more about it AWS now offers multi GPUs this idea of doing a distributed system with multiple GPUs it's I don't think there's a lot of results that point to it improving things the communication overhead GPUs are not very good at you have to do a lot of partitioning ahead of time in order to leverage the pipelining in the GPU so trying to have several machines do that partitioning ahead of time and coordinate usually doesn't help you but multiple GPUs yes go ahead we we have our own ours is based on cxx net which the story is a the primary author of this cxx net tianqi was our intern last summer and so this was part of his research his advisor is our CEO carlos gastrin and so he this was part of his research and so we took where he was and we built on top of it so we forged cxx net and added some stuff to it but this is a this is our implementation this is our kuda library or implementation of google so I'm I'm around I don't want to make you guys also here I'm around so please come by say hi tell me more about what you guys do and thanks so much for your attention