Charlie Maalouf, Image Similarity and Transfer Learning

Charlie Maalouf, Image Similarity and Transfer Learning

Recording: Charlie Maalouf, Image Similarity and Transfer Learning

funny for hosting us and thanks for Nitro and Alexi as well for giving us the space so today I want to talk a bit about image similarity deep learning coupled with some other machine learning applications to provide a simple very simple solution to a problem that some of our clients currently face so let me start so a little bit about myself my background is not in computer vision nor in machine learning I'm a time-series econometrician by training a few years ago I switched over to the machine learning world where I was building engines in the intersection of the weather and electricity sectors today I work at dotto I'm a senior data scientist and my job is to help our clients build successful intelligent machine learning applications that satisfied their needs so about us we are machine learning startup based off of Seattle and we love building intelligent applications we're about 50 engineers today and designers product managers so on and we are always hiring so that happy smiley purple face can be any of you so if anyone is interested please come talk to me after the talk but very quickly I bet you've all seen that workflow many many times if you've gone to strata Hadoop they probably spoon-fed you this so I'm not gonna go through it too deeply but the data science workload flow today is decomposed of many parts data ingests after data ingest you have to take all that raw unstructured data do something nice with a transform it to something useful you take that there features now you build the machine learning model you evaluate it you deploy it as a service so on and that that workflow is not linear at all it's very iterative back and forth so we add ATO have offerings for every step of that workflow to help your work a lot easier so we have our as friends or GraphLab create and the as frame which is an open source component of GraphLab create which is an out-of-court computing data structure that allows you to scale up the size of your hard drive on your computer and do a lot of feature engineering feature transformation stuff like that and then we have machine learning models from across the spectrum whether it's image analytics graph analytics text analytics to more standard machine learning models trees and so on after you've built the model that we have predictive services which is a way for data scientists to deploy their models as a service do a/b testing version their models and so on so we have a offering for every part of that component but moving on to the talk today so why do we care about image imaging machine learning I guess that's a very hot topic nowadays the easiest example that comes to mind is Google Image Search you put an image and you get images that are similar to the images that you create that may be easy for Google if they have a lot of semantics about the images labels stuff associated to the image but when you don't have labels and which is most of the image data in the world that's a very harder problem to do another very common use that is starting to show up is visual recommenders and that's an article two days ago on technology review basically saying shoes comm and Pinterest are both using images as a way to recommend so an example of this is you're searching for a dress or a shoe or whatever and you can visually say what you like but you don't know how to exactly describe it in word which is the usual way of doing or buying things so you could say I want more like this shoe or more like this dress and just drop a picture and then you get recommendations for that picture and then finally one last application which is my favorite application a client of ours or kampala G in the waste management industry and they're literally working with garbage so what they have is these cameras install them big dumpsters and they classify the state of the dumpster so that they optimize garbage truck routes so basically the truck routes our own trucks are only going to dumpsters when they know it's about time that they need to be serviced anyways today's talk is about image labeling in particular especially when you don't have any image data so to start with that all of these kind of applications that I've talked about so with a very higher-level questions like how do I compare similar images so when we talk about similar images we usually take an image and we somehow represent it as a numeric vector right and then that vector could be simple pixels so a pixel is a RGB float from 0 to 255 for example or could be a more complex transformation of those pixels right so you take those two images you pick a distance function and then you can measure some similarity across two images there's no science here so the problem is raw pixels tend to perform very badly I didn't try to compare similarity of those two images but it could be quite misleading so feature engineering in images is actually a hard thing to do it requires a lot of domain-specific knowledge and the main knowledge of domain-specific software so one example in in the Python world is scikit-learn is used for almost everything right people love scikit-learn I love scikit-learn but then when it comes to images you have to switch over to psychic image and that's a bit underdeveloped it's you know there's a lot of issues you go to OpenCV or something like that then it requires a lot more knowledge way less intuitive to work with and so on so that's really hard and this is where deep learning where the advances of computer vision have brought to the table some new advances that are very promising that lend themselves to that type of problem ok so before moving on I need to just say this is not a talk about deep learning I'm is gonna briefly cover a few important things that I'm gonna use and then go on to building the application that I'm talking about okay so what do we mean when we say learning the features let's start with the most simple classification problem right you have two classes represented by those two deterministic lines and then you have input to the data and you want a problem classify the probability the class given that input data right very simple problem if you use any form of a logistic or linear model so on you're gonna try to fit a line between those two curves write a line between those two curves is impossible you're not gonna be able to fit it so what you end up with is a discriminant something like this anything in the blue areas blue anything in the red areas red but obviously there's a lot of errors right so from this point what did you do you could sit down on this problem create smarter features interactions of features quadratic features and so on to try to solve this problem or you can add one more layer of complexity to the problem right so everyone's familiar with this diagram it's a neural network so what the neural network is really doing taking that input space doing some transformation to the input space so it represents it in a nonlinear manifold that then you can linearly discriminate against so what happens is you end up with something like this right so if you look at that image those curves now have been pushed to the sides right and this is what the hidden layer actually does so the hidden layer really does most of the work it pushes those two classes apart from each other such that any activation there you have at the end layer can perfectly discriminate between them right so now this begs the question in many cases I could basically chop off the end layer and use the features that are created in the middle layer because those are very useful and I could put whatever classifier or regression model I want on the end layer and still have something that is pretty useful with very easy fluency so deeper networks or deep learning is just many stacks of these hidden layers where each hidden layer is a complex transformation of there that it it fall it precedes right now convolutional networks which are commonly used in image problems are a bit more specific than this the layers themselves are a bit more engineered and have more specific properties but I'm not gonna go into this right now if you have questions I'll be happy to cover this again so deep learning is great right we can learn all these complex features rather than sitting in wasting time and it also have shown recently some really good impressive gains everyone knows this or not like saying anything really special especially the problem is it's really expensive to Train expensive to train in the sense that when Google open sources tensorflow they don't have a problem with that because you know there's a lot of deep learning software out there but google has the most rich data in the world and this is where Google like has its gains or advantage in any way right so they like it requires really a lot of data to train and very computationally expensive to train right also it's very hard to tune if anyone's actually tried to build a convolutional network here without looking at some research they probably know that it's actually hard to get something successful like Alex net or Google net or something like that ok so here's the solution transfer learning that's pretty nice you have lots of data yeah you you have some graduate students in a lab somewhere sitting living off of a thousand dollars a year you know that's what graduate students do and they they learn they spend a lot of time building a very good network that does very well at some problem and then people like me come and they take that architecture they have data that is unrelated to the data that was used in the original training but what they could do is chop that network off at some point and extract those features that were learned from the pre train model and lesses has shown some really good generalization properties in many different applications and I'll walk through this in a bit ok so just one more thing before I stop for a second and switch to the demo so let's say we found similar images to one image so I have an image a query and I've searched for its most similar images so I have in this case a toilet and I have a data set of images if I use the deep features that we were just talked about I'm gonna end up with first similar image seconds most similar image and so on right if I go ahead and do this for every image in my dataset then I will end up with something like a similarity graph right and there's a lot of complex things I can learn from this graph and then we'll talk about this in a second what this graph says a note is a picture and an edge between every two nodes it means that those two pictures are similar to each other now a clutter of edge of nodes indicate that a density of images that look quite similar right and I could use this to my advantage one way or another so before I continue I'm gonna switch off to show some demo and then we'll pick it up from there all right so I'm gonna start off by importing graphlab and Network X for plotting Matt lat lib you know the usual plotting libraries I'm gonna upload my data which is in an S frame as frame is our data structure that is very highly compressed and what the S frame currently has is an image ID an image and then some features but for now ignore ignore those features we'll get back to them in a second if I look at the images of my data structure this is what I'm gonna have so there are images of real estate okay and that's the problem one of our client has they have an inflow about 40,000 images a day that they need to cluster classify label somehow and they get them from very different sources where the labels are bound to be wrong so they had an idea of using the places data set which is a known data set for images problem is they can't license it in a commercial way so we were worth trying to find a solution so that they can label their images without that data set so anyways moving on what I'm gonna do now is use something the future extractor so what we do is in a soft engine GraphLab create we have a pre trained alex net model based off of the election of architecture and what you could use it as a transformer so you pass in any image you want and you could extract the features that are the feature the the output of the layer before the last layer prior to classification in that model so I'm not gonna run this now on the CPU I just like cached it from earlier but basically those features in my ass flame earlier are basically those deep features there's about 4096 of them in every image and they are very sparse so as you can see here only a few of them are activated the rest are pretty much zero so about 5 percent of every of features of every image are activated ok now to give more kind of intuitive insight to what those images are I'm gonna look at those two images which are quite different and then I'm gonna take their features and compute their absolute difference from each other so those two images are not similar at all so if I just take those features I would hope that the features 0 to 4096 are quite different and that appears to be the case if I move on to two images that are similar to bathroom images and I compute their differences then you would hope that the differences now are much smaller right so the features are obvious the representation this is much closer between similar images than none again nothing magical here it's intuition that we expected and it's working so I'm gonna go ahead and build a nearest neighbor model to build my similarity graph ok so we have many tool kits including nearest neighbors to get I'm gonna just create a nearest neighbors model passing images the the s frame and use a distance cosine function and the reason why I'm using a cosine function as opposed to Euclidean is because of the sparsity of the input space so I'm gonna create the nearest neighbors model in a bi query image ID 71 which is this image I could see that the first four images are quite similar to it in the sense that they're all bathrooms and they all have a similar type of design to the bathroom so we're starting to get somewhere in the sense there's some similarity across those images that is tangible okay so I'm going to do that and just extract the entire graph from that nearest neighbor model so basically it's just a for loop that's going through every single image in your data set computing it's more similar neighbors and if I do that and I plot it very quickly I'm not gonna plot this right now because it's good to take them some time I'm gonna end up with this now what this is is I'm looking at a representation of my my entire image data set right and I can see some natural clusters arising in my data set so I have a cluster here I have a cluster there and so on I still have no idea what those clusters represent and that's the next step but at least there's something right now just intuitively what's what the visualization library is doing it's a form of clustering it's a nut it's a form of not very scalable form of clustering it's MDS scaling and then basically projected on to coordinates and you get that plot that is not very scalable so instead of using that the coordinates here what I'm just going to do is do k-means on the data set and use that as clusters I'm still gonna use my similar similarity graph for something else in a second okay so I'm just gonna build the k-means model and then if I look at my k-means you can see that the cluster IDs the average the the centroids of each cluster and so on nothing really fancy here so going back to the terrace talk so now I have I have that similarity graph right and I have these clusters that I really don't know what they are what they represent right so is there any way I could take that cluster and find the most representative images of that cluster in order to describe what that cluster represents right so I take a cue from page rank so everyone knows what page rank is right it's an algorithm used by Google search rank websites and the page rank is a measure of how important that website is intuitively for those that are not aware with PageRank it's if you were to randomly surf the web it's the relative time you allocate randomly for on each website overall so I'm gonna do the same thing on my similarity graph so I have a similarity graph in which the edges represents how similar pictures are from each other so intuitively the the image of each cluster that has the highest PageRank is the most representative image of that cluster so if I can take each cluster and look at the top 10 PageRank images of that cluster if they're all of the same type that I could with some confidence basically propagate propagate that label on to the entire cluster okay so let's go ahead and do that so I'm just gonna write a function here what it does is it basically takes the similarity graph that I have it's gonna take the cluster IDs it's gonna create a sub graph based on all IDs that belong to the same cluster now in the sub graph I'm gonna do page rank and display the top 10 most representative images right so if I look at cluster I ID 0 and I run PageRank it's gonna take time so the SRAM is a is it has lazy evaluation so basically I've been running a lot of operations that haven't materialized so ok anyways right I ran PageRank on the first cluster and I ended up with all these empty rooms right they're all very similar they're all very empty kind of ugly nothing special there if I go to the second cluster and I do the same these are images of plants you know a lot of times you go on Craigslist you see a floor plan you go on you're buying a house there's a floor plan in that data set right so it seems to be working pretty well I'm gonna go on to the third cluster and it is front yards of your typical American house right so it seems to be pretty consistent now obviously this is machine learning things don't work always very nice so if we dig deeper in some of these you will find errors so this these are a different style of bathroom and if I look over these pictures I have a picture here that looks like a very ugly kitchen otherwise a bathroom you know so obviously that's a miss classification but that's fine you know that's machine learning that's passive nature of things and I'm gonna keep on going and going if I keep on going I'm not gonna run them all but you have kitchens over here you have nature and scenery you have living rooms and dining rooms you have more different style of toilets and then you have like this very like sharp front view of houses so now that I've represented all my clusters which I have an idea what the cluster is I'm gonna just forcefully assign the label of the 10 most representative images onto the entire cluster now that's not the smartest thing to do I know we could do label propagation which probably will work better but I just didn't have time to do it as simple as that so what I'm gonna do is just do that assign labels and we plot my map and this is what my map looks like so there's some cases where rooms are very separated so like the plans are very separated from the bathrooms that are very separated from the kitchens except for some ugly kitchens in here but as you can tell so for example in here there's a bit less separation but that's fine because you know there's reasons so I met this one so this is like the two sides of empty rooms you have some errors that we could work on fixing okay but this is where the demo ends in here so what's the next step as I said this is a very simple model that I built and I've assigned these labels in a very naive way you could do other things you could do label propagation on the subgraph or you could build a classification model on the the labels that you've assigned with some confidence and you could measure confidence from page rank or something like that and then repeat diagonals find the errors try to fix the errors and finally once you have something good you could deploy that model as a service in which people are querying with different images to kind of get labels from that image so that's the end of my talk today yeah yeah that's a very good question okay so I did this very heuristic Lee when I did MDS scaling over here idea or where was the image sorry yeah yeah so no no it's from here I just the scaling suggested that there's about 10 representative clusters not the most scientific way I agree but just for fun it seems to work fine obviously there's two types of bathrooms that if I were to decrease the number of clusters they would merge and if you look at the bottom graph in here at the last graph you can see that there's a lot of similar rooms that you could merge so let me go to the bottom so those two over here those are two styles of empty rooms so if I were to decrease the number of clusters they would have been merged together but that's fine you know so non-scientific in any way more visual than scientific this is K this is using the X the X Y coordinates generated from the MDS scaling and just do color coding on it so it's the same embedding as the one before I I mean yeah if I displayed an embedding based off of k-means then you would get perfectly separable obviously so it's not the nicest waited yeah I sorry can you repeat the gun I mean this is like wear it away so I played around enough for the data set that I know the types of bathrooms like a classified as kitchens literally it's like so a lot of bathrooms have this like wooden color cabinets with white tile and whatever and a lot of ugly kitchens have the same style yeah yeah so no no clustering mixed it up right so I could do more sophisticated clustering so this similar this is a graph of 2000 images if I did geodesic base clustering I would almost certainly get a better result than k-means the problem is geodesic clustering does not scale more than thirty forty thousand images and that's a like for that specific client they had like forty thousand images a day so that is not a good solution anyways so if you want to learn more about dado please visit our website you could download us and try us for 30 days for free if it's personal use you have a longer trial as well so everything I did today except for visualizing that class that first plot the network plot is in GraphLab create so you have graph analytics image analytics clustering lower-level toolkits and whatnot we scale as much as your harddrive fits data in its we're out of core and if you want to learn more we have a lot of events a blog and a lot of notebooks to start and learn from and finally again we are hiring this is like my bag to hire please apply yeah okay so there's two things one the reason why I didn't use the Aleks net model right away to label or classify these images is Alix net was trained on a very general data set so it's classifications tend to be very bad when you look at a domain specific set of images okay so once you have a domain set of set of images you have to do something different either train if you have a corpus then the best thing to do is train your own convolutional net work on that corpus right the second part is Alix net itself or most if you train a convolutional network smartly basically you need to make sure that you take into account things that are related variational and translational changes of the images so in the training set what you end up doing is flipping a lot of the images you rotate some images you defer the gradient of the colors on the images but that's part of all the data preparation part that goes into training the model since we're using a pre trained model they've taken into account all of that and we just extracted the features that are invariant scaling and translation and movements of your image okay oh yeah so so for the Alex net architecture we can score on a GPU about 200 images a second on a CPU is three images a second so that's why I didn't run in here our deep learning library is built on top of CX net for those of you that are knowledgeable of the convolution and network data which is one of the faster libraries out there and soon we're moving on to max net which is at this distributed more scalable version of it yeah I mean so here's so the question is benefits versus time spend the fine tuning things right so the Alec I mean if you have good label data set I'm pretty sure you can find you in the network itself right but you're talking about over like two hundred three hundred thousand like very well labeled images so yeah I mean if you have five hundred thousand like really well labeled images don't even fine-tune because I mean there's a lot of new papers that show of fine tuning is not all of that and you could start with five hundred images from scratch and train your own network very well and so there's a the places data set is licensed by MIT and it's a 1.3 million real estate images and it does way better than Alex net with fine tuning right so but and they build the architecture from ground so I would say if you have that wealth of data I train your own network for sure yeah we so are deep learning library you could build any architecture you want but in the sense of like pre train models that you can extract features from currently we only have the Alex net architecture you could build a Google net architecture but you need data you need table it takes time and effort as opposed to this where I extracted the features from the Alex net in about 10 minutes so there's so yeah so the cafe model zoo has about last I checked like about 15 pre train models including Google Nets the problem with cafe is a the models are not very interval to other platforms right and in terms of popularity I would say now with tensorflow coming up and with MX net like improving a lot I would say Cafe is not top 3 most use deep learning frameworks out there so there's cost of using cafe I think my my best part of Cafe is the fact that model zoo exists which other platforms do not support yeah yeah TSN does not scale as simple as that I mean if you want to use the s and E you need to start by doing PCA if you have 4,000 I meant like this these images let's say they have on average 80 by 80 pixels or whatever you need to do PCA reduce dimension or whatever reduces dimension down to less than 100 and then on top of that you can do t-sne with more than 20 30,000 images I mean in this live demo I could have done TS and he actually this is what I wanted to do but it's not a scalable solution in any way yeah awesome thanks for the Tibet so I haven't read the nucleus yeah I haven't seen the new picture I played around with the old the the first version of the SN e and your limit you're very constrained yeah yeah any other questions well thanks thanks Alex