Text By the Bay 2015: Shourabh Rawat, Organizing Real Estate Photo Collections with Deep Learning
my name is Sean rabbit I'm from the 2d array design stream I'm a senior data scientists there and this is joint work then with Marcelo data scientists up in of Joseph and Shira so how many of you here are familiar with Trulia a good number that's great and how many of you are actually searching for homes these days okay okay so so the the crest crux of this talk it would be about like us would be able to use computer vision and specifically deep learning and also understanding the text that we have for properties that you see on the website and sort of somehow combine both of these two information and create and the face that can provide users with collections of photos so let's say cellar photos or swimming pools photos across all the truly off active fossil collection so so Trulia so I'll begin with some introduction about Trulia so Trulia is a real estate search site you go there you you can search for homes that are for sale and it'll only require you know not only provides you with in with a way of interacting with the properties that are active or for sale currently but also interface between you and your agents and and also actually captures a lot of data around the real estate so there are some really great visualizations that we have by extracting data from various sources like wildlife fire thread or earthquake threats and like crimes rates so so Trulia has like we recently been acquired by Salo and what it does is provides us it makes us one of the largest branch across across us with the the largest inventory of real estate content and it also composes pretty much the biggest chunk of unique visitors that two real estate websites so okay so the major problem here is that we have a lot of unstructured data I like all of all of the web and then a lot of images that needs organization so that people can come and find what they are interested in so if somebody is interested in they say oh I'm interested in a bathroom which has a fireplace and pink Falls or something like that so those are interesting queries and we might want to support that so looking at what data we have at Trulia in terms of images it's not large it's not but it's pretty modest we get around 20,000 to 25,000 new properties added to our collection every day and then there's some also like 72 thousand properties as a whole over a month and this amounts like 9 million photos at any given time we have around 2 million active for sale properties and by looking at that and around 10 photos per per property we have like 20 million active photos so there's a good amount of photos that we have at any given time that is we can display to users and sort of create those photo collections that people can look at so one thing to note here is that the photos as soon as the property goes out of market we basically pull down that image so that is no more part of the collection so these photo collections would stay as long as the properties actor so they are always sort of new data coming in so there is in a stream so so real estate photos of basically if you have homes and like every time so the way the the real estate industry works is basically the somebody whose wants to sell his house some mr. listings listings why MLS is comes to us and then we put it on the website and then the two important things that you would basically see with a with a particular property would be the images and a textual description about that property where it is located sort of what are the key attributes in the house that whether it has a fireplace or not whether it has granite countertops what type of pool it has and so on and so as you can see they're like two sets two types of unstructured data here one is the image is that you don't know anything about these images don't come with any captions they don't come with any manual tags they do even their names is like a hash code so so basically there is no information about the image at all most of the conventional search engines or image search engines rely heavily on the text attributes that come or text metadata that comes with these images right so if you go to Google Image Search the major result that you see at the top are not from a computer vision system but they are actually highly boosted by by the textual attributes that are come with the image and then we have the other and structure set of data which is the text itself which primary comprises of descriptions that users have are the property owners have provided with that property so what we want to do is we want our user Trulia customers to be able to sort of take this last collection of two million properties and a twenty million photos and then sort of form a browsable sort of photo collection for them so example like crystal chandeliers and dining room photos right here some of the examples that some of the collections that we have made using our our data like like cellar set of photos like something like center island kitchen so not only are we looking at kitchens but we are looking at specifically kitchens that are actually a central Iowa you can also see the captions that is exact it sort of to suggest like what part of the description is referring to the center island in many cases like you will find that there is in some cases and some of these photos it is pretty clear that it's not very clear like there is a center island or not some things like crystal chandelier so tsukushi chandeliers in dining room photos we can have similarly like architecture styles so we want to look at homes that have data Mediterranean style you want to look at photos of pool which are like infinity boobs and so on so how do you go about creating such a collection so there are multiple ways one way is like you can use user generated content right most of the current websites if you look at like Zillow takes interest most of these websites actually run by users uploading photos and they're providing enough metadata that actually allows them to sort of organize this content and this is pretty powerful and you actually can get really good quality photos you can get really good organization of content and make browser experience really good but there is downside of this as well like because like if you see most since these are uploaded by users these are tagged by users the process itself is pretty slow right and so there's a slow growth of content or inventory that'll that you would gradually stack up and the other problem is it's sort of expensive so let's say you are doing crowdsourcing sort of you say oh okay not only uses tagged my data but maybe I can crowdsource it to Amazon Mechanical Turk or maybe I'd do it CrowdFlower whatever and it's it's expensive there because you probably have to pay more to have fine-grain tagging of the images and they can tend get stale so even though I have like oh maybe I have like hundred 200 new photos being uploaded by users every day or like more than that in some cases the the content would actually become stale after a while because there's not much activity happening there and so you'll be showing the same swimming-pool photos all over again so there's some of the limitations with like so why we can't do it because we can't do it because as I said like most of our data is just coming from real listings MLS's there's no content in those and what we have to do is we cannot rely on a user-generated content here we have to either to craft sourcing or find some other way of organizing that collection through computer vision or textual mining so the other ways okay so we know deep learning is really getting a move on it's sort of fueling our industry in terms of breaking many benchmarks and why can't we use that and yes we can actually it so if you the kind of current like convolution networks considerably or the conversation networks you can do pretty well in terms of recognizing scenes recognizing objects but what about attribute recognition like so like let's say we have a scene and then obviously identifying a kitchen is easy but then identifying finer grained information within that kitchen for example granite countertops what if is it a large bedroom where is a small bedroom what is the texture of the comfort of itself given that the image has been taken from a certain perspective where the texture is not at all visible texture is actually a very big problem specifically because texture gets affect like it's really a fine attribute of any surface and then it really gets affected a lot by the perspective at which you are viewing it the lightness the brightness and so a hardwood floor can look a concrete floor given to a machine learning algorithm given certain considerations so and then there is a problem for clusion absence or recognizing branch so for example of what if I want to say Oh show me all bathrooms that have Kohler fixtures right ok so I need to first locate that there is a fixture there and then how do I identify the brand right that's a tricky tricky computer vision problem occlusion so the center island is somehow occluded from the way the user took the picture of the kitchen right because we don't control how the picture is taken and then there is a absence so it could be I either occluded or it could be absent at all right so there is no in that picture of the kitchen there is no center island so so that's why we cannot like completely do computer vision here and then Robbie obviously like we do so like for a new domain like let's say real estate or let's say you need a win like you do reading a web where you have these HTML pages of images and text like create creating all these strange samples for each of your attributes or each of your objects and classes would require a lot of training data so domain adaptation would become really hard if you are doing it completely by a comprar vision so as I said like a lot of challenges you have illumination you have rotation you have scaled variance viewpoint variations occlusions and scintillator with especially with scenes there's a lot of clutter happening in the in the scenes and so how do you differentiate between a so like a particular room which has a bed but then it also has a sofa and then it also has a chair and there's a lot of things that each signal to it being a different scene type and then there's a lot of interest variations as well so okay so the base data that we have are to begin with as images and then you have the textual content and what we want to do here is that we know how to do well on identifying scenes at the higher level in the textual level we know how to extract interesting key phrases or content from the text and then we want to do is transfer knowledge from what we understand from the property descriptions over to what we understand of the scene so for example if I know that granite countertops is associated with kitchens and then I'm able to find in that big collection of photos per property that there is a kitchen and I can assign those attributes over to that that photo and by sort of tuning the way I score each of those photos I can actually create a retrieval system that can sort of given a query given a key phrase return me an ordered list of images that match that query so that's it so the gist of the simple idea you take you understand the associations between the scene and your key phrases and then transfer that information from the property descriptions to the visual modality that you have detected so this allows you to actually not train like thousands of models or 2,000 models of all the possible key phrases that might are interesting in your domain but just train all the one the few models that you think would perform really well and then use the semantic model based on the text to transfer that information over to the visual model so for example again is an example of you have a kitchen here but then it has so many attributes that are either occluded or the perception is difficult to have God lecture here bay windows a violin kitchen hardwood flooring white kitchen cabinets reset lighting all of them mentioned in the in the description but cleaning an individual classifier for each one of them would be expensive so as I said the key idea build a visual model to recognize major scene environments we can use deconvolution networks we can use also traditional driver first models we can use then use the text that we have to build a semantic model of how things are organized in a house based on how people describe it which we'll use word embeddings for it and then we basically do an knowledge transfer so we transfer information from the actual model over to the visual model and this sort of falls into a category of zero short learning where we don't my visual model doesn't know anything about granite countertops or hardwood flooring but I infer that information from my textual model so I hope there's 30 minutes left so next 30 minutes I will be talking about high-level about deep learning I think the crowd has already knows much about it then I'll talk about how we build our visual model the textual model and then how to do the transfer show and then we'll follow it up with the Q&A so so deep learning has been around for a while and really sort of took off in the last 4 3 to 4 years and it's finally here like you see a lot of applications readable applications where it's being applied and if to quote Wikipedia the deep learning is a set of algorithms in machine learning that tries to model high level abstractions and what it how it does that is via through multiple layers of nonlinear transformations of data so so you take some input into some nonlinear transformations and then at the end you have represented whatever classification or regression whatever you want so often like so I was looking at a particular talk by Andrew ng like a few days back that was in the CUDA conference I presume and he was talking about like there is a lot of hype about deep learning and then deep learning is because it mimics the brain and then it was he quoted a Michael Jordan and which said that it's just a cartoon of how the brain works we don't know nothing about how the brain works so the hype is there but it's not exactly how we brain works it's it's still far away okay so what do we mean by high-level intermediate representations so for example like for example in face recognition you would say okay take the image take it take the raw pixels take the the raw pixel values and sort of learn these higher levels it each level for example layer one if you see is pretty known to computer vision scientists as like okay Gobbo filters or like edges and corners and the second layer is basically a combination of these so if you take these multiple layers and sort of combine them you can think you can create all those parts to face paths and then if you combine those and you're going to be construct a face so these are the high level abstraction that deep learning is able to similarly for speech recognition and I think speech recognition is is one of the areas which actually sort of sustained deep learning or like were the first places where deep learning really performed well and they're the first the idea is to take the raw speech signal and then to sort of identify latent representations that can do very good at phoneme classification so what is the neural network to do code bench you're like any terms it is like like running several logistic regressions at once right so if you see basically in this particular Network like each neuron basically each node that we see there is connected to all the nodes in the previous lesson fully connected architecture and so what it means is like every node is basically at WT X where X is the input to it and a transformation by that so it's like a linear classifier right so each neuron is a like a linear classifier and then apply the sigmoid on top of it you have a logistic regression so in a way you can say it looks like multiple logistic regression is happening at once the key advantage of deep learning is like we have end-to-end learning so you have this continuous differentiable function that you can optimize for instead of doing part based learning so depending is now state-of-the-art across various Pancham benchmarks be it NLP images and speech recognition a lot of recent work has gained popularity around captioning which is sort of work that sort of relates to this talk in terms of taking images and then sort of generating captions for them automatically this is another slide from some time back where like the effect of deep learning or speech recognition is shown to make a significant impact compared to the last decade of research so now so coming to how to build the visual model so we'll talk so how many of you here have worked with computer vision in your earlier life - okay and how you work with bag of words model so yeah so it seems like years are like like history but back of earth model has peener has been the area of research for the last ten years of computer vision before convolution networks came in and now you hardly see them in any conference so this is how the image recognition pipeline used to work before you had these images and you used to perform feature extraction in you would encode them and then you do the prediction right so you have these standard features that you would extract you would extracts if Daisy Hogg and then you have encoding so you have the bag of words just like we call it a visual were just like backwards you have Fisher Fisher vectors DSP and they have spatial pyramids and all of those and then at the end once you have these representations then you use a linear classifier or a kernel SVM to do the classification and now it looks much different it's like instead of having all these different sub parts we have one end-to-end learning with the convolution Network so this is a like this is a snapshot from the Christmas key at all the first real deep convolution network that proved the benchmarks and emission at 22 well I think okay so I would go with sort of a brief description of a bag of words model so that we understand what they do and then why is deep learning doing much better on top of that so each image the way computer sees it as is like it's basically pixels right and pixels have a value from 0 to 256 across three channels RGB and so the way the initial scientist sort of sorted it out today okay so what we can do is we take the image and then we've taken into these patches so each image is composed of several facets right and then we if you were able to represent each patch then you'll probably be able to represent the whole image as a distribution over these patches and so this is this how it used to be just like type of words you basically take all these patches represent these patches and then create a codebook and then define it represent each image as a distribution over those patches to go more deep it would be like this so you take an exterior of the house photo you sort of extract these let's say four by four patches and then represent them by one of the feature features that we describe mostly these features are basically gradients to capture how edges look or edges or corners look and as if and then you take these a bunch of these and then learn a codebook so unlike words where each word in itself is its own cluster mostly its own cluster here the each each word is basically a vector so you probably have to do quantization or clustering some like Jamie's or GMM and then once you have that you do an encoding any of the Fischer encoding or gaming space encoding and then you have a representation and then you do some sort of nonlinear transform usually hiding the colors and then you have applied linear classifier to do one versus all classification so there's certain drawback as we can see some the spatial intolerant formations are not captured effectively we have to sort of enforce them by encoding representations so you'll probably do spatial bag of words or you will probably do train a different color based feature extractor and a gradient based feature extract and sort of fuse them either in early or late fusion and so there are lot of handcrafted features which have taken like decades of research and then this multi-part learning which is I think the major bottleneck each of these components are optimized individually and not as as an end-to-end system come convolution networks they try to solve this problem within an end-to-end learning a single differential differentiable function across and it has like three important parts one is the convolution convolution you can think of convolution as sort of similar going back to how computation was done like you take a bigger patch and then sort of sort of represented right so convolution does that it takes it takes a patch and sort of represents it and then it it conveys what basically applies that sort of patch filtering or across the whole image multiple times so you have these different kind of maps that you are creating there it has non-linearity so the non-linearity is important is because if you are stacking multiple layers you probably want to do non-linearity at each layer so that you actually end up learning a nonlinear function rather than a linear function TN so so usually use rectified linear units or sigmoid and then you have cooling cooling basically tries to sort of say okay now that I've convolved ever allegiance together I want to have some sort of invariance in my in my learn representations cooling sort of does that through like by aggregating the neighborhood activations so here's like a simple explanation of how you take a 14 by 14 pixel space and then sort of convolved wet down by applying with the stride of 1 to 10 cross 10 do pooling where you are basically taking two cross two nearby patches and then combine them into one usually use max pooling and then doing the process over and over again so so this is this is the alex alex net and this sort of takes that step further you have now seven layers several layers of convolution and max pooling and at the end there is the typical fully connected a neural network and a sigma pass in a sigma soft max classifier and so the training is pretty it's a pretty standard algorithm that we see at over and over again in all the deep learning models that's the real power of deep learning is like you have the same learning algorithm that can apply to any problem any domain and so sample data do a forward pass to get predictions back propagate the errors update the weights and repeat until you have until you are satisfied with the results just to give a comparisons across the two models like this is the alux net it has it was trained on 1.2 million images it has around 16 million parameters basically the connections that are happening across the network and it classifies a thousand classes and it's seven-day st a bag of words model that we train has around 0.4 million parameters so this definitely like in terms of the number of parameter itself it's like much more representative of the domain a bit of history the conventional networks have been around for a while fukushima pension and then leak one the famous sort of a menace data which has been one of the benchmarks for any computer vision algorithm there was first time leak on applied a convolution network and showed groundbreaking research and this was followed up much many years later by the work of christmas key and Hinton with LX net that broke the image and competition so it's worthwhile to sort of mention image net because it i think this is the the data collection that really made the difference in terms of in terms of really boosting deep learning in computer vision so let's say like there's a good amount of images that are required to bring such big networks of around 16 million parameters and again i have sort of a reference of how things have changed 2010 you used a standard backwards pipeline and come to in detail everything has changed significant improvements across domain so the bgg network here is actually 16 to 19 layers deep compared to the 7 layers deep two years to two years back google and is a much more complex network but it has less number of parameter and actually performs better slightly better than the vgg and the emission in 2014 competition so this is just a comparison like we were at like 26 using official vectors and medical solution 2012 SiC Krzyzewski at all we had 16 point for soon after the Zira and Fergus who now had clarified get around 11% and last year we had the vgg and the Google net perform six person I think this is going down even further to like four to five percent so what are the different layers learning so basically what they're given a property of the house all these players are sort of learning higher and higher level features edges sort of combination of those edges and then sort of the object representation itself and then followed by a trainable classifiers again some work from the winning submission of Zilla at all during 2013 where they actually visualized all these different layers in the network which is pretty strong seeing what each neuron is learning so the way just to clarify how this neurons aren't basically learned it's basically saying that this particular neuron is most activated given this average photo so what is causing the activation in this particular neuron okay so for this experiment we have around 40 scene object classes which we think are sort of occur significantly in our data to be not like clearly identified by a complication system we use to set of models we used by buffers model the the ones especially Fisher vectors and then we also use a pre ten convolution network from imagenet and then extract layer six and a seven from it and then learn an l2 SVM by a mini batch STD implementation so okay that was about the visual model I'll quickly come into the semantic model so so one of the things that we have to see in terms of semantic model is first we need to know what are the things we need to attack right what are the phrases that are interesting for us to be able to identify automatically from a collection that we can then assign to the images right the data is noisy so it's like user-generated sort of typed data and so it kind of spelling errors abbreviations in synonyms so you can have appliances as a ppl s but also as appliances and so on you can have B ICC or built in china cabinets and so you need to be understand how these things are related so that you can do a much better job at assigning these tokens and then what about multiple phrases collocations so we first I'll sort of say what makes a real estate what makes a phrase a real estate phrase and then we use a very simple model that was proposed by her strength of my Yuko and around 2003 which basically takes two measurements for defining stuff a domain freshness like this the domain nests of a phrase to say or in this case a real estate so when is the phrase nests was like how commonly in this particular domain does this phrase or this stream of a sequence of tokens go upper right and then second is informativeness which says that given this domain how does this particular phrase is unique to this domain compared to some background domain so to do so we use a Wikipedia so we have the hippy language tax which sort of represents the general English and then we have the cooler description which sort of explains what are the interesting phrases in our domain and then we basically build an Engram language model to compare the two what we use here is a KL divergence scale evidence basically sort of tells how two distributions are different from each other and for phrase necessary we simply do the language model for ground and Engram so basically if you have to prompt you're looking to a two-word phrases then you would actually look at you'll train a bigram model on your foreground corpus that is truly a corpus and then and a foreground corpus and and a unigram model in the foreground office so if if it is highly likely that you have a higher probability of this sequence of words in an Engram then in a unigram then probably this is a good phrase right it's it's quite a lot in for animated Ness is computed the other way around where you basically compare the the foreground model with the background model so you say okay how common is this particular phrase in my trulia corpus compared to how come how how I get it in terms of the background corpus so this gives a good sort of gives to phases of the coin like the phrase Ness which sort of tells whether it is the phrase or not and informated noise which tells whether it's interesting to my domain or not and then we sort of combine them it could be a beta combination or not so these are some of the words that we learn automatically from our data so when running this algorithm on all the property descriptions would find you interesting elements of what's interesting in your domain like kitchen bathroom which is me the same types as tuna grams and then you have these combinations of backgrounds like hardwood floors and French doors and crown molding vaulted ceilings ceramic tiles and and so on and then you have these trigrams like wood-burning fireplaces so which you would find less commonly and Wikipedia texts the second concept that you want to sort of leverage here is like the suspicion semantics so I think the first talk already talked about the meaning of distribution semantics that you shall know about by the company peeps and it's actually pretty important to understanding our content so the way people describe is usually is like when this is described in a kitchen then probably they will continue to describe the kitchen so they probably say kitchen has granite counters an overabundance of cabinet space big husband stainless steel appliances right here we have good an amount of information that is in certain association between granite countertops and kitchen because they occur seem to be occurring in the same context and that's a very good signal right in many domains that my pretty useful so we want to exploit that so one of the common ways of doing that is have a context window and then sort of compute these co-occurrences counts and for example like kitchen would go over more with island then with fireplace and bathroom would cope and more with showers and once you have this matrix then one of the earlier ways of doing things was today that take the co-occurrence matrix and then do some projection using SVD or later submitting analysis and then get a word vector which performs the way same way as you would expect a word back or skip the model to work here we use the script Kamal though so I think there are a lot of talks that would sort of talk more about the script car model here I will sort of give a brief intuition about scripta model in terms of what it's trying to optimize assuming that you have you want to create a training data which basically says a pair of two words one is a word you are interested in and then the other word which is either in part of the context so for example in this scenario it would be granite countertops and kitchen it's a pair a training sample and some random and your second sample would be your negative sample would be any random word and put with kitchen so it will be like kitchen and comma let's say lush lush lush yard and so what it does is basically forms these two data sets D and E - and these pairs of W and C and then it's basically trying to maximize the function where we want vectors we see n VW v C is a vector for what the context word C and W is the vector for the word itself we want words that occur in the same context to be closer to each other and words that occur randomly to be far away from each other by enforcing this we are sort of capturing the contacts contextual information and it also it seems that once we learn this model it'll also capture certain semantic interpretations so in our training model we basically use negative sampling instead of the hierarchical softmax we remove dry words we did some sample use sampling frequent words and use a window of size 10 so what are they capturing one of the things that they are capturing but we wanted them to capture was a tribute to seen relationships so we should know that granite counters are more common in kitchens than in living rooms near synonym so we come to know that waterfront and lakefront are the same thing spelling variations like appliances and so on there's a lot of spelling mistakes and our data brands like so how would we understand brands and how do you understand like relationships like cherry cabinetry and Silestone countertops and actually we see in our data that the business between cherry cabin cherry and cabinetry the same aside stone and countertops okay so looking at like so what I've done here is basically try to give an intuition of how things are organized so you have this so I've picked like this is like a collection of like 110 words from a collection that is sort of signified what we're trying to learn here the ones that are marked with green are the major scene types that we want to which is our visual model and then everything is basically assigned to that Vishram also if he's detect let's say spiral stairways and we know that it's closest to hallways so if I get a key phrase pile stairways then probably I should use a hallway image to assign that to so one of the interesting things that it's capturing is like the differentiation between an indoor scene an outdoor scene so if you see everything on top is like the one circle is like an outdoor scene like swimming pools and gazebos and exterior and architecture types and then everything on the right is sort of indoor representations it sort of falls into the I mean it naturally it makes sense because this is how we people describe right they'll be described indoor scenes and then outdoor scenes in different contexts just to give an idea a visual idea like so you have water fan side because swimming pools gazebos and decks or outdoor scenes for something like interestingly we have a Jacuzzi which actually lies pretty close to both the bathroom as well the swimming pool so you'll find as a koozie both in terms of like in both environments right another instance was chandelier where your channel is actually found in hallways close to hallways dining rooms and living rooms so that means when you're looking for chandeliers then you should look at these three photos so this is an architecture cyber like right at left corner you have these architectures and then you see all the Victorians and the Colonials and the story books all jumbled together so let's say I want to create this collection I can do how easy I think I'm running out of time here so real quick again like if you want to sort of understand the branch of posits you can actually see there's a cluster like there there is cherry cabinetry granite counter tops and one of the interesting thing was balusters palaces are basically all those you find in stairways and then we find that the losses are pretty close to two hallway or two surveys so one can easily when somebody queries for balusters we can show them stay away so to summarize we want to do is 0 short learning so learning without examples we use a visual model to detect the major scenes and then we use assigned key phrases in the proper descriptions to images using that semantic knowledge that we have a few other examples of kitchen with granite countertops cue chandeliers yep that's it Q&A so people describe the whole property they don't describe individual image and so you have a bunch of collection of images and then a text then you sort of sort of cherry-pick and then assign it assign it so it's like yes it's done using the semantic model that we learned so that's why so there are two things one thing that I didn't explain here was when is a semantic model which sort of gives us a global interpretation of how things are related but then there is also the NLP model which sort of say takes into account the localization so if you have granite countertops which is actually referred to as bathrooms with granite countertops then you won't assign granite countertop the global model so you actually have a weighting mechanism so based on so weighting mechanism takes into a both the local context as well as the global context you use a couple of those so so most of our framework is in Python one of the things that we use is cafe a lot there are a lot of other ones that are out there Siana torch and that pretty good 2d planning for j i think is good too so I think yeah there are a lot of open source tool that one can use - we also use Jensen or for the word back implementation yeah I think yeah I think the algorithm is the advantage of these algorithms is that customisations is sort of minimized like for example like the word back model if you can just take any domain it and push it to it and then it sort of captures the good thing is that it sort of fits into the model that you were trying to capture in this particular this might not apply to every domain but for us this is what we wanted to capture we wanted to capture how things co-occur in the data and were to work model sort of does that effectively so that clearly fits a bit we had three yeah so we had 300 dimensions okay so I think we are almost done yeah so we had 300 I can talk more later see really cross the line