data.bythebay.io: Maarten Bosma, Image Retrieval using Short Texts
Recording: data.bythebay.io: Maarten Bosma, Image Retrieval using Short Texts
my name is Martin Bosma I'm a data scientist / machine learning engineer at whisper and I'm going to talk about retrieving images from short text that's maybe that sounds a little bit academic but it's actually very important problem for us so I'm going to going to first talk about what we do exactly and then talk a little bit about the problem and then talk about how we approach how we involve our approach over time to tackle this problem so whisper is a social network with over 30 million monthly active users every post on our platform is text on top of an image and what is unique about our platform is that posts are not associated with your identity there's no profile you just post something and then everyone can read it and the hope is that people share things that way that it wouldn't otherwise share on traditional social networks and then once you post something other people can interact with it using by what we call a heart which is basically like and then by chatting and by replying to it with the post it's in the same format so I've got some example C of posts that were popular yesterday i will give you a minute to read it so people can create this post by first entering the text and then we will actually suggest an image to them and if they don't like that suggestion they can see more suggestions and if they don't like those either then they can search for other images by entering a search term or they can upload their own images so image suggestions are a pretty crucial part of our creation process for posts and the first version of whisper doesn't didn't have this so they once we introduced this we drastically reduce the friction in our creation process and partly because of this more than fifty percent of our users actually create content which is pretty unique for social network or any site that has user-generated content so and we're trying to get images that are relevant to the post that are high quality and that's a support the emotional state of the post that's actually the most important function we're not so we're not trying to literally show what you're saying what we're trying to give it another dimension by adding an image so when we first implemented this we already had image searches so we basically expand it on top of that by generating a search term based on the text and then we will do in a search for that term and I'm going to talk about exactly how we generated those search terms in a minute but then of course we had caching so to make the process a little faster but we'd always have to go to like third party API to get the images and the reason that we chose this approach is well what else would we do right we could eat a directly search for the text but that doesn't really give you good results especially if it's a longer sentence or two sentences if you even in an image search engine like being or Google if you put in a long sentences you lose you don't get any results at all or if you do they're usually not very good we actually want to highlight one specific term in the text as opposed to finding something that has every keyword in that is in that text like if you think back to the previous example with the dancer the image showed the dancer doesn't didn't show a boy a brother or parents or whatever and then another way would be to use convolutional neural networks and there's been a lot of research on this last year especially the other way around from going from text to a sorry from going from images to a sentence that describes it but we don't feel like those are accurate enough and even if there were we there's not really a data set that has all of the image classes that we're interested in because we must be interested in emotional things almost used search terms actually love so an image net has like a thousand kinds of dog breeds or something like that which is not really what we're interested in and then our own data set would be too noisy probably and also you would have to do face detection to detect emotions so this approaches out so back to the pipeline so what I left open is how we actually generate those search terms and we tried many different approaches including just hard coding a list of terms that tend to generate high-quality images and then going a little bit further is doing sentiment analysis on the text so if the Texas negative sentiment we would get give a term like rainy day or storm cloud and if it has positive sentiment then we would do sunny and beaches and people smiling and going another step further is actually extracting a keyword using tf-idf and scoring so this were those approaches all work okay but we found what works the best is actually making use of all the image searches that we already have which is a giant data set and a learning from that what people search for given a text so and then what we use it we when I say it works better than we used Amazon Mechanical Turk to ask people hey which image fits the most and we also did AP tests where we would use different strategies to see where people create more whispers when they abhorred the create process and when they would do more searches or they would accept the suggestions so a first approach for learning from that data set was using nearest neighbors so we just retrieve the 20 most similar terms and I was 20 most similar whispers to the whisper that someone was trying to create and then get the terms that associated that are associated with that whisper the image searches so we only use was for several created using image searches and then scored them somehow so the main advantage of this approach is that it's better than what we had before which was nothing so but the main disadvantages of course all the reasons that people use don't use nearest neighbors so mainly that you the time that it takes to actually generate a suggestion is dependent on the size of your training set so then we moved to actually using similar terms so we're treating similar terms instead of similar whispers so we would compute the cosine similarity each term but we represented as a back of words so as one giant document and then we had computed cosine similarity between the term and the newly created whisper and would add a prior because otherwise you would always get some very rare terms that match to the new thing that you're trying to create because it just happens to have the same words and so the main the main advantage over the previous approach is that it's more scalable and as a side effect we also have a fixed size vocabulary now so we can actually pre-compute or we can create a collection of of images for for every term so we had about 36,000 terms and the accuracy of this approach was two percent which is actually a lot if you think about because out of 36,000 terms in two percent of the case would guess correctly what that person search for and there's many possible searches that would be correct the quote-unquote for a given whisperer so now we would have a fixed list of terms and we'd have our own repository of images and would generate that repository offline and we also removed some low quality images by doing text detection so we don't want images with text on text and by removing images with white backgrounds and removing low-resolution images so this worked relatively well and to be honest we actually still using this in production right now but we've already developed the next version where we looked at the youngest as the classification problem as opposed to an information retrieval problem so basically here we take the back of words as an input and then create a 36 thousand dimensional vector and then use the softmax so we can view it as a probability distribution so it sums up to one and would use stochastic gradient descent to learn so the main advantage of this is that it optimizes a metric is very related to accuracy so you'll be used cross entropy well the main disadvantages that's actually not feasible to learn this it's so w alone which will be 36,000 by 1000 which 100,000 was the size of our vocabulary is would be 16 gigabytes so why that might fit in memory I would take HS to learn so maybe you could do something with like sparse representations but what we did instead is use a neural network model with what we go to a thousand dimensional hidden state and then we use rectified linear units as activations and a degree to train it and we trained this for three days on the GPU and again this directly optimizes the metric related to accuracy and it's now feasible and this actually gave us thirteen percent accuracy which we were actually very amazed by that this is possible but it's partly because of the long tail in what people search for so the top 100 terms account for i think twenty percent of all searches and then what we did next is and you probably see this coming if you've been following recent development in natural language processing is used by direction LCM and unfortunately don't have time to explain exactly how this model works but we basically it's an alice am that goes forward over the text and also goes backwards and then we use the 36,000 dimensional softmax to get the final state so and after seven days of training it this actually gave us fifteen percent accuracy and for late model with two layers this gave us seventeen percent accuracy so we're planning to implement this model soon and i have some examples prepare for you where we actually did better with the IBM model so I cherry picked some examples where we actually get different results for other models so this one says I rearranged my room last night and I had trouble sleeping hopefully I will get used to it so the cosine model gives us trouble which is okay but it's not really a good image search term it's very abstract then the back of which model gives us bad the lstm model gives us bedroom so that's a two-layer LCM model and it's actually what I search for so here we were able to correctly predict what is it wrong then any girls want to chat with the sailor which is also a common use case for a platform to be honest which two people just want to share with other people and the cosine model said Sailor Moon which is of course completely incorrect it just happens to have their void in it then the back of which model said girls which I guess is ok but the LCM models both said years Navy and then the actual search term was Navy and not US Navy so but of course like those search terms are equally good and then another example that I have is I want to love love like in a romantic 80s movie so the cosine models at 80s which I guess is ok then the back of which model set a notebook and I am honestly not very good for films but this appears to be a 2000 film so it's actually wrong so that the LSD em model but the two-layer LCM model said pretty in pink which is an 80s movie so it was actually worked and but the actual search term was clueless which is a 90's movie so our models more smarter than our user in this case so that's it I guess we have some time for questions and also we are actively hiring a data scientist right now we're actually located in well as California right actually our offices in this picture so if you have a fields too cold in San Francisco come talk to me after the talk yeah yeah process in this talk this general question like how do you clean now or has you need some kinda boxing yes all part this is because it depends on the sources of course if this was so very very clean and not obscene then we don't really have this problem so we don't really have a lot of obscene images but also because it's static we can if someone sees a bad image then we can actually manually remove that image so we can keep improving it over time no we do that manually we do a lot of moderation and menu work as well what is difficult because we have 1 million images so yeah no oh this is this is about the lstm model right so we that is actually model that views the text as a sequence of words so there's not really a notion of history yeah I think whoops this one right so this is the whisper yeah Sega yeah they're two different ones so also I'm not repeating the question so this so this is the post it's a short piece of text on top of a background image and this is what you can view inside of the app and then you can reply to it or you can chat with that person he sauce I the images that are there so the question was how do how exactly do we attack the images and we have of the answers we have a variety of third-party sources and we try to take the searches that our users search for and try to find images in those third-party sources that correspond to the searches so other than that we don't have an ontology but if we cannot find an image for a given such them then we will just not use that search yeah so the question was how do we avoid having the same image over and over and the answer is we try for search terms that are suggested a lot we try to get a lot of images and we randomize the images that we present to the user and we generate for search terms for every post and were limited to 10 suggested images but we do have some images that appear more often then so maybe images are used 10 times per day for example and that's acceptable because a user can only see so many posts any other questions thank you