data.bythebay.io: Samiur Rahman, Building semantic search using Deep Learning
Recording: data.bythebay.io: Samiur Rahman, Building semantic search using Deep Learning
hey guys uh so today we're going to talk about how uh you can build your own semantic search engine uh we can talk we'll talk about like what that even means but essentially as a summary it's more of a conceptual search than just like looking for phrases in uh text documents that you might find a lot of you might be familiar with elastic search or solar that's usually what they do they try to find like tokens of phrases inside of the documents for what you're looking for but conceptual search is a little bit more than that and we'll we'll jump into that before we do to give you more context of why we were even solving this problem why you would even listen to me uh first let's talk about what mmark does so I uh head up the data engineering team at mattermark I've been there for over two years now we by data engineering really most of what we do is a lot of machine learning and a lot of that is in in the text domain so a lot of NLP so why do we even need to do NLP so what mamar does is it tracks and organizes information about businesses we kind of collect information about 1.3 million businesses information like who who what people work there what industries they're in how how many visitors they're getting on their websites what their mobile traction is what kind of social media interactions are being put out for them what news is coming out about them and uh how many employees do they have over time things like that so you know to get all that data lot not a lot of that is like available in structured form on the internet if it were then we wouldn't even be creating bmark right otherwise there wouldn't be a business proposition but uh so a lot of what we do for NLP is we extract these kinds of structured information from unstructured things like websites news articles um so on actually if you if any of you attended my colleague Evan's talk he talks a a little bit about how we do like one of our information extraction Pipelines um we also create proprietary data so you know predicting what companies uh what industries companies are in like knowing that uh mamar is in the finance industry or galvaniz is in the education industry no figuring that out there's no uh kind of taxonomy out there telling us like what what what these 1.3 million businesses have in categories so we have to create that ourselves we we also have features like figuring out what companies are similar to each other automatically so a lot of um unsupervised and supervised machine learning there and what we're going to be talking about today search so how do you find what company you're looking for a lot of that could be structured search like U for example this one is show me all the companies ranked by growth score uh this is our own proprietary metric for how fast a company is growing that are is in the I don't know if you guys can see the query but uh on the top level we we're looking for Bay Area companies that have at least 10 employees and fewer than 200 so that's a that's a kind of search right it's like filtered search but there's also textual search and that's what we're going to be talking about today uh I think combining these two things has made mattermark a really powerful platforms for platform for you know salespeople VCS um and the the search Improvement in search search is a huge part of that so let's talk about what traditional search appliances are so a lot of what you're probably familiar with are you know databases that do their own search like Pro postgress provides its own full text search it's a very like rudimentary version of uh search that can find tokens within uh text bodies that you're putting in your database uh there's more intelligent ones like Lucine based indexers like aache solar an elastic search that can take documents and then index it for you and then really quickly serve back results based on the phrase that you're searching for there's also managed products like AWS Cloud search which actually in the back end is either solar or elastic search and then there's algolia which is a company that handles all of this for you and you don't have to do any of the kind of Ops level work so wait you might ask if there's so many products out there why would you want to build your own well at first we didn't um we use elastic search and we still do but um we get we'll get into how we use it now in a little bit so we use the lasic search to do autocomplete right so all of you are familiar with like you know let's say I have 1.3 million companies and I'm looking I I know which one I'm looking for so I start typing in m a t t e r uh I should get a list of companies that are you know start with matter so that I can find mattermark really quickly so autocomplete we use elastic search for that we use it for full text search so by text we we you know scrape a lot of information about companies we use machine learning to figure out what news articles are about what companies so we have like all the scrapes from the company's website the company's about page we have information from their Twitter uh Twitter account of like how they describe themselves all that stuff so we have a lot of text from both the company's own information but also like news articles about how companies are being talked about in the media so you know elastic search is also really powerful as I was showing you to do filtering on non-text attributes that are like Industries number of employees location all that stuff so it seemed like it was a really good choice early on so and and it worked really well so other than the pain of uh I don't know if anyone any of you have used elastic surge before but it's a pain Ops wise other than that um it customers were really excited to be able to perform full teex search on companies and the fast auto complete was really great too so um you know why why would we even build our own sming search so really in the end results were okay like after the initial hype that we you know when we put this out to our customers the initial hype was oh this is so great now I can search for terms like sharing economy or uh you know the new buzzwords like home automation or something like that and a list of companies would show up turns out that the list wasn't very good it was better than not having a list so they were excited at first but later on we realized that it's not really serving the need that the customers have um so plastic search solar Lucine all these indexers use a pretty basic NLP technique to figure out what should be shown and it's based on matching algorithms on bm25 and tfidf so these are very token level matching systems right if this is great for looking for like an exact phrase so if Uber mentioned sharing economy on their website then it would be a great way to match that that's not always the cas because people are looking for Concepts so like uber doesn't say ride sharing or sharing economy on its description or its Twitter so when you search for that you're not going to have it come up so the customers aren't really getting what they need because they're doing a conceptual thing like in Google you can get results for web pages even without exact phrase matches because Google is giving you a conceptual search result so now having that problem what do we do uh you know we have a strong NLP team so we decided wait we we could easily build our own search um so what we needed was semantic search so obviously I'm not going to read this it's Wikipedia um probably should have prepped a little bit better with my presentation slides but I'm going to leave that up there so what does semantic search mean it's essentially like can I search for a concept rather than a phrase so in this case I want to search for companies that match a given concept uh so like chip manufacturers or cognitive training or sharing economy or things like that people will be looking for a concept that matches a company that a company might not necessarily mention themselves at least not in the exact words that the customer might be looking for so what did mamar do so before I get into how we built it let's why don't we do a quick demo of uh what we ended up building and how it compares to last six search so I'm going to go quickly over so on the left side I don't know how much how much can you guys see there should I be zooming in a little bit more yeah let's see it's a little bit more zoomed in yeah that kind of breaks the site but you know who cares um yeah so right now I have have just like I mentioned I have chip manufacturers as a search on the left side which is our old system and then on the right side it's the new one with the machine learning kind of back back end so obviously at least okay maybe it's not obvious but I think the right side is better right wouldn't you agree um anyone else want to try why don't you anyone want to try any different searches to test me out see if this is I don't know worth even doing sure you talking right why don't we give that a shot so I'm going to do this is the old one is my internet not connected ah I got logged out sorry guys okay so the the right one is is already there okay so actually on the left one it's not terrible it definitely got some of the stuff probably lift mentions ride sharing on its site but Uber doesn't Ola doesn't grab taxi doesn't all these things right so on the right side we have a lot more results we've got 76 there and 22 on the left side let's try some crazier stuff I don't know this was the righty o I don't know if that's going to work that's scary man oh Facebook you broke my system man but at least it got some some of the right stuff break the system yes let's try breaking the system and let's try crazier searches traffic sorry go ahead traffic cake traffic cake what what is that even supposed to bring up all right sure no result there I I'm I don't know what this is all right how do you deal with named entity recognition well it's just kind of like cakes so I guess it kind of got confused and said you probably don't want traffic cake you just want cake um but yeah this is obviously no one mentions traffic cake but it some I guess it works sort of go ahead howy oo I like that okay so some companies I guess o is bamboo HR related to employee morale maybe I think these these comp no no no on the right side is the con conceptual search and I think the top three are related to employee morale and on the left side is just stuff that I don't even know if these are actual companies cool anyone else we can move on if huh dat science Miss typos which one I don't know if that's going to work that's definitely not going to work but I don't I don't know about this one well I don't even know what these companies are something yeah I think let's make sure that the actual one works okay cool all right awesome nothing else we can we can actually get into how this works cool so so hopefully you guys think this is actually worth doing um so what did we actually do well one of the problems that we have that Google or bang or actual search companies that aren't a tiny startup have uh is a lot of data they have a lot of click-through data um did you have a question sorry okay um uh so we definitely don't have enough click-through data we're a small B2B company that serves about a, users a day so we don't have anywhere near enough searches to say we can do this with a statistical significance so we really couldn't create a supervised machine learning based ranking method well tough luck so what do we do now uh obviously the hot take of the last few years has been can we do unsupervised representation learning obviously deep learning or other kinds of things but can we one of the ideas we had was to project both companies and short phrase queries into a single Vector space so if we could project them into the same Vector space then we could use a distance measure to say what is relevant and then show the results that are above a threshold of relevance obviously a lot goes into to figure out what the distance measure measure measure should be a lot goes into figuring out what the threshold should be uh we're going to talk about how what Vector space we even mapping these things into what are the things you might people who are familiar with uh NLP and topic modeling might say is that well isn't this a solv problem couldn't you just use like LSA and LDA that's to do topic modeling on your company documents so I expected it obviously I'm going to try it but I expected LDA and LSA to perform poorly mostly because they tend to have biases for the length of the document that you're vectorizing so it's great for comparing similarities between similar size documents but when you have something like search queries where what you're comparing similarity to is a short query phrase and a very large text document it's not it's it tends to not be as accurate when you're comparing between other documents so even though I thought that I still added it as like a baseline comparison right um obviously I'm also using like the base elastic search implementation as a baseline too so then we went through uh if any of you are familiar with what we do at mattermark we've done a lot with word Toc and combining with tfidf so we we definitely wanted to try that as a as a way to generate document vectors and query vectors we used uh paragraph vectors another extension on top of uh word vectors that kind of tries to encode large text documents into a vector it it's also very good at uh encoding short documents into vectors so that was that in my mind that seemed promising uh there was a great paper uh by I forget the name Ryan Kiros I think uh about Skip thought vectors and in which he talks about how he could use lstms long short-term memory networks or convolution neural networks to uh encode larger bodies of texts uh the way it would work is it's basically can you train so if people are familiar with word embeddings it's kind of how do we train sentence embeddings that with recurrent noal networks or convolution noal networks that tries to take the previous sentences in a document and predict the next sentence so that kind of um neural network so if you want to learn more about that approach it's a little little newer than some of the other ones uh I have a link to the PDF otherwise come up to me I can tell you a little bit about that so okay we're going to you know we're going to try all these approaches but how do we know which one's better so this involves even though we don't have a lot of click-through data we wanted to figure out what would give us the best results for the things that most people are searching so we at mattermark we have a dedicated data analyst team that helps us with a lot of these kinds of things so we identified what are the most commonly searched phrases so things like home automation adtech sharing economy fantasy sports a bunch of other things and we ran experiments on that so um we did like multiple iterations with our data team like generating results having them tag like is this relevant to the thing that people are searching we collected that data and that's how we compared all these different approaches uh that's also how we decided on what threshold we should be using right uh this is the kind of approach this is it's an approach that's worked really well for us like an approach that help works with an internal data team who are sort of experts on these topics right uh and generating data from nothing sort of if you have any questions about that I can answer that later too so after doing all these experiments we landed on paragraph vectors so the results from the lstm's convolutional neural network models were actually very similar to paragraph vectors the the actual like precision and recall of those approaches however paragraph vectors were far faster to calculate on arbitrary inputs like you know when search queries were coming in we need to calculate the the vector right right away and return the results um on traditional Hardware we needed to run we needed to be able to run this stuff in in the cloud in AWS inside of our regular software stack so we needed stuff that would you know AWS has GPU machines but they're expensive and not really as good as something you could run on your own computer so we needed models that would still be able to run well on CPUs so what we ended up doing was we used we used cosine similarities for the documents and query vectors and returned the ones above a certain threshold uh and the threshold was calculated from all these experimenting to figure out what's the best like Precision recall uh cut off we want so let's talk about all the other stuff because even though we decided on a Model it actually wasn't quite the most difficult thing in our uh system so we needed to integrate our powerful ml based search model into our current es-based infrastructure because we had all these faceted search queries like you know being able to search about Industries or location or how many employees a company had it needed we needed to be able to do that powerful querying alongside the search queries so you should be able to say can I see the sharing economy companies that have more than 100 employees so to do that we're we're already heavily invested in the elastic search methodology so we really needed to back into integrating those two things together we also needed to ensure that we can process hundreds of search queries in a second because that's probably the max we're getting today um we we should also be able to scale out like we should be able to throw in more machines and it should scale that's that turned out to not really be the issue the issue turned out to be well how do we actually integrate with elastic search um we took we looked at a number of different approaches we could have built our own Java module I'll explain in a second what this graph is about uh we we could have implemented our own Java module that plugged into elic search but it's a lot harder than you would expect uh one of the things that I noticed was that solar and Lucine actually makes it way easier than elastic search to create your own like custom scoring functions than elastic search does so keep that in mind if you are ever trying to do this again maybe use solar instead of elastic surge um so what we did was we fronted elastic search with a proxy that would figure out if something is a full text related search and then do some proxying before we get into that we also built an ml an an an ml search service basically the the thing that we just talked about well it's a model but how does it respond to real time input from customers right to do that we fronted it with a simple rest API that would accept short phrase search queries and then it would creep well not then uh it keeps a paragraph Vector model in memory and also keeps the pre-computed company vectors in memory um every once in a while it will do a batch Rec comput recomputation of that of those uh company vectors when you know information about companies change which so we'll do that like once a month we will recompute the vectors for these companies and uh restart all these services so it keeps that in memory and startup it turns out to be like 8 to 12 gigs of memory which isn't that bad for modern servers so as queries come in it computes the vector for the query then does a cosine similarity um against all of documents in in our company Vector space and then Returns the matching document IDs for things that are above the threshold we we wanted so so so that's like part of it it will return like document IDs but you know to make this integrated into elastic search like I said we needed to have this HTTP app that would act like elastic search anyone who is quering elastic search wouldn't need to know that there was some fancy magic going on in the back it just needs to know oh I want to query some text so it look the same to everyone and the results are just better um I I want to put a plugin here we use closure first time we got closure into a production stack I'm a huge fan of closure um and it was the right choice because it had really really good stability and throughput anyway um so it detects if queries and when a query elastic search query comes in you know there's a there's a DSL to this so if the DSL for you know if the query has a full text query in it we have ways to figure that out so if that's if if there is a full text query in it it forwards that query onto the ml search service which then comes back with the document IDs that should be included and then a re a query is reconstructed and forwarded to down to elastic search which then Returns the actual results so to a user it doesn't seem like anything's changed the interface is still the same yet the results are better well actually I didn't mean to end it that abruptly but yeah that's it um any questions uh so in the uh paragraphic uh Vector model uh so for each paragraph uh do you actually represent that as a company name plus description on the company so how do you generate a paragraph right good question oh I can just use this one right um good question so what we actually did the paragraph Vector model itself was trained on um trained unsupervisedly on lots of news articles that we have so like that is domain specific information right like we have news articles about businesses like 4 million something news articles so we train the paragraph Vector model on that and then so the model is trained we pass in to encode into a single Vector a concatenated version of all the text that we have about a company so like it's description it's description on Twitter it's description on other things and then it's about page text it's FR it's landing page text so taking all of that text together we passed it into the um the model which gives us a vector for the company so do you um create a vector for each in individual terms in the query and aggregate it together and then use it for matching or do you create one single Vector for using paragraph to V and then does it matching yeah good question so no we we create one vector in the same way that you would put in a large document into the paragraph Vector model we put in the short query phrase um we did test what would it be like if we just like concatenated sorry we just summed up the vectors averaged out the vectors of each term turns out that's much worse and it kind of makes sense because um it does it loses the meaning of order that paragraph vectors can have for you because it uses windows so the way paragraph vectors encode a vector is windowing through whereas if you just use the word vectors you would lose that so like sharing economy would be the same as economy sharing where as in paragraph vectors it wouldn't is it similar to like actually Compu the vectors using word to work and aggregate it across all the document right how does that approach compared to what you're doing using paragraph to work um so I think in in in uh economy sharing sorry sharing economy and economy sharing spark would be do the same thing in this aggregation because it would just be summing the vectors so addition is uh transitive is it transitive anyway it can it it would be the same on both directions right uh I was wondering what uh paragraph vectors implementation you used and if there were any tips uh as far as the parameters dimensionality modes that turned out to be most useful cool um I'm assuming you're familiar with Jensen yes yes cool U otherwise that question it's a little hard to answer uh because we use Jensen uh Great Library by the way for text analysis probably the best word to VEC Library also um anyway what we found found out is that we we tried a whole bunch of parameter optimizations actually across all of our like models that we tried the lstm the RNN uh sorry the sorry the CNN we we Tred to optimize all the hyperparameters right we it's not like we took shortcuts because we were like oh we have a favoritism towards one of these models I actually didn't know which one would be better but so for for those who are who under who know paragraph vectors you could do it in two different ways you could do the um distributed bag of words or you could do um DMM I forget exactly what the acronym is but uh one of the D DB works better uh for comparing short phrases to large documents than DMM does uh but we found that out when we did all this hyperparameter tuning so um it turns out that if you you definitely want um shorter Windows not too short something around five if you're doing this kind of textual search um and you want to use DBW what about the dimensions it um typically it's said if you sparklet it's more than anything more than 300 is better U but it has an infrastructure limitation the memory aspects Etc what have you used successfully so we tried three different things we tried 100 200 and 300 with 300 we we're also able to do something where we were able to initialize the word vectors with pre-computed word vectors uh so in in training a doc model you you're using a word Toc model really in the in the bottom uh but what you can do is you can initialize the word vectors with the kind of general purpose word vectors even even though you're training on a domain so so what we did was we used the glove implementation word vectors the the general glove trained on common crawl so you could do that on the 300 dimensional word vectors because they put out 300 dimensional vectors but they don't have three like one 100 or 200 dimensional vectors so we did the hyper parameter tuning that was part of the hyper parameter tuning the dimensionality but we ended up with 300 Dimensions with the intersection with the glove vectors and that worked really well more questions okay then thank you so much Samuel Rahman metamark thank you thanks guys since I have a little bit more time I'm going to make a PSA and say uh not a PSA but an advertisement more uh we're hiring machine learning engineers at mattermark if you want to you know if you're interested email me we're working on some really awesome stuff at least I think so cool cool thanks guys