Text By the Bay 2015: Gershon Bialer, Identifying CrunchBase Entities in News Articles
Recording: Text By the Bay 2015: Gershon Bialer, Identifying CrunchBase Entities in News Articles
uh show hands everyone who's who's used crunchbase and uh who's edited crunchbase who's that who's put their name or their company's name on there one a few of you all right so out of this crowd a lot of you have actually uh used crunchbase and that's that's part of what makes crunch space a little bit different than a lot of these other things is we also have a lot of user input that goes into crunch base um and so what crunch space is you know is we're crowdsourced you know we also have uh the nlp aspects that i'll be showing you about um and we also have data and our data is used by a lot of different people we're used by matter mark we're used by data fox we're used by different academic papers and all these things uh you know we're transparent anyone can go to a crunch base and use our data and we have a lot of entities in our crunch base as you can see we have many you know people we have companies we have investments in different startups and we have you know a lot of different users that are actually using crunchbase and so this is sort of the basic entity page in crunchbase so this is the entity page for apple you can see that we have the you know description we have information about people who've worked at apple and we also hadn't have the website and you know we have so many people so many organizations and so many products now there's also the news and every day there's a lot of news articles that come out about each of these companies each of these people and each of these entities that's in crunch base and so what we we're trying to do is we want to be able to take these news articles and put them into the crunch base entity pages and then we're able to gather more information about the different entities that we cover on crunchbase and so basically what we've traditionally been doing is we've traditionally been we have a teams then india and locally that go through the stuff we also have people in the crowd who are editing crunch based entities themselves and so we have a lot of data that's already come on to the site but there's also there's a ton of news articles that come out every day you know as you can see and we were trying to add want to add all these news articles to the news section on crunchbase so we designed algorithms to add all these news articles to the news section on crunchbase to figure out you know which uh entities they're talking about and you know which person they're talking about and to disambiguate these things so to do that we sort of came up with a high-level architecture that involves basically we scrape the web then we put that into a news database and then we parse all the stuff and then we go to the production database now the thing that we have to you know keep in mind is we want to parse out millions of articles and we have um you know a lot of very large number of entities that we're trying to identify in these news articles and so you know we had to design you know some infrastructure to handle that and to sort of match with our existing uh data so we obtained the news actually about 100 000 articles per week uh we have our bot that's scraping on all sorts of different sources and you know looking for the latest news articles uh that are in all the sort of tech publications uh and we're adding more um and then once we get the news articles we have to scrape the news articles to get out the content in them that's sort of the main body of the text and find you know which part of the text the article is uh contains the body of the article from the raw data uh and so basically uh we didn't really you know put the most complicated system in place uh for doing it uh we were able to really get away with a pretty simple system and you know what this really shows is that a lot of nlp is not necessarily having the most sophisticated algorithms but is also having the data available that you know identifies which entities are are which um and so basically we just uh you i just use the p player speech tagger and we identified the proper nouns in the articles and then basically those get run through uh crunch base and you know we also look for like companies and things that are not uh already in crunch base for our teams to add when we run the things through crunchbase this is a chart that shows sort of the candidate entities of what what we find so basically what we're doing is we're looking at the different uh strings and the different articles and we're trying to see um how many of those strings show up in different uh are actually crunch based entities so you know we look through basically the articles and then we run each proper noun basically just do like a text search through crunchbase and we say you know is this apple does this show up uh you know we also look for some companies names are a lot more ambiguous if you have a company like drop that might you know be uh that could potentially be the word drop or the drop and and some of the company names are a bit ambiguous um and so basically uh to develop our algorithm it's sort of been an iterative process where we initially we just sort of matched sort of the direct matches of the entities and then later on um we looked at you know trying to match less direct matches on the entity strings and the reason for doing this is we're able to instead of having to tag the full articles we were able to just tag the stuff that was matched with the candidate entities so basically we have cisco so initially cisco might ever match just whatever we had labeled as cisco in crunch bay so if you not familiar there's multiple companies named cisco there's cisco systems there's cisco investments there's also various people with the last name of cisco and so we had multiple people you know in crunch base but you know basically initially we just sort of tried to do the direct matches and then we'd sort of tag and see whether that was the correct one and so then we were able to tag less training data later on we sort of expanded that and we were able to um you know have train more data as we sort of the project progress but it allowed sort of an early prototyping stage without having to necessarily train as much data and you know part of it um also with the matching of the names was looking for aliases now crunchbase has had some sort of aliases actually in the entities of crunchbase for a long time but one of the things that we're able to do that you know not necessarily everyone is able to do is we can actually you know have people manually go through and edit our data and update our data and when we updated the aliases in crunchbase ourselves you know we have a content team that's able to look at different news articles and different sources and then you know they're able to gradually sort of keep the data um they keep the data up to date and so then you know the higher quality sort of our initial data is the easier it is when we apply the algorithms to determine you know which entity matches which other entity and so one project that was particularly you know helpful with this was basically trying to uh you know clean up our aliases and so you know here's just the simple alias for google mail which you know for gmail basically and so you know having that alias correct means that you know we can just match that string instead of having to necessarily come up with a you know more complex algorithm to figure out that that string is referring to the same entity as google mail in the article and so some of the difficulties in this was that our data is not just editable by our staff but anyone can go to crunchbase and edit any entity basically and we do have moderation processes and such but people make you know often make mistakes in how they edited entities uh so one of the problems that we dealt with was first aliases a lot of people would sort of confuse the elias field with the location and so they put in names like san francisco california or united states or even dayton ohio and then when you know we went to go match those entities we'd realize that those entities you know some article was mentioned san francisco and then that would get matched against some obscure company that had you know that had just happened to be located in san francisco and so there were a lot of sort of errors in the training you know in labeling the initial training data so it was just based upon the issues of data quality so you know we have two sort of aspects of what we can do when we face these sort of errors one one thing is to go back and clean up our data another thing is to try and you know make sure the algorithm can handle that and so it's sort of a combination of the two of improving the algorithm on one hand and then also going back and improving the data quality on the other hand you know because we have to sort of keep the algorithm resistant to further changes in the data quality where even if we clean up the data at the moment somebody might come back in the future and then they'll edit that entry again and then the data will be further corrupted again and so we want you know our algorithms continuously running parsing out the news articles and we're running on the latest and greatest of data that are that's in our system and so we need to make sure that the algorithm is is capable of running on that latest data and that the latest data is going to give us an accurate recognition of the particular entity now when we decided to recognize entities in articles we really looked at the two things presence and relevancy and the reason for this is a lot of articles can have a particular entity mentioned that may not necessarily um be what the article is about and for for us for our purposes um like a lot of articles mentioned ipad iphone apple if you if you look at the business and tech news just about every company that's coming out with an iphone app is going to mention you know iphone although iphone is not necessarily for a crunch based entity page for iphone we don't necessarily want to mention every iphone uh article that mentions just on iphone app because that may not really be significant to apple that's not really saying that apple's improving their iphone or doing something new to the iphone it's just saying that there's a new iphone app there's other cases if you look at articles where there's basically just something mentioned as an aside in the article so basically to make the training process easier we really split it up into the the two criteria of um presence and relevancy so the presence uh is just saying is this present in the article uh you know no matter in whatever form it is and then relevancy is is sort of a more complex criteria that that was sort of more difficult to determine and the reason this is useful also for a training purposes because the presence is a lot easier sort of to determine based off of the data when you're training it because when we maintain the training data basically we just have sort of a google spreadsheet where we had people label different things and so if you're going to try and determine if the thing is relevant to the article that's a very sort of high cognitive task determining looking at through the article and saying is this entity actually in the article you just look for google and you say is this talking about google the company uh more complex entities is there's a startup called say friday that does something or other but you know every time that the word friday is used in an article it's not going to refer to the startup friday most likely and so you know those those type of things you know can get disambiguated out with the presence but uh not the relevancy so here's sort of just um a sense of like how we labeled the data um for various articles so if you look at the spreadsheet this is just the google docs spreadsheet so this is sort of what we actually used for labeling the training data so you can see that apple is both relevant and present in the particular article this is referring to and a bluetooth is mentioned in the article as present but it's not really relevant the topic of the article i think the article uh mentions that bluetooth is used in the iphone or one of the devices in the article and so you know you can sort of see this and so we're able to sort of try and disambiguate this down at the bottom you can see a lot of different place names locations are actually a bit tricky because there's a lot of a lot of like obscure towns around the world that have names that are very similar to a lot of different companies names you can see marvel arkansas and that that's you know there's a company marvel which is probably what that's referring to but the little town is gets picked up and so we're able to sort of filter out most of those because there wasn't a lot of entities associated with those towns and so we were sort of able to eliminate that but sort of from a pure matching perspective we sort of had to see what that was and we tried some methods of disambiguating the company names and the product names although that wasn't necessarily all that effective for what we did so the features that we used for identifying entities a lot of the stuff was based off of the graph and so basically the way that crunchbase works is we have a graph that throws the connections between all the entities and we actually use a graph database and so for features we're able to look at the first degree connections between entities which was like a single hop in the graph second degree connections you know which is two hops and sort of different um ways of sort of looking at the the measures of the connections between the entities and that proved to be a very useful feature uh in looking at you know which ones we also looked at like the total number of entities that were matched for like a given connected for a given entity and you know that sort of tells us if that entity is a popular entity that's important in crunch base or that's obscure startup now doing those types of features are a bit tricky because new companies get added to crunch space all the time and so those features may not be in existence yet because maybe we don't necessarily have full data on each of the companies that's added to the crunch base and so for those there may be less data and matching is a lot harder to do um and we also look you know from on that no we looked at more entity specific features which were sort of you know looking at just the type of we're trying to look at the type of entity in the news articles we didn't really get that sophisticated we just used some open source uh libraries to identify the entity type uh but it that wasn't necessarily totally indicative because uh you know there can be some uh references and articles that refer to two different entity types sort of in the same text where things get tricky it could be like microsoft windows there's there's names of companies that are also names of people and it can get tricky in trying to identify those but that was another feature that sort of helped in the end more entity specific features the other thing you know is obviously just like tf idf type frequency features looking at you know which entities show up a lot of news articles and which uh entities you know are show up and are actually present in the news articles um using sort of a second pass of the algorithm for that and so using that we were sort of able to figure out you know have a bunch of features that we were able to apply and basically the algorithm that was used was basically you know just to summarize you take the proper nouns in the text we look for you know candidate names that match apply a few some heuristics and we determine if the candidates are relevant uh and basically i didn't do anything too fancy basically just i actually used the random forest implementation r was able to just export it to ruby and then run that in ruby in production which would sort of fit in with the rest of our production systems i was able to really keep a really very simple implementation that was quite effective so here's sort of this roc curve of the entity resolution so you can see that it's quite accurate uh model uh that uh we tried to uh keep down uh the the you know keep keep it keep it accurate and so this is basically what we used um and you know not a and you know some stuff that we found with you know now isn't necessarily uh relevant and the relevancy does does sort of vary depending on the type of entities as well because a company like apple shows up in the news a ton of times every single day it'll uh and some new startup will might show up in you know some sub a reference in some article once and that new startup is you know the one news article about that startup is going to be important and significant just because that's the first time that that startup appeared in the news and so if you're working for that startup you're going to care you know apple on the other hand it shows up so often that uh sort of the relevancy is is not just uh you know a matter of is it relevant in the article itself but it's actually a function of how often it appears and so that that sort of metric uh sort of helped play into trying to determine what was relevant so just sort of to summarize we have a lot of different uh entities in crunch crunchbase and we're parsing through a lot of different articles to figure out which articles go on to crunchbase and we consider probability relevance and frequency combining you know manual curation in the community with sort of the automatic natural language processing stuff so any a company's questions startup yeah it's more significant that if it appears on the news can you somehow adjust the factor of relevancy by this novelty aspect yeah well i mean it it's just a matter of how many times how many news articles the startups appeared in before so if the startup has appeared in a thousand news articles before the relevancy is going you know the relevancy score is going to be different than if that's the first appearance in the lifetime of our collecting the data so that's basically how we adjust for that other questions uh what about the training the sets uh so basically what we did was we looked at um the strings that matt so we basically did did a look of uh the proper nouns and then we did a text search through crunchbase to see which possible entities would possibly match and then we just um and so then those were the candidates and then we'd mark those yes or no the reason we got a lot of no's you know because there's a lot of entities that just have similar names to the ones that were in there or there's ones that yeah there's locations that have similar names as well yeah yeah i mean i mean the negative examples would be the ones that matt this if the string doesn't match the name then that's just not considered basically if um in terms of you know the mat i mean we're only looking for proper nouns too if it didn't get tagged as a proper noun properly that would sort of get excluded and that wasn't really considered but the proper noun tagging is pretty good um in terms of things you know i mean stuff that that doesn't get tagged as proper now it's it happens occasionally but it's you know that's pretty strong you know if this string was was way off or something and different there may be a few cases where that occurred but that wasn't too frequent so yeah so the most important was the graph features that were basically you know if it says steve jobs and it says apple then we know that steve jobs is linked to apple and so it's going to be a very strong indication that it's about apple instead of about i don't know some other company named apple so that was that was definitely the strongest feature there were others that were just is it an exact match uh but in us also another thing that's that's not necessarily uh i don't necessarily want to say it's the strongest feature because is just looking at which entities have appeared a lot in the news before we know that apple computer appears a lot so you know that's a strong feature in a sense of of saying that you know because it disappeared a lot it's likely to be there uh those are basically the strongest it's just one way so you're trying to extract knowledge entities have you considered maybe the other way like changing people to actually annotate their articles using new entities directly so that you know they they tell you they communicate oh this is about albert this is the i mean people there are people who are actually annotating stuff with crunch-based entities they just use the url typically i mean internally we call it the permalink basically of it'll be the slash organization slash apple where if there's multiple entities that are the same name then the url will be slightly different so that's basically i mean internally we have a uid but it is most of the time it's easier just to use the permalink any other questions all right you