sftext.org: Babak Rasolzadeh, NLP and Data Science at Meltwater
Recording: sftext.org: Babak Rasolzadeh, NLP and Data Science at Meltwater
hi everyone water I think a few of you at least familiar faces that I've seen before at similar events I wanted to start by asking a couple questions just to get a feeling for how I should do the talk how many of you here are engineers or have engineering backgrounds all right perfect how many of you have computer science backgrounds within engineering perfect okay so so good I think the level at which I have made this talk is not going to be too too high it might even be boring for some of you but it would be a good good good level I think there's a few equations and stuff in there don't don't get scared I'll speed through those all right let's get started this is the outline of my talk I'll give you a quick company background and what data science and in particular NLP means that not water I'll talk a little bit about what the challenges of NLP technology natural language text processing is at large scale and then of course the sort of meat of the of the talk will be entities and I'll talk about what's called name that is recognition on the social domain and entity level sentiment and if there's enough time we'll see how many questions and stuff there is I'll say a few words about the knowledge graph all right what is my water so my water is a Norwegian company originally it was founded in Oslo as at the time a media monitoring company it's its core business has always been to analyze news blogs and as of the past couple of years of course social media for the benefit of customers who are looking for insights from the outside of the firewall so traditional ERP CRM systems of course look at the data that's within the firewall the the company data the business intelligence data and try to deduct sort of advices and insights for for the decision-makers of the company what we do is that we we do the same thing in terms of providing business insights but based on the data outside the firewall and our core philosophy is that the data outside your firewall the data that's outside your company is far more important for your business and your business decisions than the data inside our founder and the CEO is only second who started the company out of a garage in in Oslo with with three guys and and grew it bootstrapped it wasn't until a couple of years ago that we we took in VC so a bootstrapped up to two hundred million dollar in revenue we're about ten employees in 30 countries sixty offices all from that initial garage project and I think $10,000 and our tech offices are a bit more concentrated it's most of the offices that we have our sales offices and what is it why are the insights that we provide relevant why are why are we providing this service at all right of course the informed decisions is at the core of it and your own brand is something you definitely want to monitor out there you're compared it is another thing that you want to keep track of your own industry in general you can use it to to find leads right you want to keep track of partners now they're doing your products how are they doing out there in media all of this are the components of what we call inform decisions and we we believe that when a decision maker and executive comes to work in the morning they should have everything they need to know that day in terms of everything outside of the world outside in order to make informed decisions we call it the morning coffee inform decisions here are a couple of examples of customers besides the big ones the big names there like coca-cola BMW are IKEA these are the more interesting cases in my opinion of use cases of our technology for instance securities uses my water to find out about new instances of vandalism and breakings obviously being a security company surveillance company that's that's important because often those victims are in need of their service right lead generation Sanofi which is a patent technology company uses us to be alert or when certain patents in target markets will expire Gold Corp gold mining company i use in my water social listening to estimate and prevent infrastructure attacks actually kimberly-clark uses us to help determine how public perception of their drugs is is trending or evolving and the indian news channel and DTV uses us to monitor performance and popularity of news anchors so this is just a few examples amongst our 30,000 clients of usages of our service how do we do this what we do it through this dashboard it's a fully configurable dashboard where all sorts of data inputs so these news these blogs these social media channels can be represented in terms of grass where the analytics that we do on these these channels can be very easily digested again hopefully with the morning coffee I won't talk too much about our product and the values of it you're not here to listen to that let's get a bit more into the technology behind it so basically what happens is that a new news document comes in or any type of document really comes in and goes through a pipeline of a sequence of enrichments these enrichments that we support in in 15 languages today there are things such as sentiment analysis of course it's important to know what the sentiment around your your product or brand is thematic analysis categorization keyphrase extraction and of course what we will talk more about today entity recognition and resolution or disambiguation as it's sometimes called and of course the enriched documents then go into a storage and and our dashboard is really an application using that API for high-performance search so we do obviously a bit of other things within my team the data science team other than just pure NLP we for instance have built a recommendation engine that allows our users to fine-tune the analytics the the sorry the content that goes into the analytics of their dashboards so so a real-time recommender engine will look at the consumers or users behavior and determine how to find adjust the specific set of documents that are important for this customer we we work on correlation and predictive pattern recognition also for for for for customer usage of actually knowing what will happen if it has happened a lot in the past I won't talk too much about these another technique that we use a lot is it's Vorta Veck techniques where we map words into a abstract abstract vector space and enables us to cluster words and therefore do searches much much better than just pure string matches so these are just a couple of the technologies that we work with other than just pure NOP but of course our bread and butter is is NOP and text analysis since a lot of you have computer science engineering background I won't dwell too much on this but I thought would be worthwhile to introduce some machine learning technology I'm assuming a lot of you are familiar with what training and testing is but essentially the type of algorithms that I will talk about here today and and our industry predominantly engages with is what we refer to as machine learning algorithms that typically in the supervised case and this is the supervised case require a set of training documents with labels that represent whatever it is you were trying to achieve with the machine learning algorithm and then you show you know the samples the labels basically input output to the algorithm and then you ask it given a new document a new input what's the label alright it's almost like training a kid right you show them how to do something and then you ask them to do it once or twice see if they succeed these sets of course bigger than one or two examples they're typically in order of thousands or tens of thousands so what are the challenges with doing data science or in particular NOP at scale well from a map water point of view of course since we process you know buildings of documents per month social media documents tweets Facebook posts Instagram posts news articles blog posts we need to have a very high throughput so the documents per second frequency that we typically require is about two thousand documents per second right and there's a lot to do so so here's some NOP terminology but there's things such as tokenization limit ization stem part of speech tagging categorization of course sentiment analysis named at to the variation all of these need to be done without any bottleneck alright so how do you achieve that and you of course have racing conditions because when you start looking at these components these modules they all have dependencies in order to calculate the output of one module say that part of speech tagging you actually need a non chunker you you need an a4 resolution for instance so there there are there are non trivialities in doing this in the sense that you can't just scale linearly you can't roll throw more machinery at it you actually need to do things in a certain order so if we take the pipeline we do that of course in practice looks more like this there's lots of parallel pipe lines that run on different instances typically one for each language then even despite that certain modules need to fire before others certain things need to happen in order for other things to happen so typically as I said part of speech tagging needs to happen before a named entity recognition and this of course becomes a challenge when you want to do things at $2,000 per second another challenge is of course training data train gate a labeling what I just talked about those those sets of examples with labels is costly because you've got to have humans human experts for each of these modules that look at the document and do the work that you want the algorithm to do manually and they need to do it many many times thousands of times tens of thousands of times and then multiply that with the number of languages that you need to support a lot of money contextual information is of course expensive if you want a lot of these algorithms work in a way that if you want them to perform better the more contextual information you have meaning the more of the entire body of text that goes into the algorithm the better they typically can perform but that's computationally expensive and of course this is not necessarily something that just hinders you at scale but noise missing data a variation in the data so for instance for language it's variation in language form and types of slang for instance and and data types you know if it's news documents or patents or blog posts or tweets it's it's all going to have very different variations this offers a challenge to so that's sort of the basic of NOP at least from the met water point of view I would say generalizes quite well to the rest of the industry but what about the topic of today what about entities and where do these come in why are they so central in in NOP technology today and then it'll be community today well in order to answer that we need to first understand what entities are named entities as they're technically referred to our referral entity so entities you can you can name and refer to in a way in any way usually its proper names so you know like Paris is an entity it's it's a location entity as it turns out IBM of course is an entity Gregor is an entity Alexei is an entity these are these are entities because they have proper names and it can be single world word or multi word this was an example there I know this man he might be Charles who lives in Stockholm in Swedish there's a bunch of entities in that example now named entity recognition is extracting these named named entities these entities from natural text natural language text and you're categorizing these entities as you do that into a set of predefined categories or types so these two steps extracting and categorizing entities is essentially named entity recognition so John lives in Stockholm he works at Ericsson would be as this example John is extracted as a person entity Stockholm is a location works at Erickson excellent is an organization now what's interesting about this is if you use your imagination and try to pretend that you don't know that John is the name of a person Stockholm is a location well that Erickson is an organization you would still if I replace this with XY and Z you would still be able to say X is probably a person Y is a location and Z is probably a company so this is this is where we're starting to get at the core of what named entity recognition is and the value it can offer so basically in a text you know all of the named entities need to be extracted for for the sort of purpose of what we're or we're trying to achieve here and I'll get to that in a minute so that that's typically how we would look it's also important to remember what named entity recognition is not all right it's not event recognition for instance that's that's something that people typically forget or confuse name deter recognition does recognize entities in the text and classifies them in some way but it doesn't create any temp template or perform any coreference or entity linking meaning you know it doesn't know necessarily anything more about this entity than that it's a location it's a person right it doesn't it doesn't link it to anything else that's what's usually called resolution or disambiguation and that's so something typically in in entity recognition systems comes as an add-on but named entity recognition itself does not offer that right it's not part of the definition of the problem and again as in the example of John who were lives in Stockholm works at Ericsson name natively recognition recognition is to say an ER is not just matching of text rings we with some predefined data database or list of entities right it's you know you as I said you do you don't need to know from a database or a list of names that John Stockholm and and Ericsson are entities it only recognizes entities which are being used in the language in the text as entities so it's a context that defines their their identity in other words it's not easy I need to emulate how humans recognize unknown entities so why do you want to do this well of course as you start thinking about it you realize this is very central for for structuring information right extracting the good parts and knowing what those are even though you don't know what they are right you know it's a person it says a lot you don't necessarily know who it is until you do some sort of resolution or this emigration so information extraction systems this is the key part robots handling of proper names is essential for many applications obviously I'm not water we have examples of that and then it's used as pre-processing for different classification levels we'll talk about that later today in a couple of examples information filtering information linking which is that disambiguation step that requires entities to be extracted and two things I will talk about today entity level sentiment which is actually a special case of this pre-processing for different classification and the knowledge graph those are two very good reasons for instance to do so an example here is a German text here you have the entities extracted and if you do that properly I don't know how many of you speak German here you can actually start seeing effects is seeing bits and pieces of information expressed in and as entities that are decisive for instance for companies Stark this is just one example of how the the financial industry today for instance uses entity recognition alright so what about social why is why social special-case or why why is why is it even a topic of one of the topics of my talk well social is obviously important because it's the largest body of user-generated content today alright so there's a lot of information in there especially if you want to listen to data outside the firewall now as it turns out doing anti directly to the social is not trivial alright so it's gonna get a bit technical from your on but feel free to interrupt me for questions if you think it will help but otherwise I'll try to keep it simple so how do you do in order to understand why social is a challenge we need to understand how you do entity recognition okay so typically anti recognition is done through what's called supervised learning again I mean training data labeled training in algorithm that learns to identify certain things in this case entities and then you test it the typical methods that academia and industry at least historically has been using for this are these three types hidden Markov models is is a way of creating probabilistic models based on what's called Markov models with some hidden variables that you you assumed dictate things conditional Markov models is a variant of hidden Markov models with what's called expectation maximization and then conditional random fields which is probably the most popular method of doing this CRFs use basically a flavor of hidden Markov models where you actually take the context so sequence of data points around the data point you're trying to classify so if it's a string you want to say is this a named entity or not conditional random fields are models where they actually look at the words around that string in the in the text so they they have a probabilistic model that takes the context meaning the surrounding words into account that's that's all you need to know for now about these models training data is is typically as I said it's ground truth data collected labeled data it's expensive the way you typically address this is you either have at least four social and this is what you will see in the state of the art in academia you either do some sort of automatic name- annotation using Wikipedia articles as training data because there you have entities annotated or as we've we've done at mat water you apply what's called the lvl Latin dershlit analysis for any R detection using gazetteers and I'll come in a second - what gazetteers are but it's essentially list of predefined entities so it's actually tables when I said you don't use tables of lists of of entity names I kind of lied there's such a thing called gazetteers that help these algorithms especially in a training phase if you like alright so I'm quickly going to go through the pipeline and here we're going to start getting into why why social is difficult but if you look at at tweet for instance but this this can be any corporate so it can be editorial text blogs you you typically have and this is what you'll find in any sort of textbook about named entity recognition you typically have a tokenization block that essentially breaks the sentence into you know the constituent parts that are semantically or in some other way relevant parts that belong to each other so it's not necessarily you know it's not unless I say that each word becomes a block several worlds can become one token unblock and stop words and other things are taking into account here in English it's easy you take the whitespace as sort of breaking most of the time after that you you have something called feature extraction which is essentially picking out the compressed pieces of data from your text that are relevant so the most important part of that is probably the part of speech POS tagging where you take out the grammar parts of of your of your text typically nouns and verbs and adjectives and adverbs and you actually also have a set of what's called clothes classes of determiners conjunctions pronouns prepositions that you also pick out these all going to what's called the feature vector that describes your your text and after that you're sorry after that you you have the CRF training that you've already trained with labeled data and your classification or the model that is used for classification takes all these features into account and then classifies those parts of the text that are entities as entity person location organization miscellaneous or so there it is there's a whole range of types of entities there's a few different standards that have been set by the research community but the four top ones there are the typical ones that you will see and the ones that we focus on at Mount water gazetteers as I said they they help they help a lot it's a again extensive list of specific names for a specific category so for persons its first first names female male and answer names with frequencies you can typically extract these for for languages for occasions its cities countries of course population organizations its name of companies from the yellow pages typically right so these these help but of course as you can see it's a costly thing right the difficult to create expensive to create and maintain especially commercially trust me their usefulness varies a lot depending on on the category there's ambiguity involved of course you know I have an example later of you know Washington it's that the person the location it can be a lot of things word occurs in yeah so work and occurring more lists right different of different types then there's something that helps here called Brown clustering I think historically sometimes even refer to as IBM clustering it was invented by IBM 30 or something years ago 20 years ago Brown clustering is essentially a way of predicting probabilities for things you haven't seen by grams typically but it can be you can even be trigrams so basically sequences of words and let's say so this example let's say we want to estimate and this is actually the problem that Brown who at the time was working on IBM was facing let's say we want to estimate the likelihood of and I think was working on the airline reservation system or something you want to estimate the likelihood of the bigram to Shanghai how likely is it that this this sequence occurs but let's say it doesn't occur in your training data all right so what you can do there is if you can cluster Shanghai with other city names like London Beijing and you for those city names have the likelihood of because you have seen those examples of to London to Beijing to Denver then their Bron clustering allows you to predict or or guess have a qualified estimate of what the likelihood of to Shanghai is without that ever occurring anywhere so it's kind of an estimation or extrapolation of a probability that you don't really have real data for real observations for so Bron clustering as I said was proposal Brown 92 writes that was that was a year at IBM it's a sometimes referred to as a hierarchical class-based labeling method it's bottom-up because it comes from the data you don't you don't impose that you don't teach it it's unsupervised therefore doesn't require label data it just requires lots of lots of text to you know see the probabilities and frequencies essentially of different words and by grams and trigrams it's what's called a greedy technique greedy maximization algorithm of the bigram mutual information and mutual information this is one of only two equations in all my deck so so don't get scared but mutual information is essentially the log probability switch what's called likelihood also of of the probability of seeing those two event it says events here but it can be words right of x and y and their individual probabilities it's called the prior probabilities and essentially merges words by contextual similarity so if they occur in the same context so let us take this example and I'll get in in a second to why how we use bond clustering so large amounts of data is usually what's required you have similar words appearing in similar contexts right that's the essence of predicting the probability for combination of words when you haven't seen that combination so similar words have similar distribution of words to their immediate left and right right so it it occurs in the same context similar context so one good example is there an R but they're both so called determinants in linguistics speak and you look at the frequency of immediate words to their left and right so here for instance in of there are dark part stores these you can you can combine these in any way so therefore these are each of these are clustered last on Tuesday Monday you know so so these kind of naturally then belong to the same cluster these words so yeah we can jump over this but this is essentially how you do the clustering so this sounds like easy then I mean so describe the components and you saw the pipeline's what's what's the deal what are the challenges in real life real applications and what happens when you apply this throughout the languages we've focused on English here and does it equally well apply to say Chinese or god forbid finish you will see in a second why finishes it's a God forbidden God God forsaken language I hope we have no finished people here and and then of course social what does that imply okay so I mentioned this earlier in the example of Washington so this emigration problem is of course something that affects entity recognition in reality right Washington as a string isn't always just directly identifiable it depends on how it's being used and that's that's sort of the challenge is real namely the recognition needs to identify for a result so what is the category of Washington is a personal organization location faculty vehicle geopolitical entity these are some of the the ones in the bottom there are more exotic types so of course in different languages then the challenge is as I mentioned with Chinese for instance words are not separated so tokenization is not really possible not in the traditional sense part of speech so nouns and verbs this is what's called inflection problems which is conjugations for for verbs basically problems that you have irregular regularities and impact and temporal tenses that don't exist in some languages and exist in abundance in other languages finish for instance has over 10,000 forms of them actually does you can you can take a verb and create 10,000 variations of it and finish approximately or a thousand a thousand variations in them you can do it for about ten ten different verbs and in finish and of course you have what's called the shape feature capitalization is it's not the capitalization you would think of here it's not capital letters it's it's it's a linguistic term but it's it's also it's a challenge here's a comic and all some of you may have seen this before I like this because it really describes the meltwater case really well we were sprung out of Scandinavia as I said so so Swedish amongst the other Scandinavian languages like Norwegian and Danish is a central language for us and English of course is the central language so in English guy the English I says you know a dog what's that Brett says the Swede the dog two dogs but few other forms of dog right the dog it's still dog right a dog the dog still the word dog it's not true in some other languages that you will see so the sweet Gozo conjugation okay I see we have in hand home them for Hyundai Hyundai now so I switch you actually the form changes if you use are the so if its infinitive form or some other form the Germans on Wednesday and suddenly and says wait I want to try to no bugger off Germany he says nobody invited you and in German as some of you may know it gets even uglier there whom the ein hoons vie whom there and then whom behind them einen hundun Dame who then so so you have variations and then and then the Finnish guy comes and says hey Sweden guys can I join in nein nein they get all really upset and then he starts listing the different inflections of her dog and in Finnish Sequoyah means dog but then there's career on Quora with to a career on again quite rasa called Rasta and then he goes on and actually each of these forms is a real one these are all inflection forms of dog in Finnish yeah and that's just the singular forms and in the end he goes oh and now for the plural forms yeah so languages can be tricky and study of the linguistic properties of each language cannot be taken for granted you can't just rely on the algorithmics to do it for you you know there's there's a lot of confidence today of course in deep learning and some other techniques to sort of be the holy grail of machine learning that the ultimate black box in which you can throw everything and it does magically solve any problem linguistics is probably the best example of why that can never be perfected in you know in in an unsupervised way without linguistic expertise so I've been promising that the social is is something where I talk about and social is is in fact tricky the more you were moved towards user-generated content typically for a lot of these algorithms not just name that the recognition performance drops and it does so for the reasons we all kind of know right it's unstructured language it's it's if we take tweets for instance and use some off-the-shelf say name that they recognition there could be anything really it can be sentiment analysis it can be categorizations you name it tweets are short of course we know that we know that there can you know in those hundred forty characters cover a whole universe of topics they're typically not written in any humanly recognizable grammar typically written by people who don't even know grammar meaning children kids and I've written fast and posted anywhere a lot of misspellings and people don't really care about them being misspelled because humans kind of can deal with that so if we focus on namely recognition now for from for a moment we would need a solution that considers social characteristics of texts some of which I've listed right now examples of noisy data yeah I mentioned these you know you can have misspellings OMF instead of 'omg and not not because I mean oMG itself isn't even a word right but you would need to be able to resolve that so yeah hashtags of course add mentions retweets multi-world abbreviations and to put it mildly non-standard spellings okay what's our solution well I'm not water we've we've developed an in-house solution to this that at the core really do relies on adapting existing features so we tried a lot of exotic approaches to this that are in the academic research core program but but ultimately we found that adapting existing features to social properties is the way to go and this is some work that we've published at a famous NOP conference about a year ago and we've we've mainly focused on the post tiger but also that the brown clustering that I mentioned so there we have a lot of of course in terms of tweets you have a lot of lot of just raw text data and as you remember Brown clustering is on super vines you don't really need labels you need to see what words typically occur without what other words so that that helps you especially with these sort of self-invented words that you have in social okay I promise you another equation this is the last one this is a CRF this is the mathematical format for a CRF yeah what you typically try to estimate is or learn is a model the consist of a set of weights these lambdas over features that you have so these can be part of speeches and you know tokens in the text and and features that come from the brown cluster so once you've identified a word you you kind of identify which cluster it belongs to in your block brown cluster the tree that I showed you these are sort of the features that you could gather from the text and what the CRF essentially does is its maximizing the conditional likelihood which is this so it's you know you could basically it says what's the likelihood of getting the output I'm getting given a set set of features all right so it's trying to determine what feature weights it needs to maximize what it's seen the features that we use are 35 features and it's basically these types we call them internal external features and ISIL as I said it's the tokens it's a post taggers it's brown clusters and it's features coming from the gazetteer matchings that you have from the text so you throw this into a big nice soup and and you get something like this right you get the weight importance of each CRF feature result of your training hi this is for all those 35 and as it turns out when we have tested this it outperforms state of the art said of the artist is by this guy at University of Washington I believe know if he's still there Ritter that is the orange one well it's actually the blue one there and the orange one but what were we significantly outperformed Ritter on this of course we had a lot of training data seventy six thousand tweets labeled by human editor and this is English only you can you can do the math of how much that typically would cause and we require what's called an inter an inter annotator agreement of two annotated so we need on the labels we don't we don't trust one label or we want to labelers to say the same thing independently about a label to say okay this is correctly so that's so guaranteeing a certain level that in the in the labeling and a test data is about nine thousand of those hour and additional nine thousand actually okay so alright now we have we have a way of doing name data recognition for editorial and kind of cleanly written text we have homebrew of how to do it for social text which is a bit more challenging and we believe we can do this at an acceptable level why do we want to do this we talked a little bit about at a high level information extraction systems and some of the other things I mentioned but but essentially it comes back to you know define from a met water point of view identifying entities in a business context all right Pepsi or Spooks coke with with Halloween theme at and you kind of want to know what's going on here without having to read all of that content out there for that you need to identify these entities of course and specifically after this ad there was a lot of sentiment in social media going on around Pepsi and Coke I don't know how many of you have seen this or heard the story behind it but but yeah it caused a lot of people weren't really sure who was spoofing who unless they saw this but but yeah there was a lot of sentiment in social media that both Pepsi Twitter airside Pepsi and Coke were interested in tapping into so for that you need what's called entity level sentiment which is the second part of my talk so what about sentiment and why is entities important here well if you look at sentiment how it's traditionally used you know you you have a document and you have a human looking at it and you ask them what's the sentiment in this document this is what the industry to prevail first year's document level sentiment and this kind of works and the human reason and says oh this is a positive text and unfortunately it applies equally like the method applies equally to to whether it's social texts or blog or news article but then you show the text to another human and that other human says oh no this is this is not positive this is just neutral likewise a third person might say no this is actually negative what do you mean it turns out we do this exercise and there's the studies that are shown this the typical inter annotator agreement is about 80% doesn't get higher than that so on average humans only agree when it comes to sentiment and this actually is a sort of red thread for a lot of the NLP stuff we're doing some of it is you know that it a Natur agreement is you know where's where is the punctuation in the sentence people typically agree on that very highly but some of the other things like sentiments not so much it's subjective right why is that well I'll get to that in a minute before I do so how is this used and well in machine learning and you know P what you do is you somehow gather human labels you know you have annotators who are agreeing you use only those documents for instance and of course you gather lots of them thousands and thousands and then you feed them into your magic box machine learning magic it's typically some sort of supervised learning right I've listed a couple of them for those of you interested naive Bayes so Bayesian methods support vector machines of course maximum entropy models multi-layer perceptrons so so deep learning neural networks but really it doesn't really matter what method once you've trained the model the expectation is okay I feed a new document into it and it tells me the sentiment woohoo yes we have a solution that's pretty into production okay we further production and it turns out the accuracy is somewhere around 60 and 70 N okay granted this is not too terrible given that the human annotator agreement is is at best 80 why is it so hard why why don't humans agree right what's going on here is it something fundamentally wrong with how we perceive sentiment so my water for instance we have the document sentiments and we gather them into one of those dashboard analytics but if you look at one of these documents might shed some light into what's going on let's take this this article here I think the article came out as negative all right so so it talks about these politicians it talks about Rudy Giuliani Scott Walker in a very negative way clearly when you read the text but then you you see that it talks usually about this guy Obama and you wonder you know so what's the sentiment of this article but clearly relies heavily depends heavily on what your perspective is are you a meltwater customer someone on the Obama campaign well then you would say this is a neutral document you don't really care about what it says but Rudy Giuliani Oh Scott Walker if you are the you know PR responsible of Rudy Giuliani as meltwater customer then you want no order to tell you this is negative documents how do we solve that how do we tell the document is negative and positive at the same time another example here McDonald's in this it's a Facebook post I think it's negatively mentioned these other companies Panera Bread chick-fil-a Chipotle Oh positive so when you look at a text just like I showed with the politician to Rudy Giuliani Obama text you will find different entities in there mentioned in different different with different sentiments and that's why when different people look at it they perceive it as different sentiments so when you feed the whole document into any algorithm and ask it to give you one answer it's not going to satisfy all three and on average is going to be wrong as I said the best algorithms today are about 60 70 % so this approach document level sentiment I argue doesn't make sense for our industry at all incomes entity level sit there for entities right so clearly there's something here that depends on the entities so we've established that document level sentiments in imprecise and wrong for customers in this industry entities our main importance for customers we already have named it is recognition technology here's a crazy idea why not identify the sentiment towards each particular entity in a text so you run in the D recognition you the entities and then you ask a human not to tell you the sentiment of the text but the sentiment of each of the entities in that text all right as an example BMW positive machine is neutral to it or negative and of course you do this lots of times again some sort of machine learning box and the new Rahman comes in you run in yard and what the machine learning box gets as an input is a document with named entities identified and you ask it what is the sentiment towards each of these because it's already been trained with that so I'll jump ahead here we've we've done in-house technology around this we've explored in benchmarking a bunch of when vendors of this technology out there it turns out it's really difficult I won't get into the nitty-gritty of why this is still difficult I mean you would you could argue that hey accuracy should be higher now that we have separate entities and stuff but but it turns out introducing this additional level of complexity of actually identify entities which itself can be well it's not 100% accurate right and adding that into the equation of them doing sentiment just isn't but my belief my hope is that if the industry if the research community focuses on this way of tackling the problem instead of the sort of the old way of just taking the sentiment of a document we will see progress a lot faster and a lot more than we have seen in the past 20 years with just traditional sentiment okay I know how I'm doing with time all right I'll speed through this the other reason from a meltwater point of view why we are so excited about entity recognition besides doing proper sentiment analysis is what I call the holy grail of really business intelligence at large the grass anomalies and here we're not just interested in the entity in the in the sentiment words entities we're interested in the complex network that these entities live in which is the real world right so these companies or organizations NGOs these people or journalists bloggers you know key employees that companies c-suite level people these products and services and brands that we have around us all of these have relationships to each other right a product is the product a company a person is employed somewhere he's friends with some other person a company is a subsidiary of another one another company is a partner of another one they all have relationships and and there's a connectivity in here that's rich and all of this is kind of exposed through the sometimes narrow sometimes wide window of media social media news media blogs tweet all of these talked about in one way or another in the order of billions of documents per month these relationships they talk about something happening someone acquiring someone a company releasing a product so it's all out there and if we have the building blocks that the tools of extracting these pieces of information the entities being the core of this network then why are not attempting to build this ourselves so really what we do have my water is that we introduce in this architecture thing I showed you earlier we introduce this this component here that we call the graph where those 2000 documents that are swishing through our pipeline could we indicate and talk to this graph tell it stuff and ask it stuff why is this hard I mean okay that sounds like okay that's it we've solved it no it's it's hard and and it's hard for these three reasons data scalability and the new features that this requires relationship extraction just being one of those so data acquisition is is difficult because you basically have this trade-off that you need to to manage right in this pyramid here on the top you have low quality at the bottom you have high quality so low quality is typically you know special crawlers smart algorithms that you can use to harvest the data from websites and whatnot but you don't get the richness of a quality that for instance a human could give you if they list all the properties of IBM number of employees you know what what kind of like TechCrunch does in the crowd crowd-sourced way they they have a lot of information about each company or a LinkedIn it's high quality but what what it also is is it's expensive unless you come up with some smart way of doing crowd crowd source like like TechCrunch and and LinkedIn does so in this and you have expensive and cheap expensive of course acquisition and partnerships low volume high volume manual acquisition is is a way of doing it you get the quality but it's expensive and it's typically low volume so you can't have all three so whatever you do you get a point somewhere on this on this in this triangle that's one of the challenges the other challenge is of course scalability you know we're talking about in the world in the order of typically hundred million companies five hundred million people if we if we only focus on the key individuals in these companies and not all the social influences and what all what else that's out there it actually so that shows that excluding media sorry for that typo and products let's say in the order 500 million this is a lot of entities that means we have about a billion entities with all the connections between them theoretically speaking some buildings of nodes and trillions of edges on an average order this is a scalability problem that you know is in the order of magnitude of all Facebook faces and third challenges of course new features and I will I'll jump through this really because it's not that interesting and actually jump all the way to the end okay thank you yeah let's open for some questions Andrew right so the question is how do you define what relationships that are interesting I think it's it's something that needs to be very business centric for us I'm out water it's typically the relationships that our customers are interested in so I mentioned a couple of examples of that it's companies having partnership subsidiary competitor relationships to each other its products being competing products with each other that's a relationship between product and product and then of course products being products of a company if we include influencers as we're looking at doing it then for instance an influencer can have the relationship to a product being he's an influencer he's an important voice when it comes to say iPhone 6 right that's a relationship and then of course between people there's those relationships and network relationships but it really needs to be dictated by the business requirements that you have with your graph with your knowledge graph yes yep yeah a good question so the question is a word to Veck that that recently emerged the technology a pro CH that was I think pioneered by Google if we use that amount water we do we've we've more or less successfully used it for categorization because the the the unsupervised strength that were to back that basically Maps words to a vector space where you can actually do math with the word words and you can actually say you know what is what is if what is Paris so what's Paris - what does Paris correspond to if longer corresponds to UK and then you expect it to say France because of the vector relationships this kind of you know arithmetic swith words allows you to forces document categorization much more accurate that's what we use it for yes much smaller space because because you can suddenly start cluster things and and you have only the clusters to take to care for rather than individual parts yeah of course a foot for the in your stuff and no not yet no yes so I think the question was how I relate word to vac method to was the conditional random fields really it's it's in the way those classes are so it's they're essentially achieving the same thing word Tyvek offers a much simpler way of achieving that like the the matrix algebra of calculating the word Tyvek relationships it's much easier than solving a conditional random field equation because that would require label data where Tyvek approach doesn't require both the label data right so so it's the simplicity that it offers so so essentially a lot of that is in the research paper that we published it was essentially to add additional features so so you know I showed you that some of these abbreviations and things that occur in social text they need to be taken into account and if you just run a standard off-the-shelf say tokenizer or basic limitation a lot of these things might be thrown away for instance but if you have features that really take the social characteristics of social content into account you can use pretty much the same models and then the other thing that is special is the brown clusters that need to be trained yes basically yeah mr. Simmons we have not we haven't yet I should say that's something I'm I'm personally working on it requires buy-in from from my boss basically and then the stakeholders in the company but we were carefully deciding which parts of our code base that can be open source but we intend to okay so namely to the extraction or recognition is essentially about non dictionary based - the gazetteers non dictionary based recognition of of entities right being able to say this is a company without having a database or knowledge base that tells you this and this emigration can be done really in two ways one is to resolve it against a knowledge base right the other one is if your body of text is not just a small tweet or a small document but large documents or maybe a bunch of documents in which you know I'm actually sped through speeded through that example so for here for instance an IBM big blue I mentioned there clearly referring to the same entity a human can identify that maybe not so much without knowing about IBM and and big blue sort of terminal or the term big blue to refer to either but but still some context you can actually realize oh this is probably the same company it's talking about without knowing what that company is so that's unsupervised assembly Gration where you basically cluster within a large body of text references to the same entity but typically in the industry this emigration is done by reference to some sort of external knowledge base Wikipedia is a good one crunchbase you know LinkedIn these are of course not commercially available except Wikipedia but but yeah that's the essential way you first recognize in this emigrate against database other questions web crawling we do not I mean we have small projects in-house where we do web crawling for other purposes but if you're asking for the purpose of the knowledgebase we do not if you mean for the purpose of gathering news articles and whatnot yes absolutely we have our proprietary crawling technology for news news websites we have about 150,000 new sources Google News in comparison has 40 50,000 new sources okay no other questions and thank you