Text By the Bay 2015: Vita Markman, Topic-Based Sentiment Analysis in Customer Feedback
Recording: Text By the Bay 2015: Vita Markman, Topic-Based Sentiment Analysis in Customer Feedback
good afternoon thank you for coming to the talk uh my name is Vita Markman I am a staff software engineer at LinkedIn and this is a joint work with a manager of business analytics uh Yan Jang Jang uh topic based sentiment analysis mining member feedback and social media for actionable insights so what is this uh talk about well um it is about mining raw text for key topic phrases obtaining sentiment about these phrases and then uh finding more context to explain the sentiment uh just so that uh from the beginning I will explain topic based sentiment is also sometimes known as aspect-based sentiment and it's sentiment attributed to a specific Topic in a document or an aspect of a product uh versus the sentiment of a document as a whole or a product as a whole and more frequently you would see the context of uh let's say phone reviews that would separately review battery life versus camera Etc so uh let me situate this a little bit more in some broader context uh so uh for example in uh any company in this case at LinkedIn we have lots of uh Text data about our products it comes in various forms in this talk I will address primarily social media data that mentions say LinkedIn in some broader category like a JS or LinkedIn and some broader Cate category like inbox but you can also think of this a raw text Data coming from uh customer support uh tickets or member feedback of any kind uh related to either a specific product like who viewed my profile or just some broader uh people visiting the homepage Etc so this is a lot of raw text in this talk as I said I will focus primarily on the uh text that comes from social media uh post posts and why would one want to uh you know mine this text well there is two reasons for it that are uh that I will sort of uh focus on here one reason is really sentiment related so uh business analytics and uh product owners various stakeholders in the business want to know uh what people are saying about them when they contact customer support or uh what people outside are saying that when a new product is launched it's very it's a very frequent thing that people sort of are interested in finding out and so usually you would get like this stream of raw data and it mentions some broad loose category like um jobs and Linkedin and then what what what are people saying about jobs with respect to LinkedIn it's a very broad question to ask uh so what we want is a sentiment related rationale for it is to obtain the more actionable information that requires a greater Nuance so what specifically do people like versus dislike when they mention your company and the particular product or broader area H there's there is a second reason that is more content related which is when you look at a a raw text on some broad topic a and some raw text on some broad topic B you would notice that they may uh have different semantic Landscapes so there there is micro topics that are mentioned in one that are not mentioned in the other so for example if you have a collection of text about LinkedIn that uh mention jobs in some form and another collection that mentioned LinkedIn in inbox you would notice that the types of things people talk about in one is very different from the other and it's useful to find out what are these topic Landscapes so if you present it to a product owner and say Well when they mention inbox they talk about inail policy or they talk about uh member communication or Talent support Etc you you kind of get a some sense of what specifically is mentioned um so with that I will turn to the challenge that I will be addressing in this talk so first is discovering informative topics in a noisy text um second is finding the sentiment about these topics and third is explaining topic sentiment by some word attributes that give you more context the data that I will talk about today comes from publicly available social media posts that mention LinkedIn and that are grouped into these very loose macro categories via keyword searches like jobs or in box for example so what that means is we uh you know have this data that talk about LinkedIn and they mention jobs and it could by the way be Steve Jobs because this is all keyword-based and so you just are given this uh dump of text that you would uh can look at right here as some actual examples the next big thing you've missed linkedin's quests to get a job for everyone on Earth uh research shows that 75% of jobs are found through networking hashtag compareed to 10% through advertisements blah blah blah understanding networking and using hashtag LinkedIn as a tool to help you find a job Etc so these are like the kinds of raw data we start with and so what we want is we want to get some uh more Nuance sense of what this is about when uh people Express uh these uh when people mention jobs in LinkedIn so I will begin with what is a topic in our sense so a topic I'm going to give a working definition is an nram of size two or greater um in this case it's actually two or three because of short uh because of the shortness of the posts of social media uh that uh respects the following two conditions it is a syntactically well formed phrase so it's a noun phrase or a verb phrase and it is also semantically informative with respect to a given Corpus so something like Talent Solutions versus Solutions of so Solutions of is a fragment um inmail policy versus good day so this is an important example here um if uh I tell you uh I read this collection of um posts and uh they're about Good day uh okay what does that mean somehow you didn't receive any information about it but if I tell you there about inmail policy some new inmail policy that we have implemented you got a little bit more information than you had before that is because good day is even though it's a perfectly well-formed uh phrase it's actually not really informative semantically with Ure the Corpus that you gave me um because maybe because it's so frequent or something like that uh invitation to connect versus connect with again this is a fragment the other one is not so um there is multiple ways in the literature of how to get topics uh one possible method is you can uh part of speech tag your Corpus and then Define uh very well-known linguistic units like noun phrase verb phrase adjectival phrase and just look for these patterns of part of speech tags and it works fairly well when your Corpus is uh sufficiently well formed grammatically has normal punctuation doesn't have like millions of hashtags and fragments Etc but it doesn't work so well on noisy text in addition it will pick up some semantically uninformative phrases like very good and greatly appreciated that are actually perfectly um perfect phrases because a part of speech tagger doesn't know that greatly appreciated is not something you're interested in it just knows it's some sort of adverb and some sort of participial um and so it would it would never distinguish between talented employees and good day because both are adjective noun patterns the other method which is what I will be talking about today is that to make absolutely no reference to part of speech tags which is suitable for noisy text um eliminate semantically uninformative phrases like good day and eliminate fragments like COO of and this method is actually fairly simple um and I will illustrate this in this one um box diagram because everyone has to have one as a one of the previous talks mentioned so uh it requires you to start with two things first you take your Corpus and you separate it into literally the engrams that as they come so the book on LinkedIn in jobs jobs on LinkedIn recommendations good morning is looking career opportunities Etc and so you you put these s into some set and you keep it there and then uh you do something um sort of simple but interesting you take a completely different Corpus from someplace else in our case it's a corpus of uh like publicly available um articles that people write on LinkedIn whatever it could be anything as long as it's not that Corpus that you were looking for to understand the topics of and then you separate them into tokens and you take the top Upper Crust uh 1,000 words from that other Corpus usually and not not usually necessarily it will involve the common stop wordss the of is and blah blah blah blah blah but it will also involve a lot of words that are not stop words that are just frequent adjectives like good and great and uh take give have all these verbs and uh that that are kind of U not necessarily very informative for the purposes of understanding what the topic is and then you simply do the following you look in uh through the engrams and you say because my engrams are so small they're just two or three words long if the engram begins or ends with one of these words is I take it out it's not interesting to me because it this eliminates immediately all the fragments because a fragment is something that would end in an of or the it would also eliminate uninformative phrases like take care or the book uh because even though it's not a fragment but it is uh it's it's two words long and it one of those words contains a really frequent uh word that you see everywhere else so you kind of don't care to uh it doesn't really give you enough information and it would leave things like career opportunities or jobs on LinkedIn because on even though it's in that frequent list it's actually um embedded nicely inside of a phrase and so um and so you do that and you can use some stopping Criterion so you Pi for example you want to say I only care about these phrasal and RS which I will refer to as topics uh if they appear end times or greater in some cases you actually don't care about the count and sometimes you actually want to get something that's really rare but it's really a well-formed phrase because it might be interesting to you maybe because your customers are mentioning something even three or four times but it's something really important and you want to look at it it depends on the on your goal for social media because there is so much data you probably would want to have a stopping Criterion at some point so this is a very simple algorithm that I just went through so you define an external word list then you separate your Corpus into engrams and then you step through this engram set and if the first word and the last word are not in a phrasal are not in the external word list you add it into the topics and you would get things like Business Development hiring manager profile and Linkedin and you would eliminate things like development and to hiring and good morning so you have uh obtained these topics and now you want to uh understand when people mention things like Professional Network or Business Development are they happy or sad about it is this a good thing or a bad thing so uh we propose the following algorithm for obtaining topic sentiment that is also very simple uh we find for each topic all of the Clauses that contain that topic if because the social media posts are very short usually one post is one Clause but that doesn't necessarily have to be that way so we'd separate it and get actual Clauses uh from the posts and then we will rate each clause for sentiment using a trained svm model that was trained on social media data on label data and then we are going to um label these new examples so for instance this is an actual example LinkedIn is the world's top Professional Network um so it mentions Professional Network and it has a rating of positive from the svm or I use LinkedIn a lot for business development also has a positive rating by SVN and then we aggregate all of the rated Clauses to get a majority vote on each topic so if if let's say business development appears in a thousand Clauses and it has you know this distribution of sentiment then uh given the Clause ratings then that's what the distribution of the topic is so it's a majority vote kind of thing which is nice because there is quite a bit of error as you see the recall and precision numbers but we don't really care as long as most of them are right so here I give specific distribution but I might as well give it a binary so it's positive it's if above 0.5 or negative otherwise or neutral otherwise so so um that's uh that's the general approach that we take to obtaining the topic sentiment uh but once we have we have obtained the topic sentiment we want to dig deeper and we want to understand uh why the sentiment is the way it is so sometimes it's quite obvious if someone says uh you know amazing social network or something like this then you can kind of see in the topic itself the words tell you why it's positive but often times it's not the case so for instance if you take a topic like um LinkedIn recruiters use LinkedIn it's not obvious the positive uh sentiment of it is it's not immediately apparent so you want to get some more context every time they mention recruiters use LinkedIn what else are they saying next to it and so for each topic we will then find all the words that appear in the same Clauses as that topic t uh and then maybe use some tfidf to um retain the attributes the word attributes that are informative of a given topic but a rare across topic so attributes are basically words that appear in neighboring context of the topic and help explain why the topic is a positive negative or neutral uh they can also be used uh to find semantically similar uh more like synonymous topics and combine them together but today in uh for the interest of time I actually can't go into that uh but you can imagine if someone says view profiles and browse profiles it's basically the same thing but View and browse are not obvious synonyms that you can look up in a dictionary because you don't say uh view magazines and browse movies they're quite context dependent but if these uh two topics appear with very similar attributes uh of word attributes next to them then you can say Well they're basically the same uh through context distribution so some examples of uh recruit like attributes for a given topic uh recruiters use LinkedIn modern hiring identify candidate Source potential interview improve executive resume hired qualified and recruit so these are what a sentiment literature likes to call like positive happy words or LinkedIn lead generation yeah so by positive attitude uh towards recruiters use you mean among recruiters uh among recruiters who like to use l this is not the people who get the we don't really know what what it is yeah but maybe but apparently the social media has spoken and appears that everyone who has posted uh has has said good things about it so LinkedIn lead generation is engaging create content secret branding and potential so um it's hard to distinguish positive from neutral but you can imagine well in some other context positive and negative would have like really different distributions um and so that's uh that gives you a sense of why a particular sentiment is associated with a particular topic and then moving on to the content related reason we want to say that if if you look at this macro category of jobs in LinkedIn Corpus versus U inbox and Linkedin Corpus from which we started they have really different Landscapes of topics which in some sense actually might be somewhat surprising or might not be surprising depending on how you look at things so topics for job category involve unsurprisingly things like hiring manager interview questions LinkedIn recommendations Talent Solutions Talent acquisition and career opportunities whereas for the inbox category it involves networking tool lead generation uh social network LinkedIn invitation template inmail credits inail on LinkedIn sending multiple emails uh friend requests uh Etc uh what is more interesting is that there are some topics that are not even uh present at all in one but are present in the other so uh while clearly jobs category mentions things like resume career hiring and talent while inbox mentions invitation in mail and leads frequent topics in the job collection like tance Solutions and career opportunities are entirely absent in that other collection and so it gives you a sense of uh separating these two uh types of things so it gives you a um kind of a semantic landscape that allows you to Summarize each of these categories by the overall um type of topic distribution that they have so um in some The Proposal uh for topic based sentiment approach that we have done and implemented it works very well on language of social media as well as in the more well structured language of a customer support interaction with uh members uh because it removes uninformative and fragmented phrases like good morning or CEO of while Leever by leveraging an external uh Corpus of text and so this one thing this external Corpus of text allows you to get rid of both the fragments the syntactic fragments and the semantically uninformative but perfectly well-formed phrases which is good um it then it allows you to obtain the more actionable insights by looking at a nuanced topic oriented sentiment as opposed to trying to say when uh you got this Firestorm of uh tweets or or if you have this Firestorm of like social media data like going at you what is this about well I don't know it's about many things and so instead of trying to identify that you want to separate it into a more topic um oriented approach that gives you a semantic lens Cape of a collection of text a versus a collection of text B and obtain sentiment related to each uh of these topics uh which then you can get more information on by attributes that surround uh each Topic in the context uh in some this can be leveraged to summarize a variety of corpor regardless of the noisiness of the raw data it doesn't have to be social media it can be pretty much anything you want thank you have questions yes so you mentioned you take up the you fragments and also the sort of un informative Parts using this external Corpus and you look at a top thousand so how was that number choosing basically you so yes so this is as as as anything uh it's kind of a trial and error it is a heuristic so the more the deeper you go the more you run the danger of actually taking out something that does matter and then depends on how resilient you are to loss of these important or sort of important things the ones that think are good candidates you have another stage where you actually V them somehow they're actually good or not so uh at this point we are basically just looking uh through them and looking at the counts and so are saying that the count will decide um in it ideally you could imagine looking at things like well does it contain frequent words that are frequent for this Corpus so if something contains LinkedIn a lot then you kind of don't maybe don't care as much um there is uh other way to combine him using some language modeling technique and to know that a networking tool and Linkedin networking are maybe related into this broader more informative topic the longer the topic is the more informative it would beks so when you made those attributes you labeled them as like positive or negative or neutral um how did you judge neutrality is there like is it just something that's not clearly posi or is so let me correct this sorry uh this maybe I was unclear so we uh rate the topics as negative or neutral by the sentences that the the topic is neutral or negative or positive given the majority of sentences that mention this topic the attributes are just uh they are not rated for sentiment at all they are just words that appear next to it and we hope and sometimes our hope is Justified that they will shed light light on uh why the particular topic is positive so if let's like recruiters use LinkedIn and it's like hiring and um jobs and career and growth then you get the sense of why we do not rate the attributes which would be an interesting question in of itself judge the neutrality of topics based on based on the svm model and and if it's neither very positive nor very negative you just say it's neutral yes it's it's extremely so Neal is is I I hate this category myself I would always rather do a positive or A negative because to me if you talk about something I do and it's not negative it's positive uh but other you know the literature sort of tends to take a more three-way approach do you make any attempt to group the topics after you found them like say you have career opportunities and employment opportunities yes I do have so so this is as I said uh we do use as specifically the attri distribu tion of the topic in contexts uh of the attributes which provide the context by saying that uh employment opportunities that's an excellent example in career opportunities would be similar uh but maybe some other word with opportunities like uh life opportunities and career opportunities are not because of the distribution so yes similarity other words yeah yes that's the idea the distributional similarity of one and two yeah use the second cor to confirm your engram as top what is that so it in this in our case it it's a collection of uh articles of blogs that people publish on LinkedIn do you also do for that no just words just literally so I only use it it could be anything it could be the New York Times in fact the farther it is the better because in my case there is some overlap with LinkedIn because it's still from LinkedIn so you I we literally use it as to obtain a bag of frequent words top frequent words is all I care about if if God gave them to me I would take from there but God isn't very good to me and doesn't give me word bags you also no just words just single words engrams you could do that too which is uh but the you know I like simple uh the simpler the better the less work you can do if you can get away with it the yeah uh yes so um I guess do do you guys look at how things change through time like I suppose all those associations change Through Time sure so so this is fairly new uh but this is part of the idea is is to uh to use the simplest possible model to get the topics so you could look at them through time whenever you launch something or whenever you take something away you want to understand um what's happened so yeah thank you