Text By the Bay 2015: Nemanja Spasojevic & Adithya Rao, Large Scale Topics on Social Networks
my name is nemanja and today we are going to be talking about large-scale topical assignment on multiple social networks this work has been done by cloud data science team and cloud got acquired by lithium so now i guess part of the lithium data science team so first we'll introduce what actually cloud is for those that are not familiar with it then we'll go up over like problem statement and some of the challenges when dealing with the large amounts of short form data from social networks then we look at into the some of the system details and the future engineering and finally we'll just look at the how do we build models and look at some of the results so what what cloud actually is cloud is a social influence measuring tool so usually user comes to the cloud.com register his social networks and then on user's behalf cloud goes and collects the data from multiple social networks in addition to this we get like a full mention stream from uh gnip so basically we get like a twitter stream as well for unregistered users and then using this data we extract various features and derive cloud score as well as different topical scores so what are we dealing with in in this set up basically one of the main challenges is being able to process like huge amounts of data on a daily uh fashion with very limited uh cluster capacity right so every day we get about 650 million new pieces of data then we are assigning topics for hundreds of the millions of users across different social networks and then our topical ontology is like uh pretty big it has about 10 000 different topics in it and all of this needs to be updated in a timely fashion so we basically recalculate the scores every day another uh set of challenges comes from making sure like we can integrate different sources so basically making sure like you can collect the data from different social networks and then even within the social network there are different types of data you can collect from user graphs to user profiles and different interactions and then at the end of the day you want to create topics which are publicly recognizable so basically if you put on the public profile set of topics that's next to somebody's you know picture like those topics should be recognized by his peers and not be only like uh you know interest of user itself so some of the applications of these topical assignments throughout the product are one that we mentioned before it's basically if you come to cloud.com you can see like the topics that we think you're interested in and then maybe you kind of um are expert in then some of the past and the current product usage is where basically if you're asking a question you wanted to have ability to route given topical question to a top influencer that is interested in a given topic and then also like be able to suggest like who to follow based on topical interest but also based on kind of the person you're about to follow interest and then on the business side we also use like topical assignment to say uh give free perks to top influencers which are interested in certain topical domains and then most recently we added like a content discovery feature to the website where basically you can go and browse through uh look at the content that is relevant to your topic area of interests and also maybe like if you're trying to optimize your influence outside the cloud dot com basically want to be able to look at the quantity that is relevant on aggregate level to your audience and then some of the main challenges as we mentioned before when dealing with the social media text is usually you're dealing with big amounts of data basically 650 million new pieces of data daily and then on top of that majority of texts are very short form and they are grammatically incorrect and then in social networks a lot of users actually are just very passive consumers so what they do they go and browse like the facebook or twitter but not too many of them actually produce the con uh content right so that's another challenge and then uh in terms of scaling actually a lot of times you kind of do want to use like uh sophisticated nlp techniques and kind of do like language understanding and stuff but in our case basically if you would to use let's say a stanford nlp named entity extraction and just process like a 650 million pieces of the content it would take about like 82 cpu days so basically for uh for the nlp approach that he was like on operating on raw text we actually went with the dictionary based approach where you spend a lot of time designing your dictionaries which are mapping the phrases to entities and then those entities were combination of the freebase wikipedia cloud.com like dbpedia and the wiki concepts so with a pure dictionary based approach you could speed up things about like a 150 times so instead of spending 82 cpu days on processing all the data you spend like less than half a day and then another benefit of operating on the dictionary uh usually like your phrase is mapping to a certain entity and then entity is a record in a knowledge base so basically in the case of uh shown here stanford the named entity extraction was able to recognize the rule of law but dictionary based approach was able to recognize the role of fall of law nsa and eff and on top of that you have the mapping to uh to the entity itself in a freebase which you can use later to say expand the topical context of the sentence using that entity or use it later for this invitation and also like a lot of time language models are available only in a limited set of languages while knowledge knowledge based basis are much more well a better scale in terms of international languages so so one can ask also like why do we even uh care about multiple social networks maybe like twitter has the biggest coverage and has like a his most dominant network maybe use only twitter well it turns out like if you use more data i mean it's intuitive basically you build like a better system so if you look at even how verbose users are on different social networks uh on twitter and facebook people do kind of share a lot of information and produce a lot of content but on the networks like linkedin and google plus there is much less content however the value of information is uh sometimes this disproportional so for example on a facebook maybe you talk a lot but a lot of conversations are not very topical in the nature they're very casual talk about birthday or a party or something like that well on linkedin maybe even if you have 100 bytes of text you get the very high value information right from for example looking at the linkedin skills the other benefit of using a multiple social network so if you look at the right side here what is the overlap in the keywords across for a same user across different social networks so even for the most active users this overlap is at most like seven percent so what me what this means is basically usual users usually tend to express difficult different topical interests on different social networks on linkedin maybe you are expressing more interest regarding your profession in life and then on the facebook you're more talking about your personal life and then uh how the system looked like so as we i mentioned before basically we collect data from twitter facebook google plus instagram we also incorporate like a wikipedia mappings for the users that have wikipedia pages and instagram as well and then we collect user activities user profiles and user graph information so those uh all that connected data is extracted to the text features after that we project text feature to the topical feature and then we do like um normalization of topical features very basically for a single user uh for uh for a single user like we map all the future values from one to one uh from one to zero and then we train the data on the ground truth and every day like we basically uh calculate the scores using the trained model and after that basically like uh topical assignment gets different applications throughout the product so system details like we are using pretty much what everybody is using around the valley like we use like a hadoop we store our data on hdfs and then we use like hive and for serving age base and hive show to be like pretty useful because it gives you a couple of things one is like it gives you like a data catalog so you can kind of put your features every day in the new data and partition put them by the feature name and basically neatly organize them and then on the other side like you don't have to write mapreduces so one hql query basically just worries about the data transformation so instead of writing like three map reduces with maybe two secondary sorts you're writing writing one hql query and it just does the thing uh things you want and on the other side you're forced uh to encapsulate like a complex uh complex like uh to say like modules like let's say like a language detector or name entity extraction into the udfs so by design you kind of have to design these modular pieces which are going to be later reused throughout the different pipelines and then we do we train the machine learning models occasionally like once a month we update them and then build the scores once a day and then on the on the side of the resource how much resource this is taking so uh the total pipeline takes about like 66 cpu days and then the reads and writes are on order of terabytes okay so a little bit about the feature engineering so for every user we basically store basically username and then like we have like a feature name and then for a given new uh source and the user we have basically bag of topics like a vector of uh bag of weighted topics and then particularly important thing was making sure like we can use as much context from each piece of the data so basically uh like as we said before a lot of users are basically casual consumers and then not too many generate so that's why we kind of introduced like this attribution uh into the future naming where for example if you generate if your original author of the post like we say okay this feature is generated uh this feature you gen it was generated by you so we encode like a generated into the future name if you let's say reacted about like original on the original post then topic of original post is assigned to you in a reacted way and then if you for example haven't done anything to be assigned with the piece of content let's say you got your obama and you got listed on the twitter list basically we say like this feature was uh credited to you so in similar like uh with the mansion right if you if somebody mentions your name you actually didn't do anything to kind of get mentioned but still that piece of data is uh attributed to you and finally we had like a graph attribution as well so uh in the future naming like uh the encoding views basically we would capture like the network the data source it came from optionally the time video window that the feature is applicable for and then finally attribution and the idea was like that it's very easy to add or remove new features because oftentimes if you spend too much time investing in a single feature you kind of run out of the time and then you don't get time to experiment much so in this model we would basically constantly add or remove new features and then the ground truth so usually like one of the main problems is like how how even to get the ground truth in this case basically it's not as easy because like you need to like for a for a given uh social network user which may be like a high influencer or not high influencer you need to able to label some topics so we build like a small web tool where basically all the employees within the cloud could go and then we would display them their uh first degree uh for uh first degree like audience so basically twitter people they are following on twitter or maybe like their facebook friends and then we would give some uh first guest topics uh for those friends and then people uh the evaluator evaluators will just go and mark topics as kind of positive negative examples or skip them if and they're not sure about the interest and then uh the data set consisted about 32 000 data points and a total of about 750 users was evaluated so the finally like uh we transform the data to the data points where you encode like a topic user and then for a given user you have like a feature vector of different features mapping to different values and it turns out we calculate the precision recall and coverage for different features and it turns out like some features are really good they have very high coverage like for example like uh twitter graph graph based features but then on the precision side they are not as good like maybe like about 30 percent and then on the other side you have a features which are very good uh basically like a twitter list if somebody lists you on a given topic like it's actually it's a very good signal but they have a low coverage and then on the other side like uh twitter turned out to be like a much better signal than the facebook which does make sense because on twitter people kind of tend to broadcast their interest and uh information is much more topical so the final uh precision of the system was about 60 and then the recall was about 80 like which seems like pretty low but even if you look at the evaluated data set like uh if you ask a couple of different evaluators to evaluate same person interest the agreement is all only about like a 70 so the pro there are like a problem limitations just because like it's very objective thing asking if hey is this person interested in this topic and uh f1 was i think about 0.65 cool so some of the examples of like topics assigned to the users so these are kind of like a power users but everybody knows about them like marisa mayer yahoo google technology than lady gaga music lady gaga celebrities obama politics affordable care act and health care and you can see like there are some like outliers here and there but in general topics look pretty decent so at the end like by combining uh different social networks and different data we were able to build like a pretty comprehensive system so we are able to assign uh topics to the more users and you would be able to assign to any individual network and then for a given user we were able to assign more topics uh just because like we combined more data sources for him and then just like for fun we can look at different distributions of the super topics across different networks and see that for example on twitter the most dominant topics or technology entertainment on facebook it's entertainment followed by technology lifestyle food and drinks on linkedin on the other side it's business and technology and on google plus again like it's technology and wikipedia is dominantly about the celebrities so key takeaways like uh oftentimes uh um people get stuck like uh hedging a lot of bets like on a single feature but uh basically if you wanna build like a comprehensive system you need to kind of build it so you can include as many features as possible so in this research we had about like uh finally like a 50 uh features and throughout the course of the project we experimented with 100 different ones so so basically and then like this is just like a question of topical interest and the harder question is about topical expertise so being able to assign something like a cloud score but within the topical domain where you're building like a global rank of people within the topical domains and hopefully we get a chance to talk about this some other time and uh if you like uh this presentation you can check out the original paper it was presented at the kdd 2014 so any questions uh topic uh yeah ontology taxonomy uh so there is only like a three layers yeah like there is super topic sub uh subtopic and then uh entity level what you call entity level yeah and and it's not the ideal it's mainly like because of the product decisions that have been in the past no like we have the we have in-house library specialist which kind of curates and keeps it up to date yeah yeah because it coordinates with the product and the business team yeah exactly yeah and the text themselves the lexicon is mapping to the freebase concepts so the ground truth so basically like what we did like we built like a very small in-house tool so you would be able as employee because we know like what are your like first degree connections like you would be able there to go and see your friends so basically very quickly you would be be able to switch across your friends and then for each friend we would show something like this basically we saw we show a list of topics that are that user is potentially interested in so you can go like and very click quickly just click uh you know click on the side and mark it as positive and negative example and basically using like the tool uh an evaluator could go and maybe like in two minutes mark about like uh 50 topics for a given user and then just repeat it for different users so very quickly you can kind of go through it but it was very important that actually it was a first degree connection so a person knows the other person and that's why like mechanical turk for example wouldn't work as well yeah so basically that's a very good point like because the approach taken here is two big degree like a brute force right you're taking and you're doing like entity extraction which is based on the dictionary and you're not taking too much language understanding into the context but the idea is like if you include a lot of data sources like uh hopefully you know like the data sources reinforce itself so if you set somewhere once like water polo like maybe it's going to float on the bottom and basically that's why at the end like you get a good system and actually the precision on the top 20 uh you know topics assigned if you saw them by the score uh by by the linear regression score uh logistic regression score it was about 80 percent so one in a five is kind of so in in this uh how many ground true data or how many raw data well we try to use as much as possible yeah so that's the last question yeah we can continue conversation thanks you