ST Text: Nemanja Spasojevic and Adithya Rao, Large Scale Topic Assignment
Recording: ST Text: Nemanja Spasojevic and Adithya Rao, Large Scale Topic Assignment
thanks hi everybody first of all welcome to lithium maybe you don't know what lithium is doing but uh basically lithium is a solution for the brands to manage their communities across like the uh forums like which are on their kind of sites and then across like the social media like Facebook Twitter and the others so uh my name is Nea and today with together with Adia we're going to present on large scale uh topic assignment systems on the multiple social networks and this work has been done by the cloud data science team which is part of lithium now so first we are going to dive in a bit like uh and look what actually cloud is about then we look at uh the problem statement and some of the challenges when you're processing data on the uh from the social media and then uh we'll basically go over the system overview look at some of the feature Engineers look at how we build the models and uh finally we'll look at some of the results so what what actually cloud is so cloud is the tool for measuring of the social influence uh like user comes to Cloud usually registers then connects to multiple uh social media accounts and then on users behalf like Cloud goes and collects data from Twitter Facebook LinkedIn and uh other sources uh and we also collect the public data from the Twitter so we have entire like a Twitter mention stream uh once the data is collected uh what we do is uh we process the data and then derive like a cloud score and derive like a various topical sets from the collected data so what are we trying actually to solve here so we are trying uh to solve for a couple of things first first uh first one is basically we're trying to solve for the scale of the data right like every day throughout the Twitter H and other sources we get about 650 million uh pieces of data like basically every day like a new piece of data then we are assigning topics for about like um for a couple of hundred U uh million users across the various social networks and then uh finally in our Corpus we have about uh 10,000 uh topics and all this like we basically want to keep fresh and the refresh rate is like once a day and uh then uh that's the challenge of having a lot of like data by volume but another challenge is also being able to scale uh in terms of like how many sources you get uh basically of data sources right so we collecting from multiple social networks and then from each social network you may have like different uh uh data streams right like you can have like a user graphs and then action and others and finally the topics that we want to produce they need to be publicly recognizable so basically uh like sometimes if you're like searching something maybe it's not uh something that you would like to be display on your profile page right so basically problem statement uh problem we are trying to solve problem for given applications and some of the applications it's basically being able to show like a user topics on his like a profile page on the cloud another is uh basically be able to solve various user targeting problems like being able for a given question to route right question to the expert to the influencer in the topic that is interested in answering the question right then maybe giving a suggest suggestions to the user who he should follow or uh maybe like being able to award like a free perk to somebody that is influential and kind of interested in a given topic and finally also being able to do like a decent like a Content recommendation based on users's interest or maybe even based on the interest of the user's audience if he wants to amplify his influence for example uh so one of the main challenges in the processing social media data is that basically you're all dealing with like a high volume of the data but then like the texts are very like a short form and you cannot uh really rely on the text to be like grammatically correct so that's one of the main challenges we are dealing with and then the second is uh is the scale so basically usually in Academia you're just going to you're going to do like a uh NLP approach like for example do like a named entity extraction which is uh based on the let's say like for example St Stanford NLP model if you're going to do that like for a single tweet it's going to take you about like a 10 milliseconds well you can do another approach where you're basically already pre-calculating your entities which are based on some like a knowledge U knowledge database right so like if you go with the other approach we can actually speed up named entity extraction Maybe by like a 200 times like in this example and uh basically there is another benefit on relying on knowledge based dictionaries and that's basically that every everything you already have in your dictionary you have link uh to to its uh basically record in the knowledge base which gives you much better context and ability to kind of later extract the topic so in this example for example like the rule of flow NSA e like all of them were kind of like extracted and basically they do link uh to kind of U let's say in a free based machine IDs which were kind of represented by them and uh so one can also wonder like why do we even need like a multiple uh social networks to kind of build like a better profiles about the user like uh because like on a Twitter or a Facebook people spend most of time like are other networks even important well it turns out that uh users exhibit like a different behavior on different networks and uh they exhibit also like different interest on the different networks so for example here on the Y AIS we have like a relative number of users and then we have like on the other axis cumulative number of phrases within the social media message so you can see like that on Facebook actually user like talk the most like they are most veros after that like Twitter follows and then order of a magnitude on LinkedIn and Google+ like user are less talkative and uh basically does that mean that uh Twitter that for example Google Plus and Linkedin don't matter well it doesn't because like usually on a Facebook you're Eng engaging in a very casual conversation so you're are like a uh often time not even very topical conversations are very not very topical in the nature and um on the other side also like basically if you look at even the most active U users and like if you look at overlap of phrases they use like let's say like on a Twitter and the Facebook or any other given Network at most those phrases overlap to about like a 7% so actually users do kind of exhibit different interest on the different networks so how our pipeline uh looks like so we rely on uh basically uh free base Wikipedia and actually like a cloud cloud scores and a cloud display names to build uh the basically the knowledge based dictionary and then uh we leverage that throughout the pipeline so as we said before like once user registers like with Twitter Facebook LinkedIn Google+ or the others Cloud collects that data we collect user activities user profile information and then the user graph information basically connections between the users then uh we extract um phrases like uh from the given data and after that we map the phrases to the topics and then uh we use like a basically we normalize our topical features in respect of user uh and then uh we use like a ground tro to build the model and then U like models we update like uh frequently but not automatically oops and um yeah I have your attention please building staff will be conducting a test system please disregard any alarms this is only a test rep this is only a thank you yeah cool so then uh once we build the model we basically push it to the production yeah what ground so ground tro actually a d is going to cover like in a couple of slides but good question and then we build like a topic assignment set and then basically push it throughout the kind of application for various uh things and then uh like a bit about like uh what kind of um stack we are using here we we are using basically like um uh Hado cluster like we are heavily relying on map reduce and actually we rely on hive most of the time for a majority of things like um and uh like high turns out to be a pretty convenient because it actually like you get like a data catalog for free and then uh like often time you can develop your algorithms basically within the UDF so any kind of complexity it's kind of like a bandal within the UDF and then like udfs by the nature are very reusable so they are your kind of building blocks and on top of that then engineer doesn't have to worry about uh rewr writing like bunch of map reduce and worrying about just implementing the data transformation because the data transformation comes for free basically because you're using um um hql markup right and um basically are there any questions so far about the system cool so now ad is going to talk a bit about um basically feature engineering and thanks um my name is adya I work on the data science team with nman pranik and the others here um so I'm going to talk about some of the details of how we actually uh par out this text and finally build models that uh that we can assign topics to users uh so to start off with uh the first step that we need to do is basically Define how we are creating our features so let's say you wanted to build a machine learning model the two ingredients are you need to have like a bunch of features you need to have a bunch of uh ground truth and then you would maybe in this case we build like supervised learning models for this case so uh how do we actually generate a feature out of the text the first thing we do is that we leverage the knowledge based entity dictionary that nania talked about to extract bag of words from various sources of data that we see so for example let's say we are looking at only the Twitter message text that people post when they tweet about something uh what we would do is we would aggregate all the tweets for a given user then pass them out as bags of bags of words based on the dictionaries and then use our uh use the same Knowledge Graph to transform those entities into topics so what we get is for every user we would get a feature called Twitter message text and that would have a bag of topics associated with it uh now a particular a particular topic may occur in multiple bags so for example you may be talking about the same thing on uh when you tweet but you you may be also talking about it when you're posting messages on Facebook so uh a similar uh and you may do that with different kinds of strengths so what we what we can then do is we basically transpose this Matrix where uh you can now associate a topic with a user and a feature Vector Associated from from the from each data source and a certain weight with it so this gives us a feature Vector for a pair of a topic and a user which we would then use for the ground Truth uh another interesting thing that uh is important to consider when you do feature engineering is that uh one of the challenges that nania mentioned before was uh in social media the context of a certain uh message may be very small that you may not have enough information to actually extract a big bag of words from a given piece of text for example let's say this user who posted a photograph of a car and maybe some small caption there with maybe some small hashtags now just based on the text that he posted on the top there you may not really be able to extract a good bag of words from that so what do you do in that case so what we can do is we can expand the context of that post by looking at all the reactions that he gets on that post so there are a bunch of users who come here and start commenting about it and then when they comment they you now have like a much larger piece of text to actually do your part passing from so you could say that from this post you get a much larger body of text from which you can extract more phrases which means you can extract more context as well so uh what we call this is that the initial original post is something that we call generated uh data source whereas this is the reacted data source another kind of data source is also let's say something like Twitter lists where the user is not actively involved in generating or reacting to the content but you have somebody else saying that hey look this this guy Lou CK he's influen he's he's maybe some a big uh person in the context of Arts so you know that somehow you're able to associate the topic of Arts with the person uh Louis CK so again in this case we call it accredited attribution where uh there's a third person who says that this person is uh associated with this topic another example of this is when you let's say uh tag somebody on a post uh even though that person himself has not generated or reacted to any content you're tagging him on a post and saying that uh this this particular URL is somehow associated with him uh another important piece that becomes important when we look at the long tail of users is that is the graph connections uh let's say uh you only looked at all the that you generated in Twitter and try to pass out topics from it what you're then going to end up with is that you're going to end up with topics for only the active users on Twitter which means that all the passive users who maybe follow different kinds of people maybe just consume content but never really post you're not really going to see any text for them so you're not going to be able to pass out any topics so that is where we leverage the graph connections because now you can say that hey this person although he doesn't actually uh tweet much he's following these 10 different guys who are always talking about big data so then you know that hey this gu actually has some association with big data so you're able to create a bag of words called uh Twitter followers and that has a bag of words associated with the people that he's following so these are the four kinds of broad features that we generate but what this allows us to do is to explode our feature space into uh a much bigger area so uh to give a specific example let's consider something like GP URL meta 90day credited so the way we break this down is that we consider the network the data source the time window and the attribution so uh in this case go+ is the network URL meta is the data source that we are looking at that is all the metadata from the content that somebody posted 90 days the time window over which we are aggregating the data and credited is when somebody was tagged in that post along with uh the posted URL now what this allows us to do is actually swap in and out different kinds of uh features so let's say instead of 90 days I want to actually look at only a 7-Day time window so I can immediately create a new uh uh a new feature in the system which is just a simple hi query to change the time window and I get uh an explosion of the kinds of features I could have or let's say I want to incorporate a new data source from Google+ where now I want to start passing out hashtags so I can just create a new feature called go+ hasht 90day credited and that would again give me another data source and another feature in my system so in this way what we do is we transform all the data sources that we get into bags of topics and uh those bags of topics are then transposed into getting feature vectors for every user topic pair so that is one part of the puzzle the other part is actually the ground truth when we actually want to do the supervised learning so for this purpose what we did is we designed an experiment where uh we are able to associate a user with a topic based on what his friends think about him so let's say uh um we are evaluating n mania's topics uh I would just randomly sample a bunch of topics for nania and show it in this kind of a UI and then if I am an evaluator I go in and say that hey I I know that n is uh interested in swimming he he's he he's associated with Cloud he knows C++ he knows algorithms so I'm able to associate uh all these labels to nania uh saying that he's either associated with this topic or not uh and then this evaluation is done by multiple people for uh multiple users so in our experiment we had 43 evaluators who evaluated around 760 users and they created around 30,000 labels so this is the ground through training data that we would then uh train our models on so uh this ground truth is then broken down into a training set and a test set so we transform the bag of topics to get the feature Vector we train uh this on the training set using a binary classification model because for every user topic pair you have a zero or one label and then we evaluate it on the test Set uh so some of the results here on a per feature basis are are shown there where uh you have all the different kinds of features all the different kinds of attribution and the Precision recall that they shows some features they may have low Precision but maybe they have very high recall so for example if you I'm sorry this concludes our task build system I repeat this concludes our task thank you okay yeah so uh so some of these features may have high precisions but may have low recall for example Twitter list has a high Precision but low low recall whereas something like graph based features may have low Precision but maybe we have high recall so a balance of all these Fe features in the model gives us the uh final ability to associate multiple topics to users so some of the other contrib uh comparisons we do is uh comparing how networks compare to each other comparing how attributes compared to each other and so on so just to uh show you some examples uh we are able to parse our topics like for Barack Obama politics Healthcare Affordable Care Act Congress uh these are all the topics associated with him for Maris it's yahoo Google technology and so on um so like Nan mentioned one of the goals in the beginning was that we are able to associate topics to uh assign topics to hundreds of millions of users not just like the top uh active active users and from this graph you see that uh lasta that is the system that we call it uh that's large scale that stands for large scale topic assignment uh it actually performs better than using any single uh Network on its own because you're able to pass out more information uh so on a per user basis you're able to assign more number of topics to that user but also in the total volume of users you're able to assign topics to more number of users and uh just for those who are curious uh we did some experiments to see how topics compar across networks so uh some patterns emerged like for example on Twitter and Facebook entertainment is a topic that's pretty prominent whereas on LinkedIn you clearly see like the business is a very prominent topic there uh if you look at an individual Network like Facebook you see that topics such as lifestyle food and drink are pretty prominent compared to all the other networks uh and what uh our system allows us to do is to get that information from every Network so it acts almost as a superet of all these information so the key takeaways that uh of the system is basically if you wanted to build a scalable system for assigning topics to hundreds of millions of users you cannot ignore the long tail you have to have uh uh features in your system that address the longtail of users who are not very active or who may not have enough information and uh in order to do so you may have to expand context and so on so expanding context is also very important for topic derivation because a lot of the text that you see on social media especially is usually very limited in context un so you are you have to expand it using either information from the user or information surrounding that uh post or uh any other means that you can find um and aggregating information across social networks actually helps you get a deeper understanding using things like Freebay and Wikipedia gives helps you get a deeper Knowledge Graph and uh at a level entity extraction so uh just getting a deeper understanding of topics rather than a shallow understanding in terms of just passing out text and trying to map it is something that uh is very important when it comes to getting good topics out U finally if you have to design this in a production system and scale it to millions of users it's very important to have an ability have a framework where you can experiment and prototype very quickly so if unless unless you're able to easily add new data sources or easily add new features you're going to spend a lot of time U on very narrow subjects which means that you're not going to be able to iterate fast and so on so uh experimentation and prototyping is very important when it comes to building model so this is something that an engineering uh solution is required for this so that it can scale to millions of users uh finally we here we talked about uh assignment in particular about how we can assign topics uh uh problem that is further to that is the to problem of topical expertise if you know that PE people who are interested in a topic but is also an expert about it how that could lead to much a larger set of applications and it's also a more challenging problem uh all the slides we presented here are based of paper that we published in kdd last year so if you guys are interested you can always go to this link and get more detail details sir uh I guess that's it any questions for I have a question yeah so how do you monitor new topics uh new topics yeah so I mean based on your model you could have X amount of how monitor so we regularly update our ontologies and okay yeah the question was how do we monitor our topics and uh how do you keep them updated uh when new new Concepts come in uh so there are multiple ways we do this one is that uh we keep updating our knowledge graphs like the Freebay and Wikipedia on a monthly basis so that we are always up to dat with uh the current Concepts that are added in there uh the other way we also do it is using social media itself so we use leverage the cloud data that we have uh to identify uh uh New Concepts and popular Concepts that come in every day and we keep updating our system using those new Concepts yeah how do you know that the system is as precise in assigning topics as other systems because obviously you have to process a huge volume of data right and uh you mentioned that the Stanford name en recognition wouldn't match wouldn't be able to do it but maybe it's more precise so how do you know that you're precise right so uh for this particular study in terms of the baselines there was no uh previous work on multiple social networks oh I'm sorry uh the question was how do we know that the topics that we are assigning are precise uh how do we how do we know that maybe uh Stanford NLP might be performing better in terms of accuracy but it may not scale as well uh so for this for this particular study there was no Baseline comparison because there was no previous study on multiple social networks there were individual studies such as only those that use Twitter lists but then those become Limited in terms of comparison U but the way we evaluated our models was based on the ground through data which is based on human evaluation so that's why we created like this experiment where we gathered collected this data before and then evaluated on the test set which was independent of the training set any other questions cool okay thank you