data.bythebay.io: Vita Markman, Of Rules and Probabilities: Computational Linguistics Methods
for the nice intro so first I would like to start by saying that a supervised machine learning is a very popular and we all love it and it is used in basically every domain of natural language processing starting from part-of-speech tagging to parsing named entity recognition sentiment analysis document classification however it's really important to recognize that machine learning is not magic learning especially supervised learning requires large amounts of good annotated data and there are often times in practice hurdles to obtaining such annotated data sometimes they're illegal or privacy issues such that not all the data that you want to learn from can be actually externally crowd-sourced in other cases when it can be crowd-sourced oftentimes the domain knowledge is so intricate that the crowd workers do not have it and it's not really possible to get quality data that way and finally if you have sort of pony up some money and you hire expert annotators they usually cannot annotate massive amounts of data because they are quite expensive so the focus of today's talk is specifically on overcoming the problem of insufficient and oftentimes inconsistent labeled data in particular I will focus on that a case study of improving a conditional random field a CRF model that is trained to tag and label named entities in search queries and the proposal I will discuss in addressing this data shortage shortage of annotated training data is to develop a grammar that will label some portion of the training data consistently as if modeling an ideal annotator and as a consequence of this proposal we can leverage the grammar to aid and ciara feature engineering do some post filtering of the results of the model and fix some obvious errors as well as produce more data for other relevant applications so first I would like to give you some context and what is the task that I for which the data is being collected the task is search query tagging the goal of query tagging is really to understand the relevant entities in search queries this is really an extension of a named entity recognition problems so when a member goes on linkedin and searches something on linkedin such as john smith software engineer linkedin mountain view they have typed a bunch of words what we want to do is we want to understand them as entities like first name john last name smith there is a job title title for short software engineer there is a company like linkedin there is a location like mountain you or if someone just types machine learning CMU they didn't just type a bunch of words they actually typed a something that we call a skill machine learning and a school CMU so as you see there is some classic named entities like name location organization but we actually extend it a little bit more to add a title and skill which are really critical to our ecosystem of LinkedIn because you could imagine you may be looking for jobs of specific titles or four skills like data science or machine learning as well as we made some refinements in company and school as an organization being treated as two separate things now why do we actually care about this because understanding user intent improves ranking of search query results for example if you typed john smith you probably want to go see some member page not just some random words John and Smith and different type of text and if you understand user intent better you can produce better user experience so again a little bit more about the data it turns out that search queries on linkedin are incredibly short ninety-eight percent of them are less than four words long the bulk falls between two and three words seventy eight percent so and about twenty percent are one word and only two percent are four words or longer most of the time they're actually unsurprisingly people names like Larry Page but often there are also frequent job titles like project manager or software engineer as well as skills like data science machine learning or some mixture of these things like jobs at Microsoft or data science internships or Google software engineer so this is kind of how the data looks like and again I want to emphasize as part of this talk the most important types of we want to tag is titles in skills and they have very specific definitions to our ecosystem that are incredibly important a title is really a noun phrase they can be used to fill in the blank of something like I work as a data scientist business developer salesperson in contrast a skill is something you can learn something you can be good at or also an area of work for example I want to learn blank data science machine learning or I work in the area of sales marketing data mining etc so it seems like a fairly straightforward distinction however it turns out that for crowd workers this is an incredibly complicated thing to sort of grasp because it requires reading a lot of instructions and for one reason or another even though people gave us very high agreement scores on the crowd source task it actually produced complete noise specifically people consistently label something that is a skill as a title so they see data science even though it's given as an example as a skill but they say it's a title they see human resources as they say it's a title everything is a title and as a result our model can't really learn anything because everything is labeled halfway lik this way half way that way sometimes they're right interestingly they never actually say that something is a the title excuse me that something is a skill when in reality it is a title so they say salesperson is a title they don't say it's a skill but unfortunately doesn't help us because the data comes out to be incredibly confused basically the problem is skill is confused with title and the training set half the time the CRF model mistakes skills / titles and that produces kind of bad results on that particular type of named entity so what are we going to do with this is we're going to make the training data consistent as well as improve some features and do other CRF things but let's get the bulk of it right and so what we going to do in we're going to start with focusing on what's where's the bulk of the queries well they're short so we going to only look at the ninety-eight percent of the queries that are less than four words long and ignore these other two percent we will use crowdsourcing to label locations and names and companies and schools something that people are really good at understanding so when they see a name they really do kind of correctly label it as a name and then for the remaining we will build a grammar a particular grammar that will recognize titles so what is a grammar it's just a vocabulary and rules governing the combination so we will build a grammar that would say this phrase is a legitimate title this phrase I don't know this phrase is a legitimate title this phrase I don't know and so on and then we will use this grammar to auto tag titles in unlabeled queries and add these newly labeled queries hopefully consistently and labeled well into the training set thus solving our problem of inconsistent and insufficient training day so I will start with some observations observation one is that people usually understand some new phrase that they may not have seen before is a legitimate occupation title so if you have never seen something like director of guitar division or senior dental ceramist which I store for the first time when I was preparing this talk or cheap happiness officer your you may not know exactly what this person does but you know this some sort of officer some sort of force aramis there's some seniority there some importance or you know I call one of my you know cats chief laundry inspector so that's sort of like you you don't know exactly what that means but you kind of get it's a title and how do you do that while you look for the syntax and morphology to give you some hit in particular what you're looking for is for the head down like director or ceramics or officer and that head noun has to designate what we call a core occupation that has to be a person this is why dental sera mist is a title and dental ceramics is not a title so the core occupation like officer sir ma'am as director has to be there in order for this to be a title legitimately according to our definition that we imposed observation to is a corpus of member profiles not queries but member profiles hopefully you know you all have profiles on linkedin are very useful to harvest first quarter q patients like officer ceramics direktor manager owner as well as seniority words like senior staff principal and a bunch of modifiers like hardware dental happiness guitar or basically any words you have that appear sort of in between as well as the grammatical patterns for titles meaning how are these words exactly combined most of the time so we're going to set aside our queries completely and go to member profiles and we're going to look at member profiles now in order to obtain this grammar that then we will later use to tag queries so we will is very simple we will start with about one hundred and thousand one hundred thousand titles from member profiles these are things like marketing manager vp of sales software engineer senior data specialist director of development and we will tokenize them into tokens like manager owner project manager director specialist etc and then we will annotate frequent tokens with very few select type of annotations very few tokens only the top like two or five hundred tokens we will take and we will only look for seniority words which are very stable they do not change like senior staff and principal very rarely does a new seniority marker come into the language they're almost like prepositions which we also look at and of course we're going to look for the high-frequency occupation words they are like engineer manager owners specialist director salesperson there is actually not that many of them and we will look at occupation marking suffixes these are is our or and of course there was a lot of false positives there but we will still harvest some of the most frequent ones that are fairly stable in the language and then we will look for anything else that comes our way in the top 200 or so tokens that we will call a modifier so we don't really know what it is and we don't really care it's some word it usually appears before an occupation or after an off but we will just collect them and then we will go back to our idols and we will replace literally the words with the corresponding annotation such as marketing manager will become modifier we don't know what that word is marketing it's some kind of modifier and manager because I know it's an occupation or vp of sales will be occupation preposition modifier or senior saw data specialist will be seniority modifier occupation crucially occupation has to be present as the head of the noun phrase in order for it to be a title and that is the one thing that we absolutely look for in order to say this is legitimately a title phrase and essentially once we have done this we have these grammatical patterns these templates or rules or phrase structures however you want to call them that cover a lot of new phrase is because seniority modifier occupation is the same pattern for data excuse me senior data specialist and staff software engineer or staff hardware designer etc so with that we basically just to restate what I have just said in three steps we tokenize frequent titles we manually annotate frequent tokens with occupation seniority preposition and whatever modifier and we replace the tokens that we've annotated with their annotations so software engineer becomes modifier occupation and senior data scientist becomes seniority modifier occupation and we are done we have a grammar and so now we're going to go back to our queries that look sometimes suspiciously indistinguishable from titles because they are short and if we see a query that is two words long and it has something we know we will label it according to our grammar so specifically we have let's say a rock weary like arts instructor or director of partnerships or talent acquisition specialist we will tokenize the query and then we will apply our grammar we will first look for the key occupation word in the first case its instructor or director or specialist and if we found such a word then we will look for the rest if we know especially if we know some seniority or some preposition and we will apply that and we will so yeah I see this occupation instructor I know what that is I don't know what this art is I don't know what it means I don't care it's probably some modifier good enough we're going to label it as a title if you saw let's say director of partnership is said okay I have a occupation director I have a proposition that I know of and I have some word I don't care it's a modifier and so on and we will have been able to label a bunch of titles if we didn't find an occupation work when we basically don't know what that was we have no idea we cannot make any judgment so I will skip this and go on after we've applied the title grammar we effectively have something like this staff business analyst is a query somebody issued a query like this it looks suspiciously like a title we consider it a title because it patterns exactly according to the grammar that we have built as a title and then we see something like real estate or human resources or data science and we say I don't know what that is there is no occupation where there I have no idea so now we want to figure out maybe we can learn a little bit about the skill phrases using our title grammar this is a much harder problem to solve and that is why we started with the title grammar that has very stable patterns specifically to observations so first titles often though not always subsume skills but not vice versa so what is data science it's the middle portion of something like senior data science analyst and so if you strip senior and you strip analyst you will get data science that happens to be a skill unfortunately skills have an incredibly varied morphology and syntax as well as recognizing a skill requires domain knowledge for example business development is a skill Java is a skill C++ as a skill Hadoop is a skill and spark is a skill but the latter two you didn't know if you if we were here like 10 or so years ago you may be Hadoop you already knew but spark you certainly didn't and that domain knowledge suggests that you actually need to have some domain knowledge database and where would you get it well you can build some dictionary of these skills hopefully using some friends in order to learn new things like spark that didn't use to be there now for the lack of a better approach we will again reuse our title grammar because we can't get humans to label this consistently and we will do an inference this is a noisier inference and the harder problem as I had mentioned which is why we started with the easier problem but we will do this again we go back to our titles from member profiles we set aside the queries and we're going to say we're going to use the titles from member profiles and the grammar that we have already built in order to make inference and what a skill might be and a skill is basically a title without the occupation and the seniority so if we look at titles like manager sales and marketing or vp of data science or senior VP of corporate negotiations and we remove things that we know our core occupation words and seniority like manager or vp of or senior VP of then we will get hopefully though not always something like sales and marketing data science corporate negotiations etc now there are going to be false positives this is a much harder task because you can get a lot of the time something like senior Google engineer at Google is not a skill or director Los Angeles Arts and Los Angeles is not a skill and so you have to do an additional verification again to the best of your ability to remove these false positive by verifying against known collections of texts that are locations or companies etc and then you build a final dictionary of induced skills that you use to label queries if they match exactly or near exactly like sales and marketing engineering logistics infrastructure corporate negotiations so that way we were able to leverage something we learned to label queries that fully correspond either two titles or skills because there's a short queries and we do not care about segmentation problems so if we find something like special education teacher or real estate consultant or supply chain director or IT security specialist or a field service engineer or digital marketing specialist we know it's a title and if we find something like finite element analysis corporate social responsibility fire and safety social media marketing natural language processing real example talent and recruitment these are skills and that will have some false positives that titles generally do not have because accidentally something like Google or Los Angeles might sneak in and you can't really do anything about it's not a perfect world so in addition we can use title grammar to do feature engineering for the CRF namely it turns out that some of the more effective features for the CRF model is something like word at position J is an dictionary X and not in dictionary why what that means is when you look at a query and the model is trying to feature eyes around the particular location it will say well what is this word oh it's some word like director oh actually it turns out to be in the dictionary of known titles and it's not an addiction area of skills good so I'm going to fire and eventually I see something like this and the tests that you'll have some weight hopefully not indistinguishable from zero associated with it so it's a good feature unfortunately if you have features like this and your dictionaries are all noisy and you have skills and titles all mixed up together with companies and locations then you're pretty much hosed because then you will learn nothing this feature of this would have been really awesome and actually going to turn out to be of no consequence at all so the title grammar that was developed was also used to clean title dictionaries which wrench initially were really noisy by removing a bunch of noise from it as well as skills dictionaries I to augment them and to remove titles from skills and skills from titles thus improving the quality of the features and finally you can use this grammar to post-process your results so CRF is a probabilistic model which is beautiful and very robust and has fantastic generalization and we love it but sometimes it makes really obvious errors errors that you shouldn't be making because you know they're wrong and something like sales engineer which is a two word query if it's tabs skill instead of title is embarrassing you shouldn't do that you should fix it and you should fix it by because and you can't fix it because you know that you have a title grammar that can correct something like sales engineer to a title if it is labeled as any other entity because it knows that engineer is a core occupation and there is a pattern that fixes it beautifully to what it's supposed to be and actually post-processing turns out to give about two percent improvement than the overall f1 measure speaking of that so the results on frequent short queries so these are short queries less than four words long with the original training data for skill and title skill being our main object of concern because it's only sixty-three percent f1 measure was able to go up by thirteen percent with the added training data due to the title grammar and added features again that was a reliant on grammar and the post-processing so it gave thirteen percent improvement smaller improvement in title but for reasons that we kind of were ok was it how it was and we didn't really find doing a lot of features towards improving the title at the time now the more difficult thing to do was to improve rare and long queries for which we didn't build special training sets because that requires a lot of segmentation this is actually much harder problem so anytime we have a query longer than four words it could be a bunch of things squeeze together there and with the original training data in no post-processing it was at 65 because most of our original training data was actually from long queries but with added training data and added features it went up by a person by 3% and with added training data from the long tail that went up to 70 so it was actually an improvement but a much harder thing to do because we didn't have that just the right kind of data to be able to add to it but still more than sort of more than the initial so in some title grammar solves the shortage of consistent training data by labeling new titles consistently and correctly and adding them to the training set assisting in inference and labeling of skills in queries and it also allows us to engineer better features and correct certain types of obvious and errors that the CRF makes using post-processing thus giving us about thirteen percent improvement on skills infrequent queries and five percent in rare queries that are longer and harder to get the method that I was talking about today can be tailored to any kind of short text really like search queries for product names like an Amazon or Ebay or what have you or occupation names in various domains related to occupations or really any entities with stable linguistic patterns thank you any other questions and so on when you were showing the numbers yeah and this on the long tail because we added short data because it you again the data was consistent but it wasn't what we were testing we were testing longer queries so if you add data even if it's good but it's not the right data it's going to go down if you add the right data with from the long tail it'll go up again that's unfortunately because there is so much segmentation ambiguity if you have to maybe like if you have your you know for sure that the first three words are a title then you can look for something longer you without human labels is really hard it's really hard yeah we used the query logs to get the data that we labeled ultimately but we use the titles to build the grammar we absolutely use the query logs and yeah for as the main thing from which we were learning but the query logs were used when to obtain the actual training data we used the grammar to label so whenever I say something like sorry raw queries so these are raw queries from the query logs like art instructor and we're going to label them using our knowledge of what that word art an instructor might be yeah yeah yes yeah no we only use the fact that they were frequent queries the fact that they were really frequent yeah not in this particular experiment no I actually had a so it was the bowl the bulk was the training data the features were like about like about five percent or so maybe four percent but it's not features were they were interrelated because features were able you were able to add the features because of the grammar but you couldn't if you remove that draining data for especially for short Aquarius of which we didn't have enough e to begin with the results are pretty dismal yeah i think there's next talk so i should happy to talk later thank you thank you so much you