data.bythebay.io: Xiao Fan, Scalably Internationalizing Millions of Latent Semantic Labels
so yeah good afternoon everyone I'm on the Simon team at work day we're workdays search and analytics team so the first thing you're probably wondering is what exactly does scale ibly internationalizing millions of latent semantic labels mean so in particular what do we mean by semantic labels in this case and what sort of scale problems we run into when we're trying to internationalize so it obviously requires some background and it's somewhat specific to some tools that we've built so I'm going to open up with a description of our original problem we were trying to solve job title search and then we talked about a tool we built called claws that solves this using semantic labels I'm going to talk a bit about how claws uses those a bit about how we build the semantic label data set for claws and then from there I'll seg into our internationalization approach and then some validation results and next steps so the original problem we set out to solve was job Hetal search so basically if we were trying to search for job profiles or candidates in the system based on their job title so one of the problems you run into immediately here is there are a lot of different ways to say the same sort of job title so say I'm searching for a software engineer well someone might call themselves a senior java computer programmer someone else might call themselves a ruby web developer someone else still might use like very small abbreviations and call themselves a senior you I injure the and someone else might use like random jargons and call themselves a JavaScript code ninja so obviously you're going to run into a lot of problems with textual search here because software engineer is not going to return any of these results these don't have software or engineer anywhere in so how do we solve that well what we really need is some way to basically recognize the true semantic content of these job titles so basically we need someone to say that computer program or web developer UI engineer and code ninja all kind of means software engineer so our solution to that is semantic labels and we have a tool that does that we call it claws so here find my mouse so here i have a clause API tester and I've put in our job titles i don't know if any of you can see that far but it's this is these are the same job titles i had on the slide so senior i can read them from here so kind of senior java computer programmer Ruby web developer senior UI engineer and Python code ninja movie falls I've also got thrown in software engineer at the end for for good measure so wait not found 404 that's not good just a moment sorry okay so apparently my demo here isn't working so I'm going to skip onwards so what you would have seen there is a for when i pass clause this list of job titles it'll pass bat it pass me back a bunch of data structures with a kind of set of labels for each job title so for senior java computer programmer I've would have gotten the the labels basically job function software development staff specialization Java seniority senior for Ruby web developer similarly I would have on job function software development staff specialization web specialization Ruby again software development staff for UI engineer a the specialization ui/ux Python code ninja kind of the same thing and so basically the way you would use this in search is you would at indexing time you would take your resumes into that have job titles associated you would run the job pedals through clause get back the labels and you didn't take those labels then at search time you pass your query software engineer through the through claws again to get another set of labels and software engineer here will just give you back the label job functions software development staff at which point you can use that to basically match on labels and get back the search results you were actually looking for based on the semantics of the query so kind of so basically the basic idea is we break down these job titles into smaller meaningful components or Engram so back here we were talking about you know we'll get back a job function software developed a for each of these titles how do we actually do that so if we take an example job title say senior java software engineer what Klaus will do when it sees this is it tries to recognize the the meaningful n-grams in here so software engineer will tell us that your software development staff Java will tell us that your specialization is Java and seniors should tell us that your seniority is senior so kind of we're bakey breaking this larger job title into the minimally meaningful components okay so we know what our approach is how do we actually do that so Klaus under the hood actually stores a mapping from engrams to labels so the n grams are these meaningful components they're basically n word phrases and so here's a sample labeling or a sample n-gram to labels mapping it's a bit dense here are n-grams so Java is an Engram and software application specialist is another and Graham and the labels are down here so here this says that Java tells us that your specialization is software Java software application specialist tells your specialization is software applications and your job function in software development staff it's kind of hard to read in this form so I've like summarized in a little table so here are some example in grams and example labels so yeah so in summary basic laws keeps a mapping of these Engram to label associations and given an input we will try to match the engrams in the input starting from the largest first and basically in this way we can decompose the input into a set of n grams and once we have those we can return the labels that are associated with them through this mapping so one of the benefits of this approach is it allows us to reduce the size of the space that we need to match so basically the way we train Klaus we time to start up with a sample set of three hundred thousand unique English job titles and by breaking down to the labels we can describe it by only about 30,000 labels or in the system right now in addition once we've got the N grams mapped this approach can also allow us to map job titles that weren't into the original set because you can kind of create new job petals by mixing and matching pieces so going back to our example then like I said we're going to try to break this down from largest in grams to smallest so when we see senior java software engineer we will first pick up software engineer the Ngram because it's the largest that we can find presumably have no trigrams here then we'll pick up senior and java and like i said because we're matching by n grams we can mix and match the pieces and still get the same meaning so if i swapped out software engineer for computer programmer i should still get the label software element staff likewise if I swap that out for developer I should also still get software development staff now the second example actually isn't quite right developer is a bit a little too general term to tell you that you're a software developer you might be you know in this case this personally did a Java developer but you could also be a real estate developer or an urban developer right so in this case we need to give Claus a little more context so like i said though Claus matches largest in grams first so we can give Claus more context by simply cheating a little and mapping a larger Engram so if we map java developer and assign it both the labels software all and staff and java we can still get the desired labels for senior java developer without basically having this over general term developer maps to software developer ok so we've talked a bit about how clause goes about producing basically recognizing em in grams and producing those labels so obviously most of clauses intelligence is in this Ngram to label mapping so that leaves this question how is that mapping produced so here I actually have kind of our basic process for producing the Ngram mapping and long story short it's a manual process so what we do is we take those 300,000 job titles that we've sourced from the wild and human data analysts will come and review those job titles kind of so when we source the job title from the wild we also have information on basically how frequently how frequently see them so human data analysts will come in and look at kind of the most frequent job titles you see and manual identify ok what are the interesting grams on this job title and what are the correct labels that we should assign to them once they've labeled some engrams we can actually take that partial mapping that they've produced feed into the clause and try to try to label the entire sample job set so what this does is it gives the data analyst a chance to review see if they're doing something like over matching like mapping developer software engineer and finding that all real estate developers are suddenly software engineers and it also it basically allows them to basically what is now the most common unlabeled job title so it gives them like where to continue mapping and some review to so that they can correct errors and their previous mapping so once they review they label more n-grams we read level using clause they review and so on and so forth after enough iteration of this this if it process eventually converges and when the data analysts reviews the job titles they'll say okay this snapping is now good enough we have basically basically complete coverage over the sample set and we also and the labels seem more or less accurate and at that point they can publish the data set and we can use it in production clause for enhancing their search results okay so given that that brings us to our problem of scalability basically if we want Klaus to work in other languages because of our n-gram based approach Klaus is only going to understand words that we've taught it if we've only thought of English Klaus is not going to understand French on the other hand we'd really rather not have to manually teach Klaus each language because the you know the work our data has to do is relatively specialized and pretty specific to our product so it would be you know very manual labor intensive to try to do this for every language moreover even we had if we had infinite resources to throw at it like like I said our data analysts to specialized work we'd have to bring someone in and train them up on the process and then they would have to catch up to the work our English data analysts have already done so of course we'd like to do this better can we do that and we think we can so basically to review Klaus models the world using engrams and Kylie we can think of the process that the data analyst use to map those and grams as the data analysts already know what their desired labeling on the universe of job titles is right so they give it a job title they know hey this looks like a correct labeling for that job title so kind of if we think about it if we already had a desired labeling of job title as we might be able to come with an automated way to deduce the engrams the the analysts produce kind of the hard human part of this is knowing knowing that base step of what labels belong on any particular job title so in order for that to happen of course we have a couple prerequisites one we need a pre labeled set of job titles and we also need an Engram structure that maps relatively clearly I'm going to come back to this point since it's not quite going to make sense just yet but the idea for getting our pre labeled set of job titles is we can take a set of foreign job titles and source translations for them once we have translations for them english translations since clause already understands English we can use clause to produce labels for those English translations right and presumably if our translations are good enough those labels should be the correct labels for our corresponding foreign job titles so this will get us kind of the desired pre labeled set of job titles at which point we can try to deduce the Ngram structure now I had said that the Ngram structure needs to map relatively cleanly kind of what I'm saying is here is a by text need to work I need to have some guarantee that if I have a set of French job titles they'll have they'll have reasonably shared structure with the English job titles in terms of the words and concepts underneath so kind of because our label set is in English the naturally are our labels have some some of the structure of the English language to them but luckily for the most part we think this should work out like in these examples for example surgeon means surgeon orthopedist means orthopedic cardiac cardiac professor professor neurology neurology so these all kind of like map pretty cleanly there are some edge cases like here this example professor at a university day you know sorry my French accent is fairly terrible professor the universities and medicine this translates to senior university lecturer practitioner this largely this does mean the same thing but we see some differences like Professor translates the professor here here translates the lecturer medicine usually translates to doctor and here translates to practitioner so we can hope that the labels on the English side are enough to figure this out but we anticipate that there will be some cases where we'll run into issues here that said we still do expect that in the majority of cases we should be able to rely on shared structure so so kind of going back to the previous slide the key step in this is how do we do the automatic n-gram deduction right how one gram deduction actually work so kind of the underlying idea if I argue it informally here is if we have enough job titles that contain the same n-gram so in this case I've come up with a I found a bunch of French titles for various kinds of surgeon so all of these job titles would expect on the cloth side after we've translated and label it to receive the label of something like job functions surgeon and moreover we look at these all of these French titles share word sure Jen right and in fact it's the only word that all of these titles share so we can probably deduce that surgeon in French means job function surgeon to our label set so kind of informally that's the idea behind it so let's kind of go over the overall process then we built a tool called Leica that actually does this so first what we do is we gather a representative sample of French drop titles we need to source English translations for these job titles now this at its root is still going to be manual secretly because for our purposes like automated translation isn't necessarily going to work we need some specialized knowledge of how the job titles in this field are structured or if we go back to the first slide there were some you know sometimes you get specialized abbreviations or jargon for job titles that automatic translation is not going to do very well with so but luckily like this is still more scale for us because translation is a relatively commonly requested service as compared to our task of Engram labeling which we definitely have to especially train people to do translation we've managed to find third party services that will actually find the appropriate specialists and translate the job titles for us for reasonable prices so once we have those translations we can produce labels for the English translations by a clause will assume that these are the correct labels for the corresponding French job titles because we assume we have a high enough fidelity translation and then given the labeled French job pals will try to deduce an n-gram to label mapping that produces the sample label so let's try to run through an example of that so I have here a very very reduced hand-picked set of job titles I've made sure that they have a fair amount of overlap in the Ngram so we have some chance of learning what the label looms should be so here I have chef de proudly senior shed prodigy junior state of Hardwick senior probly shade approach a senior shade protege junior so this is kind of a senior project manager junior product manager senior project manager junior project manager so we've got the English translations we've got the French job titles will produce labels for the English and assume those belong to the French and then the next step here is given labeled French job titles deduce an n-gram to label mapping that produces same lately now obviously there's a lot of work that goes on behind the deducing how do we actually do that at a very high level we basically want to examine all possible n-grams that could come from our initial set of job titles and see if they correlate well with particular labels now obviously we don't want to examine all possible n-grams so we have some optimizations here so here's a more detailed view of Ngram deduction given a labeled set of French job titles we will consider a hierarchy of all possible n-grams I'll get to what that hierarchy means in a moment basically it's we form a hierarchy of Ingram's by inclusion and the seals a graph so we can see kind of the largest parts was the top of this graph with labels from our French sample set and then we can propagate these down into smaller n-grams we have some rules on the propagation we require that a label if it's to be propagated and used to belong to Atlee two parents and it needs to occur to at least seventy-five percent of parents these numbers were kind of arbitrarily picked they could very well be further tuned but they seem to be yielding reasonable results so far at least for our initial runs of this so let's this is kind of dense and hard to understand so let's look at some examples so first like to review the the universe of possible n-grams if we look at any given job title c'est chez des produits senior naturally so this is a forward job title it's naturally going to have one forward n-gram namely the entire job title it will have two three word n-grams so these are basically just the runs of three words within the title and similarly three two or ten grams and for one word yoona grams so if we take the union of this across all our input job titles will get kind of our universe of possible engrams and furthermore if we look at them there's an inclusion relationship between the smaller engrams and larger grams in particular each smaller in gram is included is fully included in some larger and Graham it's the substring of it so we can use that to define a partial order and get a graph out of this set of n grams so now that we've defined the engrams hierarchy we can go back to our steps the next step is basically seed the graph with labels from our french label set so as a reminder here are the labels we got for our initial french job titles we will add that to our hierarchy on the full forms of the job titles at this point we would like to propagate the labels so i'm going to start off looking at one of the at basically the one of the largest unlabeled n-gram so this one is of size three it's shada prod we if i look at its parents i noticed that both of them have the job function product manager label so this satisfies our criteria it has at least two parents with the job function product manager label and actually one hundred percent of its parents have that label so we cross the seventy-five percent threshold we can propagate this label we don't propagate at either the other labels because we only have one pair with a senior in one parent with junior so next we can look at this this node des produits senior this this Ngram only has a single parent so actually it cannot receive any labels because naturally none of his labels can occur on more than one parent so actually it receives no labels and we prune that node we prune that node and actually since since we pruned it this node now no longer has any parents will also prune any orphan children that results as because of previous prunings in reality like we don't actually need to go and prune the orphaned children we just won't generate it when we're traversing this tree so we actually do is we generate each generation of children based on the previous survivors and the parent generation so if we follow this through then the next thing we do is we prune defraud we junior and some children pruned approached a senior and some children we do propagate one more label the jf project manager to Chez de projet and actually the rest of this turns out to be pruning so now you'll notice the this isn't quite a perfect result we haven't propagated senior or junior but we have managed to figure out that shade of roadway means product manager and shade of projet means project manager and once we've done that actually we're basically done because once we're done with this label propagation the nodes that remain are essentially constitute the Ngram to label mapping that we were looking for so yes the second demo would work I had a demo set up for French job titles actually I'm just going to skip that because I can't read this screen so so obviously we want to quantitatively know whether our approach is actually working so we do cross validation so the sample set of French shot paddles we had was actually only five thousand titles large we do random sample cross validation basically you know standard thing we with the part of the set is validation we use the rest is training and to actually calculate precision and recall we from the English translations in the validation of side of the set we can produce labelings from our original English n-gram mapping from the French job titles we can then produce labelings based on our derived French n-gram mapping and then to calculate recall and precision we basically do number of match labels but we do it based on number of match labels not false negative labels and false positive cools the recomposition are worth checking looking into slightly further I feel basically initially we were trying we wanted to come up with recall and precision based on match statistics on the engrams themselves we quickly found that that wasn't possible because any and given n Graham is not going to necessarily be a full match with the original n-gram or a full mismatch so here actually you know we have sure general lost OPD because French orthopedic surgeon is the English the orthopedic surgeon might produce job function surgeon and specialization orthopedic whereas the French might produce job function surgeon specializes in cardiac we can see that like kind of labels don't fully match or fully mismatch but if we count by labels we can come up with more more well-defined mastics so here we can count a job function surgery as a true positive specialization card cardiac as a false negative and specialization orthopedic as sorry specialization cardiac is a false positive and specialization orthopedic as a false negative so with that let's take a look at the French cross validation data so here are some graphs as you see for French actually it the the data don't look so good the so we had about five thousand titles and consistently throughout our tests precision hangs out around fifty percent and actually towards the end actually drops to around forty percent as we increase the sample size recall grows but it maxes out at around thirty percent so these results aren't very good but we suspect it's because our French sample size is actually too well luckily we have another much larger set of job titles and that is English English is very easy to translate into English we just take the English we can still test our method though because fundamentally what leica is doing is trying to deduce an Engram mapping from a job title labeling so kind of trying to deduce a model to produce results it's still we can still do this for English we just hide the model right so beyond that everything is about the same we have an original English ingram mapping and derived English and Graham mapping if we look at these results recall looks much better so here we get to around like you know twenty percent of our input set and we're at seventy percent recall this keeps rising two around eighty five percent precision is still pretty poor unfortunately this is consistently it hangs around sixty percent now I had mentioned we suspect that the French results are poor because we don't have enough job titles in the set and the English bears out some if we look down here around one percent so this would be so we have three hundred thousand English job title versus five thousand French job titles around one percent so this will be around 3,000 English subtitles were down to around twenty to thirty percent recall which matches up with the French results here actually if so hopefully if we continue this line will get better results down the line so that said the precision here is still worrying because if we have sixty percent precision since we are a search application sixty percent precision on a job title search for resumes means that about half your candidates are coming back as bogus now we can we can improve this slightly if we consider weighted statistics basically since different job titles occurred with different frequency we can basically wait our cross our precision recall by their frequency and if we do that this rises to something like eighty percent but that's still pretty bad you one in five of your candidates or bogus results so you still have to do a lot of manual sorting through them so why is our precision bad so I took took a sample of some of our false positive labels here so we have wind farm developer produces job function farming staff when firm developers also some reason software developer solar fabricator is industrial manufacturing welding PV applications engineer is soft rebel and staff and software applications this one actually kind of sounds believable but I looked into it pv applications engineer I think exactly someone who works with fuels so this obviously what photovoltaic oh okay the solar panels sorry yeah so this person obviously should not be software development staff and finally we have payroll settlement specialists or we get financial operations staff and payroll so I took a look and kind of mapped back these labels to what what the engrams were that we're producing them and if we look at it kind of wind farm developer it's kind of what you might expect basically developer gives us the software valve and staff label farm gives us the farming staff label fabricator gives us this industrial manufacturing welding PV applications engineer continues to be weird applications actually produces both the software applications and software development staff label and finally payroll produces payroll and financial allocated operations staff so if we look at this we can say that basically most of our precision errors seem to be due to label over propagation so so basically kind of dr i was talking about before where developer gets mapped to software development staff this is good news for us though it means we can probably tune this in bye-bye basically tuning our propagation so this is a fixable problem and not a fundamental flaw of our approach and i was going to cover so kind of our i think i'm out of time so let's see here so i'll skip the next steps and summary if you guys found this interesting as i said i'm on the work day Simon team and we are hiring yeah my email is shoutout fan at work com email me if you be interested or if you just want to talk I guess we can open it up for questions thank you back yes at the moment Ian workday we are trying to try to building recruiting search so basically for HR managers to search through their existing pool of candidates for four ones they think they would match and actually originally was built as so the Simon team was previously startup called identified and mostly did kind of kind of the same sort of thing indeed does so yes actually so job titles is the example I use but we actually also map skills and majors so II right so I got to say we haven't thought that deeply into Asian language German our hope is that basically with a good enough tokenizer those should still break down into familiar enough and grams yeah so actually one of the things i listed on my next step slide that we created was we will need to do language-specific tuning so for frantically we don't really don't do stop words if you were looking at my shade of fraud we slide we could probably just drop the dough and not lose any information but for japanese and chinese obviously we need a good tokenizer driving sample oh so you mean basically floating up interesting we haven't really looked into that that's an interesting idea so basically looking at which words occur most frequently with each other and floating those up its potential engrams yeah that's uh no we haven't looked into it yet but it sounds like a good idea thank you