Text By the Bay 2015: Vlad Giverts, Learning The Semantics of Millions of Entities
Recording: Text By the Bay 2015: Vlad Giverts, Learning The Semantics of Millions of Entities
foreign 's here ready to go so we'll just get started uh hi everyone I'm Vlad giverts I'm currently senior director of software development at workday which is totally meaningless it's one of those like lofty corporate titles um previously I was the CTO at identified and that's where the technology we're talking about today kind of emerged out of um and this is specifically addressing some of the challenges that we had and I'll get into that in a second so we ended up giving this technology a name because you know for as a startup we needed good PR so we wanted to have something catchy so we have with Simon for systematic mass normalization and it allowed us to learn the semantics of a certain kind of entity that I'll get into in a second in fact we even uh we branded this technology we gave it a robot give it a logo um and because you know that helped us actually communicate it out to kind of lay people I didn't quite understand what we were doing um so the story started about three and a half years ago and it had a lot to do with what we saw as professional data so no no not that data he looks very professional uh no this kind so we realized that there's an immense amount of professional information on the internet this particular site was just one of them but actually there's you know quite a few others if you knew how to look for it you could find them on all the big social networking sites people talking about where they worked where they studied maybe even you know what they're good at a lot more professional info as well on kind of what they call we're now calling vertical social networks but you know where you see a lot of information about people but it's focused around a particular niche I don't know if you've heard about any of these other than GitHub that one's pretty pretty common there's actually a third major source of professional data online and that's corporate team pages so it's pretty common for startups to go and just literally like list out the whole team of people on their sites actually larger corporations do it as well they'll often only list their executives and what we can do then is take all the data you know from the big social sites the vertical social networks the career you know the team pages uh and just slurp them all up aggregate all that data and and that's what we did we did that about three years ago uh at identified we ended up aggregating well over a billion profiles for social profiles public you know profile pages from all over the web ended up creating uh the single identity of record for people there's a lot of things we've wanted to do with it we wanted to create this incredible career path exploration tool for people we wanted to give companies these predictive insights based on the information now because you know knowing a lot about where people work and what they study gives a lot of supply and demand information unfortunately our technology wasn't ready at the time to deal with that so instead we decided to monetize in a very simple way as a recruiting product uh we just let recruiters search foreign content people they wanted to hire across this vast database that we assembled and because we were incredibly creative we wanted to come with a very very creative name so we call the product identified recruit now now unfortunately it didn't work out as well as we hoped so we gave recruiters access uh to the product um we did a pretty good job building out all the informational retrieval systems we knew about search and facets and you know TF IDF and all that nice stuff you know I actually used to build cert systems in another life years ago I was the search architect at a company that was once known as tagged where they had you know at that point probably like 100 million users uh in their system that needed to be searched on constantly but for whatever reason our customers weren't able to find people they wanted to look for at least not in the volumes now we knew the data was there because you know we looked at it we knew we had far more data and a lot of Industries than say LinkedIn but somehow we couldn't bring it to the surface it turns out that you know if you're doing keyword based searches well those keywords actually have to appear in what you're searching on and the problem was that for you know job titles for skills for University majors there was a really fat tail uh in fact I I it's hard to even call it a fat tail it's just all over the place um to give you an example you know looking for software developers you know what what might they be called well you could be uh you know ninja coder or you might be a code monkey or you might be uh you know a monkey ninja coder whatever it might be or you might be like member of technical staff or you know staff programmer three um and it turns out there are easily hundreds and sometimes thousands of variations on job titles and same for skills and University majors that all mean the same thing so how do we deal with that um we considered you know the traditional approach is hey the query expansion our customers are doing the work for us they'll try something maybe we'll try another search try you know do Boolean queries uh unfortunately because our product wasn't yet good enough to attract a lot of customers we didn't really have the volume to make that happen and even then uh because as I alluded there's this you know so much freaking variation that query expansion wasn't even going to take you that far so what do we do uh we actually focused initially on the health care industry so we thought you know what let's try to get a grip on this problem let's see if we can just analyze the data and make sense of it manually so we've picked healthcare because we thought Hey where's LinkedIn and everybody else in the tech industry not looking and everyone's tech industry is typically looking at Tech recruiting so that hey what about Healthcare recruiting that's the fastest growing industry in the country they can't hire uh and actually there's not that many Healthcare professionals on LinkedIn who were who we're competing with so and then specifically we focused on uh on nurses in fact unfortunately this is not what we were staring at all day long what we were staring at all day long was more like this so we had too many nurses problem and not the good kind we usually again this is you know just a brand you know sampling of some of the nursing job Towers we were dealing with but this is not what we want to present to our customers ideally our customers can mental model it looks something more like this nice structured hierarchy pretty much every major uh job title in nursing will fit into one of these boxes and uh and and then that's actually what we did we went ahead and manually spent literally like three months looking at tens of thousands of nursing related job titles and mapped them into this hierarchy quite laborious didn't scale but we just wanted to know if it was like if there's any benefit to it and it was immediately like our health care sales took off like all of a sudden the product was useful because we had some really big synonyms table uh but that wasn't scalable because if we wanted to do that across all just U.S job titles of which there were millions of distinct job titles like after some basic text normalization uh that would take us hundreds of manures like if we just extrapolate out what it took us to do this even if we got more efficient built some tools maybe we get it I don't know 800 man years it just wasn't gonna fly so uh uh we had to come up with an approach that was going to work so we had these job titles uh there's you know a handful of nursing ones and we tried different ways of using machine learning and like labeling them and seeing if we can come up with algorithms that can then kind of infer the synonyms table so we don't have to do quite so much work and we hit a lot of dead ends but one of the insights we had was that well even though yes all these job titles are very different and uh and oftentimes they don't have enough terms in common for all them to be surface simultaneously they do have engrams in common so if you look at this you know we see like unigrams and bigrams here that are you know Common across at least some of these job titles and that was one of the key insights well the key Insight was actually doing the analysis on this and finding that unlike job titles where there's a really fat tail their uh the engrams appear very uh appear across different job titles very frequently as a very sharp drop off I mean it's a cliff practically like you go after the first uh you know what's called a few hundred uh engrams in a given job category and after that the you know engrams don't reappear very often anymore which was this amazing Insight because that meant that if we could somehow figure out the semantics of all these engrams and then use them to infer the semantics of the job titles uh we could very quickly you know come up with a system that you know produces this which is exactly what we want uh and by the way this isn't used unique to job titles there's lots of categories of entities that have this pattern where the engrams repeat over and over again like I said we've applied the same technique to well I'll get into it but to University majors degrees uh skills and we're looking at a whole bunch of other things obviously we're focused on professional data because that's our business I'll leave it as an exercise to the rest of you to figure out what are other domains where you could apply some of the approaches that we'll be talking about here um so specifically for job titles what do we have to do so let's say this is a job title what we want to do is uh we're not just built we don't want to just build synonym tables because that's uh um that's not enough especially if you want a high quality search you want to do some sophisticated analysis on them you need more meta information so this is just a subset of the uh the cat label types that we care about but we want we're looking for job title we're looking for specializations we're looking for certifications and these are this is what I'll use for examples today but there's actually a couple others as well that we consider uh so let's let's talk about this the ICU registered nurse what is this uh well the job function here it's nursing and uh specialization ICU Intensive Care Unit and uh the certification is registered this person is registered in some state to you know I mean you know work as a nurse it's great so so what does that mean well there's first thing we want to do is actually break this term out into its engrams before we can do the analysis so there's three unigrams here ICU registered and nurse there are two bigrams I see registered registered nurse and and one trigram great now what we're going to do is instead of trying to apply the labels to the entire job title yeah we're not going to go top down and actually we're going to go bottom up so we are going to start and only label the unigrams because oftentimes you just by looking at it you can figure out what the semantics are for if you're focused on a domain so in this case you know ICU means you know specializations I see you registered registered nurses nursing this is a little bit of a trivial example it gets more complex than this but this will be simple easy enough to follow along um but so what we can do is actually manually label these engrams because I said earlier this far uh the engrams don't have this fat tail so you could probably you could actually label you know 1 1 100th or maybe one-tenth as many engrams as you would have to job titles in order to get good coverage across the entire space so if we manually go through it and just label these unigrams then at least in this example we don't even need to go bother labeling the bygrounds because you know what we can do is we look at the bigrams we just propagate those labels up because we see well these unigrams appear Within These bigrams so just copy these those over and just assume those must be the labels for those bigrams and then we do the same thing for the trigrams as well what this means is we can literally shave off roughly one and a half orders of magnitude of labeling work to achieve the same thing now it's not quite that simple because Let's uh let's go through another example here let's assume you're looking at ICU RN well actually this is kind of still along the same lines of thinking uh so with ICU RN we know that okay well we know we know ICU we just labeled that we don't know RN so what do we do well let's go ahead and introduce another unigram let's go ahead and apply the labels now RN just stands for registered nurse it's pretty trivial so we we add the job function we add the certification labels and then we just propagate those over great sounds Dandy it looks like we're going to be uh we're in business we're not doing too much work but let's look at another example and see how that plays out let's say nurse assistant that's interesting is this person really a nurse uh so the labels would say yes uh but no we'd be wrong now a nurse assistant is actually a different category of person this is someone known as a nurse aid now a nurse aide is not a nurse they're not ready you know they may have some kind of certification but they're certainly not allowed to do things that nurse is allowed to do um so it is a very very different kind of job title so what we need to do is actually instead of just labeling the unigrams we have to go no no we have to actually label this entire bigram so now we have a bigram label uh known as nurse aid and uh and so what we actually do is we actually go all the way to the trigram level sometimes for job titles and for things like University major to go all the way to the quadrograms but because what happens is semantics can change as you add more words in terms of the algorithm we used I'm not going to go in too much detail but it's it's very much a greedy algorithm so the way that works is you start with a job title in this case let's call it intensive care nurse BSN that sounds fancy what does it mean um well first do we try to match the trigrams are there any trigrams that match in this particular case no there aren't okay next we try for the bigrams and then the unigrows so what do we end up with well with intensive care nurse we know that intensive care actually means you know it's maps to ICU this is a label that we've defined there's many uh many different engrams that might have this particular label hence when I say we're mapping things to a common semantic space it's these labels that are the common semantic space and that's where we actually understand what what things mean and then we can label nurse and then interestingly enough we can label BSN now BSN you knowing nothing but Healthcare you may not know what it means it's actually uh I think a Bachelors of Science in nursing and that is a prerequisite in order to become a registered nurse so if someone has it it basically means they're a registered nurse or they're about to be um so that's that's how we would label this title now why trigrams why why do we stop at trigrams for for job titles why do we even go that far why why not buy grams why not quadrograms so let's let's look at another example to explore that question a little further so let's say you have a title assistant director Now assistant and director are pretty different things if you look at just the unigrams they're radically different directors you know typically a pretty senior person within an organization assistant typically is not but you take it together and it actually is someone who's maybe like half a step or a step below whoever the director is it's actually someone who's very senior so in that sense because we have these semantic modifiers where you know one word one you know single unigram by itself means one thing combine it with another unigram now I mean something else same you can have add one more a third it could mean something else entirely let's see what that looks like assistant to the director see we just we just added another another unigram change the meaning again so this kind of goes to why we need to have um yeah that too is tricky right we need to have trigrams for job titles and but there's other issues here right two and a lot of uh like if you're doing information retrieval it's typically considered a stop word right like well wait a minute and in fact there are a lot of stop words in job titles you if you're if you just include everything it's going to be a mess so it requires a little bit of domain expertise here and we've had to analyze it for every new domain that we've been applying this this technique and understand well what stop words do we have to keep because they're semantic modifiers and which ones can we throw away so things like uh in or of turns out are very extremely common in job titles those we can toss out but two and four like the example we just saw we had to we have to keep those around we also strip out almost pretty much all all characters um all special characters as well and however for universities University data so education people often have very very very long degrees right when they put it on their resume or their profile because often what they'll say is you know I have a bachelor's of science and computer science and minor in business yeah emphasis and Entrepreneurship anyway there's there could be easily like 8 10 15 word uh degrees our Majors whatever as people what people will list and they're often not broken out and it's not clear where one ends and the next one begins um so if we want to use this approach for Majors what we found is we actually do have to go to quad grams so we make that call on on a Case by case basis um and uh so let me give you an example of like what the actual output looks like because what we've done is we built a bunch of tooling around this I don't want to make this process very very quick and efficient so in the end wind up with something like this where you have this nice beautiful graph this happens to be a clinical research job category within the Biotech Industry and we've got a whole bunch of these we call these industry trees we uh we actually built a series of tooling so that we have a data analyst go do a whole bunch of research it's probably 70 of their work is research and then 30 is using some of the tooling that we built to Define these industry trees and all the labels and then the system then figures out how to how do we get job titles or skills or Majors to map within structures like these here's a couple more examples here's uh Supply Chain management here's uh here's marketing but what do we do I mean this is like a boatload of work um and if we were just using Excel spreadsheets this would be incredibly inefficient so so how do we do it so what we've done is uh you know created a tool to help Simon fly uh called manifold let me I'm gonna do a live demo of that tool hopefully everything goes smoothly all right actually give me one second I gotta turn on mirroring so I can see what I'm doing apologies for the task switch okay where is it there we go all right so this is manifold it's got fearsome animals on every load okay that's not an animal uh I don't know we're pulling off some some tumbler feeds somewhere anyway uh let me show what a typical workflow looks like so the researcher will go did analyst rather will go and spend a bunch of research into a particular job title let's say we're doing consumer package Goods let's say marketing and first thing they do is have to figure out they actually have to figure out what the canonical job titles are so we don't figure that out for them they have to bring in a certain amount of domain expertise and say okay well I've looked at a whole bunch of company jobs pages and a bunch of people's professional profiles and figured out well there's something called there's a concept of brand manager and marketing officer and copywriter so I know we're cheating a little bit here because I know it you know a lot of the more fancy approaches will actually figure out what the canonical entities are well um that will eventually add that as well but right now this was uh and the expedient thing to do so we we let them actually input them themselves then another manual task where they can actually search across our database of hundreds of millions of job titles actually in the US it's probably it's just single digit millions and just fine you know very very quickly find all the ones they think are relevant and add them for consideration and I'll explain what that means later but remember we're talking we have to go and actually label a whole bunch of unigrams bigrams and trigrams where do we get those engrams from well this is where we get them from the analysis goes does a bunch of keyword queries gets ideas for job titles that we should consider labeling over labeling is okay because that'll get handled on the back end um well you want to make sure you're not missing too much it's actually okay to miss some things because as I mentioned earlier these engrams appear over and over again in different combinations so you interview Matt Miss 50 or 100 important job titles out of many thousands it's okay odds are you're going to find a whole bunch of engrams that occur in those job titles anyway and you'll get good coverage and then you know the next couple of steps are well first you have to go and label the canonical job titles so this is actually the labeling process yeah product manager is a one of the canonical job titles and you can start uh signal this is what this is the job function this this is the specialization Etc we have nice autocompletes uh we can very very quickly go through a whole bunch of this data uh in that case it's just this one page and then so on and so forth then we go to unigrams and again we could very quickly label a bunch of unigrams label a bunch of bigrams and before we know it we've got all the input data that we need uh one of the things you might notice here is we have uh some of these are red and some of these are gray we've added a concept of negative labels and this is a hint to the algorithm so a negative label might mean that well if if these two terms occur simultaneously it means you're definitely not this thing and that happens because sometimes when we combine you know let's say a title is is three terms three words but we're we don't have that whole trigram label we only have it's two bigrams labeled well in that case those two biographs can have conflicting meetings so then we could have negative labels help override those and once we have this we will create something known as a late that we call a label set where we go bring all of these together and um and uh and actually start doing some of the analysis so let me let me give you an actual example of what this will look like so we've done all this labeling we feed it to the algorithm the algorithm produces you know nice nice results like this so what are we looking at here in green these are the canonical job titles these are the ones we manually defined in Gray these are the ones that the algorithm figured out well actually they they all map to the same canonical and then on the right are are the labels now one thing you might notice is wait a minute all these labels are different they're you in fact for each one of these they're unique relative to the labels on on the canonical and from each other so actually there's a lot more meta information even though these are essentially synonyms we still have captured the subtle differences between them and if we wanted the only certain you know do searches by job functions or queries aggregations by job functions or specializations we can do that however what's missing here is uh is the fact that there's a lot there's actually a lot more going on here so if I expand one of these wait there's even more so what happens is what you see here this is what we call representative job title it means that every job title that's grouped under here has exactly the same combination of labels and this one the one that we're showing to the surface just happens to be the most frequent one that's the one and hence it's it's the representative for the those semantics that that one represents and that's actually really useful so the reason we have this screen even is because this gives us an opportunity for the analyst to very quickly scan through the results because they just did a whole bunch of domain specific research and actually see is this correct or not because one of the things about semantics is the machine is not going to tell you what something means right how do you how do you prove that it's correct it's kind of impossible um so what we've done is we've given our data analyst tools they need so they can using their own human intelligence very quickly browse through and understand is this correct and is there is this or is this a mistake it needs to be adjusted somehow and in fact I don't have access actually through here but what we can what there is is a whole review process where a reviewer can add comments say hey this is not quite right fix this someone directly from the screen can make adjustments and then it'll dynamically reclassify everything and show you hey here's how here's how things change so we could very very quickly iterate uh on this and only occasionally look at well what's actually happening under the hood because the representative job titles capture a lot of the meeting that we're looking for so this uh this is manifold and to date we've classified about what we've done about 20 Industries and we've created well over a hundred of the industry trees that uh we've talked about let me actually show you what those look like hierarchies so so here's the clinical research one I was talking about but anyway we've actually generated over a hundred of these and that represents um a couple thousand canonical job titles which represents tens of thousands of representative job titles which represents uh hundreds of thousands of what we're calling like raw job titles in the wild so we still don't have complete coverage but we actually from a U.S employment standpoint we have good coverage of about yeah 55 to 60 percent of the population in the U.S with the work that we've done so far and instead of it taking hundreds of man years as we had projected initially with this approach we've probably put in about five or six man years worth of analyst time into doing research and adding adding labeling work obviously have several people working in parallel uh to come up with these kinds of results and we've also done the same thing for skills uh University majors University degrees so it's gotten actually quite uh quite sophisticated now we're looking at other applications Beyond just resume style data so that's that's the high level approach that's Simon and manifold and open for questions go for it so um the analysts can choose at what level to apply the labels they start with unigrams and then if the semantics change or the bigrama trigram level they will add additional labels for bigrams and trigrams right so one thing I'm not able to show right now is we have something called a black box validation so that a lot this the screen of like results of canonicals and all the jot towels and maps and we call that white box validation you're looking inside the Box you're seeing what the algorithm produces we have a black box validation process that detects conflicts like that where it's like hey someone just got to sign two different job functions that came up from two different unigrams up to this bike RAM and there's something wrong with that and that'll then it'll direct analysts to take a look and say hey do I need to make an adjustment here or not so the detection of a conflict happen can happens automatically but then actually doing something about it requires analysts to intervene yes yeah which we tried that quite a bit the problem is that what the problem that we found is it's about that we spent a bunch of time doing trying to do an unsupervised approach before we settled on you know pretty heavy manual approach uh and what we found particularly with job titles and also the other categories of data is there's these all these annoying unigrams and bigrams that might that will appear that actually don't have that much meaning but they'll throw off all these algorithms and it was very we couldn't find a way to effectively exclude them or if we did exclude them then there were content other contexts where they did have meaning where excluding the meant we'd actually break something so it was this whack-a-mole problem we weren't able to solve it the quality the quality was okay it certainly wasn't going to give us the level of precision that we have here so if you see this is color coded by job function we could almost not quite do get job function level groupings but not at the actual job title with the unsupervised approach but even the job function level grouping it was the quality was poor enough that we never we never took it to production mm-hmm yeah it's definitely feasible if all we wanted to do is something we built on using this technology was a applicant recommendation system so I'm working at workday right now one of uh workdays workday sells HR Financial software to large companies like thousands of employees hundreds of thousands common problem they have is the these companies will get hundreds of thousands to millions of job applicants every single year they'll only be able to screen Maybe ten percent of them so if you assume qualified candidates are uniformly distributed means ninety percent of their most qualified candidates are Never Getting looked at so what we've built using this technology Simon is a way to analyze the job descriptions analyze the resumes and only recommend the most relevant candidates to the top so you're focused on you know the one out of ten candidates that are most likely to be qualified uh we could and we did build that system or try to build that system using the unsupervised learning approach or just a clustering based on engrams and job titles even like we even did think things would look at co-occurring Job titles well if you're a programmer at one company or software engineer another company maybe that means hey those they're somehow related um we spent and again unfortunately because and even that was problematic because so many people made career transitions that were totally unrelated that that threw a bunch of noise into the data as well that we couldn't resolve but using the Simon based approach we get really high quality recommendations so we're finding our recommendations are generally um significantly better than any recruiter would do and better than the average hiring manager meaning like companies don't need to be doing resume screens anymore assuming they did a diligent job drafting a a job description which is very often not the case I'll go for it sorry awesome this is patented by the way so don't copy it directly hmm right so so that so I guess just to restate the question if we know that you know these two three five different profiles on the web are actually the same person and they're talking about themselves in different ways uh can uh can we use that information to figure out that these different representations of their who they are in their job title actually mean the same thing and maybe maybe even use that to power an unsupervised learning approach um what we found was that it's actually very difficult to match people across different sites so oftentimes they'll have a link so if you know people will have a link from the LinkedIn to their GitHub or from their about dot me page to like everything but there wasn't we didn't we couldn't find enough of this strongly linked information to be able to like actually feed it into an algorithm and make sense of it it actually turned out to be the opposite what we can do is even though people represent themselves in different ways on different sites we can use the fact that we can normalize their professional information in fact we can normalize almost everything we had location normalization and Company normalization um and using that normalized data we can then use people use that information to match people across different sites that's how we're able to do that aggregation that's how we're creating the single identity of record for people so it's actually it's a chicken and egg problem like once you have that you could probably use it to solve this problem at scale the problem is you have to you need it solved before you can do that grouping go for it exactly we do have that problem so when analysts do the labeling they do it in the context of a particular industry meaning exactly the same job titles might get labeled 2 3 20 times and that's okay because then what we do at runtime is when we get information about a user we can because we also have company normalization we can look up the company and look up that company's industry and then select the appropriate set of labels to apply to that person's you know professional information yes that's a great question uh I think there is there's no in this good industry list that's so yeah and there's no industry left that's right for everyone um for our purposes we started with something called the G like gics I forget what it's called these out I think it was created by Goldman Sachs and S P 500 to categorize public corporations and then we extended it to include the whole non-profit in government sphere and and then we condensed it because it was actually this something like around we ended up with around 70 Industries and in practice what we actually work with is closer to 30 to 40. so we had to kind of manually curate it for in terms of and the way we Define Industries was the structure and nature of jobs was different some Industries are actually very similar um but if they're sufficient if there's enough differences we'd go ahead and say know what they're different Industries and we're going to go ahead and do a separate labeling effort for them yes so how do we select the most appropriate labels for a given job title I didn't really go into too much depth on that other than I said well we have this greedy algorithm we do we try to match trigrams first then diagrams then unigrams it's more complex than that because like I said we have the labels uh partitioned by industry so what we actually do is on a per industry basis try the labeling and we you we have a score a scoring function one of the things we realized is for at least for job titles and then we have a slightly different scoring function for every domain but for job titles the job function label is the most important one if there is no job function label it's worthless like what you have a seniority and a certification but you don't know what the person does uh you probably in which case you're probably wrong anyway so we then discard any set of labels that doesn't have a job function and those that do we score them and job function is the most important to have a and then you know having a specializations more important and certification what happens is if you have two sets of labels and one has more more sorry more matched labels than another we'll pick the one with more and better matches also if there's other one if it's otherwise ambiguous like the scores are pretty close then we look at what industry does this person work in sometimes it's identical like development director you're going to get that like perfect matches in multiple Industries so that's when we look at a person's uh company and then try to disambiguate from there so there's kind of a multi-step process a bunch of rules and heuristics and it works out pretty well by the way that we're we're just there's more of a starting point we do have a plan to eventually employ uh probably dynamic programming to optimize the label matching process but we haven't fully worked that out yet yes yeah great question so um we consider and the stop word so we just throw it out and when we do see an and in a job title rather because we throw it out we'll just uh end up labeling both aspects of that job title so if you're a Clinical Director and you know research scientist you will have the you know two different job functions potentially and that's okay if they're within the same industry it's pretty common for people to have multiple roles and that's yeah so that's valid yes no um we haven't we haven't found the need I think what we found is and of a handful of other things we have a pretty short list actually I don't remember it offhand but it's on the order of half a dozen words that are stop words that appear frequently and don't really add semantic meaning they're just more just how humans happen to that phrase language um but Yeah so basically which means actually we are including most stop most things that you would consider a stop word yes that is a great question so how do we deal with seniority keywords like director where depending on the region actually even depending on the industry even depending on the size of the company can mean very different things if you're a director at a startup you may have zero reports if you're a director a public corporation you might have fifty a hundred a thousand reports depending on how they're structured uh or you could be actually leading the whole organization depending on what industry you're in we don't do that very well today um so one of the things because we partitioned things uh all the labels by industry that tends to emerge naturally in the but even then within say within the tech industry yeah we have no way of knowing that a director at Google is much more senior than a director uh you know whatever startup that yeah just got funding yeah yesterday uh no the size employer would be a great signal and that that's something that we've kind of spent on our roadmap to implement and we just haven't yet so but it is a challenge we actually looked at size of employer and it's been quite difficult to even based on that make that work well yes we have looked at them in those cases tokenization can also be a challenge like with German like how do you you have to you know Germany will have like one giant word how do you actually break it up into meaningful engrams fortunately there's some good libraries for that that seem to work well enough I think what we've anecdotally we've done a few experiments and it looks like this approach will work um we're actually trying a different approach though instead of hiring and training a bunch of analysts at the domain specific research in every single language that we want this to work which by the way we're working on this now uh we can we're instead pioneering this translation based approach we're going to hire translators which just translate the job title from let's say German or Spanish to English and then we were coming up with some graph based algorithms that some graphical models that will infer what the engram labels should be so we could use a small number of translations uh and then for the labels of engrams they would use the engrams to then label a much larger set of foreign job titles anyway I can't talk about the technique we're using because we haven't filed a patent yet and lawyers at my company would kill me if I did but I'll probably like you know in a few months I will talk about it questions I think we're way over time all right thank you everyone foreign