sftext.org: Gregor Stewart, Beyond Names

Recording: sftext.org: Gregor Stewart, Beyond Names

so hi I'm Gregor Stuart vp product management for basis technology and we privileged to be able to be a provider of some of the technology that go back we're talking to you about this evening and I'm going to take a little bit further so once you've got the entities but we're talking about building a graph from that one of the key steps to building of graph this to deal with this problem of ambiguity and variety right a number of different ways you can talk about the same entity the number of entities that have very similar names and need to be differentiated how you go about doing that and we have a technology we should call entity resolution which we I guess we developed over the past couple years you know using a grant from the CIA and it works pretty well and I want to kind of walk you through it so this is a very definitely a marketing slide right because you can see that distinguishing these two President Bush's right would be almost impossible right for a machine because there's no there's no real context to tell you that you know how different they are right like so for example the first president Bush you might say okay I guess I could take first president Bush that might be you know an entity and then i might go look you know in my knowledge base and so on and find some some notion of first etc but not usually spoken about in that way but so very definitely a very difficult case to do to do this but the evaluation sets are full of cases like this which are just designed to trip the algorithms up but you know the idea that that what the system is doing is taking these two um same strings and trying to say well one is you know Georgian one is this father right the other thing you can do is when you don't have things in an external knowledge base that you can link things to it can still say oh I've never I've never seen this name before but I think it's the name of something right so it's not just a random word in the sentence I think it's the name of something it's an individual entity and I might see again later right so you want to set that aside so this unknown person under an organization I know what type it is i'm going to set it aside keep some context for later and then if see something that looks like that again in a similar context I'll say ah this is also the same thing right this achieves two ends I think buck was really talking about two different things one was you know core reference resolution just generally like inside the document and across documents right is like when is the same thing being talked about inside the document and across different documents right and linking a cheese essentially achieves the same task right you just say if I can link each individual mention then I've achieved the co reference task right so big blue IBM within the same document obviously talking about the same thing you wouldn't need to know that the grammatical structure that makes that true you just need to link the two of them to the correct thing so one is an alias for the other all right so it can often mean that you can do a task that would require more difficult grammatical understanding using a fairly simple thing by looking at local context and looking at the information the world already so let me say a little bit about how it's actually done so take a knowledge base wiki wiki data Wikipedia is a really good one has lots of it's essentially very rich in context that you can use to disambiguate entities right talks about variations in the labels in the names and also gives you some sense of you know what's normal what's expected you know around the entities you project the information that you get from each of these individual entity specifications if you like into a space and it's not necessarily a vector space it can be a document space you know but it's essentially a representation a compressed representation of the information that's inherent in the knowledge base right and I guess is roughly set out spatially there right so at at right is essentially up with the projection into that into that space you can see these things that are about Korea essentially now this is one of my favorite little articles no not really sure was doing in New York Magazine it's about a leader of North Korea supposedly having his ex-girlfriend murdered because his wife didn't like her and you know kind of thing that happens every day there right and of course you know you've got lots of names that are familiar to you kim jung-eun probably some aren't you on song wall right name in the newspaper Chosun Ilbo and so on right and this is just standard anti extraction on this side and in this our courts actually fairly easy to do because it's nicely capitalized and all those other things that most text that you have to deal with and I don't have wherever what you can see also is that you can see some mentions that are kind of similar just generally right there same sort of string so Kellen song wall-e on Shawn that those are those three are more similar to one another and they are two lots of other things so we put them in these little chains or groups right which we think might we might want to resolve together so we don't necessarily go through and try and resolve every mention we'll just group them together and that gives us a smaller problem we can solve it just for these groups and if they're very similar that doesn't actually lose as much accuracy so we sold you you can see there's all by all by yourself right so let's focus on just four of these chains right and see how they map to the to the space right so bear in mind that these little these little blobs are the labels or the mentions and a little bit the context so the words around them right so highways yonghong wall mentioned right who's mentioned just before who's mentioned just after what words are used those kinds of things right these little blobs contain context as well as these labels right so how do they cost her with the things that we clustered from the knowledge base well you can see that John song walls fits quite nicely on top of an existing entity very close anyway right you know wang jian light music band not likely to make a mistake about that one right there's probably only one band called that and you know but we've got to kind of you know dodgy ones here right so you've got chosen right which is a historic or an archaic name for a part of Korea and it's also the name of the the paper right so it's been incorrectly you know left ambiguous between those two places i'd say incorrectly because we know from the rest of the article it didn't extract Chosun Ilbo correctly right because any ours got some errors in it so it ended up you know thinking it was maybe that or maybe the other even though you read the article you know they made the newspaper right Andrey soldier does know where does there's no entity that's anywhere near her in that space right so that's just because she wasn't in the knowledge base right so there's nothing really for it to match to you but similar to sort of other people in that area so you can at least get some sense of the context right and you could keep that for later to see where it makes sense right so this is what this is what the linking system did with that information it said okay well the nearest thing to counseling wall was this person I couldn't find anything free soldier so she's a ghost essentially choose a new a new entity I think says the system right and that's actually correct to Wang Shan like music band got that right that was real easy and you know screwed up chosun ilbo right so you've got the 22 um opportunities here one for correction which you can do in two ways you could say anytime you see Jose on now it means the newspaper because I'm not dealing with texts that talks about our cake parts of you know of Korea what is it or you could say in this case it was wrong and I'd like you to take the information that you put onto that cluster and put it on to the right one so that the next time is more likely to link it correctly right and in the case of resource you I've got an opportunity now I've seen some context right to go off and link that entity to a new a dummy article right or or something else that will allow it to be disambiguated better later on so I want to talk about this take take one step back and say this this is very general right this isn't just to do with text this could be anything right see these three images here contain you know I'm linked to right Manchester United not going to link to there's no Wikipedia page for guys armed with Manchester United's at you on it right this clearly is if it's referring at all it's referring to Manchester United's right so think about the sense of that text can have things in it that link to things in Wikipedia and images can have things that link it to things in Wikipedia right that's linking you can also do learning as well right you can say this image this flag right before before a certain point was only several years ago right didn't exist you've never seen that before right now you see it in lots of images right and the context that you see the images that you see it in right are similar so you cook guys wearing you know was it headdresses you've got guys with guns you guys driving around in you know tacticals and things like this right so there's a common context there around this particular image component which gives it its meaning right you're kind of looking what is that what should that link to write what does that mean obviously now we know links to is or Isis or something like that right so this is a common technique you can use it doesn't matter whether using you know this algorithm for processing components and images or for processing components of text you're projecting it into a space seeing what matches right if nothing does your assume that your tape making an assumption that oh this is a new thing right you know which I might like glitter so one of the difficulties that bobak talked about was you know doing this for text but on very short texts right because there's very little to go on ner itself is very difficult and as it turns out doing any are and linking at the same time makes it a little bit easier right because you have a very large what is it corpus of things it might be right and so you can check it against that right at the same time you're trying to get the the bit that's the name out of it so essentially you're just walk through the text saying hey is anything up to now you know something in the database anything now now et cetera right and then you can say if I move them around a little bit does that make any sense you just do that you know millions of times for each one right you know and then you end up with a list of candidates which have a score and you can say well I'm not really sure we are they are in the text because when you do that moving around that permuting you know you kind of lose the natural flow of the text so it might end up linking to a particular entity at the text level but you can't actually point to the bits of the text that you know the name is in right so sometimes that's good enough because you can say this tweet mentions these people where is it in the tweet I don't know don't care right it just mentioned that right and the person can find that out so one of the things that we've been working on for a while is um how to do that fast accurate so fast enough that you could take the whole fire hose and you know in spit out what is it very accurate links right which as you said are very useful let me talk a little bit about why they're useful so no Facebook bang you know all these other guys they're all really interested in this why right well because you want to disambiguate these things right so Tiger Woods that's a person right but it might be a tiger in the woods or it might be a wooden tiger right you know and these are important things but not closing themselves because nine times I have 10 someone types Tiger Woods what they really mean is that guy that's an adulterer right but it's really difficult when you're talking about you know things like Syria interactions with friends it seemed really important to this month right these are these are more complex things that would be really useful if you could at least link the things that they you know the the query is talking about right so in some sense they're simple queries which are helped by disambiguation every time you go do a search now you get some hits back from the graph and every time you hover over something in that entity card at the right or click on something you're giving them some information about what these words mean what they should present you next etc etc right so it's kind of inherent feedback huh this just stopped working I guess they turn the power off after a certain point trying to tell us to go cooling down oh it was that's the first time I ever really listened so yeah exactly so what is it what does it do for you right let's because I you know the images are pretty easy right so is what it actually do for you what sorts of things can you do once you have a link to a knowledge base right well notch base is very rich in information that may not be in the text right so for example it doesn't matter how much you do NLP on a text that just mentioned something about the Steve Jobs movie right you're never going to know when Steve Jobs was born from that right you're never going to know when he died from that because the information isn't in the text right however you can bet your boots it's in the knowledge base it's in Wicca data right so if you can link the two things you can say hey what's Steve Jobs birthday back right and you're good to go right another thing that people often ask for and this is sort of special cases right our topics right what's the topic of this text right you could say well it's about these entities but typically that's not you know that's not so satisfying right what you want to do is to say take all the entities right and then look at the concepts that they're linked to right now in a graph right like Ricky date is an implant implicit graph all of these things are represented right so you could say you know Obama is a president right president is part of a US government etc right so if you have lots of US government actors mentioned in a in an article when you link them to the knowledge base and you see what's common right what are they what links them right those are the concepts which are represented in this article right so it's a little bit of abstraction and the good thing about that is you know any unsupervised even supervisors algorithms for topic extraction or what is it topic modeling right give you either collections of words which represent the topics in the document right they're not human labels right there just like collections of words or kind of bad selections from that set of words that are supposed to represent the topics right now the topics in the graph are all human curated right you know they're they're human readable labels right so if you can do that projectile into the graph you know find what's common you can get pretty good concept labels for for documents right and that's just by walking a graph essentially right once you've got a link and then also think about norms right so each entity in a in a graph right has an environment and say you say take five hops right so right through the graph and you do that thousands of times you say no don't hop anymore than five times but do it a thousand times and then say what's common right to that group what's the most likely thing you're going to land on right you know when you do that thousand random walks that are five five hops along right that you might call that the norm of that entity right the kinds of things that get mentioned and the concepts that get mentioned and so on but the North for that entity now if you find entities other than those entities in a document along with this one that you've you've linked well that might be an interesting document right so for example if it's at you know prior to us knowing he was an adulterer for example the only thing you'd ever see Tiger Woods mentioned alongside was you know golf nike boring etc right those are the kinds of things you would see and then you know suddenly the day after right you know he's he's an adulterer excitement right because that's an article you want to read right it's no longer about it's no longer about and what is it our friend tiger woods you know whose dad forced him to play golf and all that kind of thing it's now about in this guy who's in adultery so that our call the first thing that mentioned him said that right that's an article you might want to read to reiterate so that would be an anomalous article by this right so just by walking the graph randomly you can pick up these things like norms so that when you come to look at you know documents through the entities that are linked if you don't find those or you find something new that's outside that set that might be a more interesting document so that's one other thing you might get from just linking these entities right so it's all about enrichment though right so we have we have JVM modules for these things what is it and we have an epi that's coming along the lets you test those things and hopefully um you know by back will be one of the one of the guys that tests it so and if you want to get in touch shares my email I'm not an adulterer so thank you yeah I mean curation and knowledge base is a difficult task right and again you're really trying to try to capture two kinds of information stuff that's going to help you disambiguate stuff is going to help you pick out that entity even though it has entities that are like it and these that have the same sorts of labels right and information that's useful right and and also more more sort of computed stuff right like this stuff that we talked about now you talked also in your presentation about how that the possibilities there just staggering right there's even if you just fully connected the graph there are so many edges right it's crazy so I think the primary difficulty is going to be how to prune it right like what what not to keep and if you think about you know a very natural way of talking about memory human memory is oh it's remembering things right it's storing things but actually few memories all by throwing things away right it's about losing things it's about you know what is it cutting things down or compressing them right so I think what people often do when they build knowledge bases is they may try to be you know maximalist right you know yeah exactly it was like oh I've got all these you know headings for things i'm going to put into every entity and then it turns out you know you better be schema-less because half of it's never gonna get filled in right so that's the that's the kind of underlying intuition I think from our perspective allowing giving customers the power to build these custom knowledge bases and to build linkers you know from them is tricky right because it's like it's like selling you know a car that you can easily kill yourself with right you know so customers can get very dissatisfied right because it does actually demand quite a bit discipline quite a bit effort right but I think the results sort of you know some of the things I've talked about you know can be can be very useful right so that's ultimately the difficulty for us is educating customers as to how to do this work right and a lot of the time you're busy doing other things so it's it's difficult right is that satisfying you know I don't have any strong opinions about graph databases what was it last one that we done experiments was neo4j right I would say that was a fairly uncontroversial recommendation if there was one certainly the easiest one to start with right and I think that the I guess the more what would you call it the trickier question is what he do you know how to query graph databases right you know in a way it's efficient right and it's a bit like I remember was talking to shy you're the elastic search finder you know a couple months ago and he was saying we're just not we're just not building things that allow people to do things that are slow right in a bit like Swift right the design of the language is such that you really can't do things that make it slow down right that's the idea right and I think that graph databases you know give you all this power right and give you the Paris do things are really slow right so ultimately that's one of the one of the tricky parts right is to learn how to do efficient queries I think we were talking earlier when the back and I were discussing this about flattening a lot of the information that you would get from a query so do create a flame you know build something like that norm and then store it in an intermediate like a cache or you know or the knowledge base itself for easy retrieval right so i think the which not which graph database i'm agnostic what is it the use of them is very important and typically this offline use and keeping in a flat form is what makes suffer performing right okay all right any others cool you