Text By the Bay 2015: James Earl Douglas, Discovering Knowledge in Linked Data
Recording: Text By the Bay 2015: James Earl Douglas, Discovering Knowledge in Linked Data
I'm James Douglas and I work at the Wikipedia Foundation uh and I'm going to talk about linked data and how we can discover knowledge buried in um very simple facts or statements uh before I jump into this how many uh in this audience are familiar with semantic web okay so about half cool so we're going to be building on a lot of uh semantic web type uh topics um unfortunately in the next uh 15 or so minutes we won't be able to dive too deeply into them but uh feel free to grab me afterward and and I can wax forever about them uh so the approach that we're going to follow for uh our adventure and knowledge Discovery is we want to look at uh uh ways that we can take databases of very simple facts uh or statements uh and explore those facts and connect them in different ways um basically discover relationships between them uh and then use all that information put it together and build up more of a complex web of understanding about some topic that we happen to be interested in uh so for example uh consider these three individual facts uh about Richard fan so one fact says Richard fan was born in Queens and a separate fact says Queens is a burrow of New York City and another fact says that uh New York city is in the United States uh and actually I guess it would be more fair to call these statements uh because I'm providing no proof that these are facts um but I will use the two interchangeably please pardon uh the semantic uh indistinction so anyway we have three facts here and uh since these are simple it's very easy to look at these and see that there's a connection um and so we can do that we can start to connect these things together so knowing that Queens is a burrow in New York City and knowing that New York city is in the United States we can infer that well Queens is probably in the United States uh so there's uh there's inherent knowledge Behind These two facts that wasn't necessarily encoded D explicitly in uh their individual content um and then from there we can take the next step and synthesize an even bigger picture so we we stated in our one of our facts that Richard feeman was born in Queens and in our previous slide we inferred that Queens is in the United States so again we can make a connection and uh infer that uh Richard fman was born in the United States and even more we can connect all of this information and say uh more complexly that Richard fan was born in Queens which is a burough of New York City uh which is in the United States um and so that's a very simple example uh in the in practice in the sort of broader uh um ecosystem the way to get there uh at least what I'm going to be talking about is by building on the foundations of semantic web uh and for those of you who AR familiar semantic web is uh is really just the web that we all know and use every day but with a little bit more intention behind structuring the information so that it's understandable uh more by machines than um than just by people so if you think about uh a web page having a link to another web page then that's indicating some type of relationship between them and that's something that's very easy for people to understand oh I go to Google and I search for findan and I get a link to a Wikipedia article so I understand that I can follow that link uh and that there's some type of relationship between that search engine and that article um but encoding that into uh the underlying um information uh actually becomes very useful for U putting it into structures that machines understand uh and so this semantic web is not just a a um uh theoretical concept it's it's actually all over the place uh or at least aspects of it uh you can find all over the place so there are databases that are specifically geared toward uh putting semantics around information and those are things like Wiki B uh Wiki data which uses wik base um dbpedia Freebase and on and on and they're more uh like this uh data.gov is one that the uh that the US government maintains music brains and so forth and then the actual web actually uh a given web page uh is written in HTML generally so there's actually quite a lot of structure behind that that we maybe don't think about because our browsers are interpreting that structure to render the page can you turn me down just a little bit on the PA awesome thanks uh so for example consider uh just a regular plain old HTML list uh this lets us take a an abstract idea like a list of things which we might Express as a flat string um electrons protons and neutrons and allows us to put structure that in this case a web browser would understand so rather than just just three uh or rather than just some sort of structure this piece of data this string uh we can tell the browser hey there's this list it happens to be an unordered list uh and it has three distinct elements so the browser doesn't know what an electron is but it knows that electrons is uh an element in this three item list and that's really all we're talking about when we say structure uh that's not quite the structure that we'll use for the rest of this talk um we're going to be basing everything on what's called triples uh and triples are really just a way to break down a simple statement such as an English sentence like Richard feinan plays the bongo drum into three parts that we can uh individually tag and identify uh those parts are the subject predicate and object of a statement so in this case uh subject is Richard feeman um he's uh the main um sort of noun of this sentence and then object is the thing that he's being related to so that's uh bungo drum um and then the relation between in this statement the relation between Fineman and bongod drum is this predicate plays thee uh so with these three pieces of information we can re we can see that there's uh kind of like our hyperlink example before there's this relation between these two entities uh and we can understand what that relation is uh so in semantic web there's uh the resource description framework or rdf um which is this enormous e system of uh standards which you will find you drown quickly in once you start uncovering these things um but within those standards are a way to uh write this same kind of triple in a standardized way that tools built on these standards can understand so uh in rdf uh these three uh Uris are the exact same sentence that we just said in English so Richard fman plays the bongo drum uh that says Richard finman plays the bongo drum but in rdf and so what it says is there's a subject which is this uh long URI which Maps somewhere in some database in this case it's Wiki data uh to an entity which is Richard Fineman uh and then similarly we have the relation which maps to some entity in a database and bngo drum which has an ID uh and maps to another entity in the database um and so this is really nice in terms of being able to unambiguously I identify and reference information uh at least if you're a computer but as a person no one's going to look at that and read any English sentence uh you're going to have to pull up a database and do a bunch of querying to sort of infer what it means so triples uh while they're powerful they have some problems um they're because they only capture statements uh a given triple if if you can dig underneath the covers and sort of get to the entities and predicates uh and objects that are uh implicated it's very easy to understand what a statement says but as I said we're people we not computers so looking at all these Uris is basically meaningless so it actually makes this thing kind of unapproachable to uh most people not only that but in rdf there's an obscene amount of Standards um they're all very interesting and really cool and have uh very uh useful specifics um but as I say it makes it kind of unapproachable if you say hey I want to learn semantic web or I want to build a tool that uses rdf then suddenly you have to say Okay I'm going to read the rdf spec and then the rdf spec Forks into all these other specs and then you end up in this nightmare of of trying to learn specs until your eyes bleed and then you give up or at least that's what I did and so as evidenced by the fact that this doesn't even fit on a slide uh here's sort of a hypothetical question uh what does this say uh so we're actually looking looking at four uh rdf statements four triples and these are real triples they this actually says something um but as you can tell you can't read it um and it turns out this says that Richard fman works for Cornell so what's the deal why is such a simple idea Richard fman working for Cornell so noisy well it comes back to statements statements are inherently simple uh and it's it's difficult to um it's difficult who use them in a way that expresses complex information without having kind of an combinatorial explosion of statements um and we'll see a little bit more um Hands-On what I mean so there are some ways of of alleviating some of this uh turtle is yet another standard uh that lets us uh Define these aliases so we can pull out common prefixes uh like URI prefixes in this case uh but it doesn't get us around the readability problem I mean yeah there's less noise on the page but I still don't know that that says anything about Richard fan uh and then Sparkle helps as well Sparkle is yet another standard uh in this case it's a a language for uh performing queries on rdf stores so this query would give us if we ran this on some rdf store that had the right data this query would give us the answer uh to where Richard finman works and in the interest of time I won't dive too deeply into this but we have four lines here uh and these will correspond to those four triples that we had basic this query will cause those four uh statements to be implicated in our search and then we'll connect uh one end of our search which is fman to the other end of our search which is where he worked and then one more thing that sparkle uh gives us is the ability to collapse um this sort of path traversal uh into a much cleaner format but again this is like really daunting uh this is something that I continue even though I work with this every day this is something that I have to look up all the time uh and remind myself oh what was that syntax what was the prefix Etc uh so the the last little bit uh that I tried to do to clean this up and again it doesn't solve the problem completely but helps a little bit is to take Sparkle and extend it uh in a way that makes uh the language more specific to my particular domain so for example if I create a namespace and then put in a whole bunch of techs that I'm likely to use like Fineman uh I work I do searches with femen a lot so I want that to be a specific keyword in my namespace and and so on then we can start to make queries that are a little bit more readable so in this case we want to know who's the employer uh of the subject Fineman so now we we actually have an English uh string that we can read uh and then the path from finan to that employer goes through these two properties uh where the first property is who's his employer and the second property is from that abstract entity that's identified by some really long URI what is another statement that indicates some English label uh identifying that so I have a couple of example queries and I uh unfortunately we started late so I I can't uh dive too much into these but uh we can start to ask things with sparkle uh and in this case this is a query that says um what happened in this uh in history on this day um and if if you're if you're used to reading Sparkle you might find some interesting um oversights perhaps in uh in the way this is written um specifically uh we're we're doing some regx matching on uh date values which is kind of a drag because Sparkle actually provides lots of interesting date functions and this uh this is actually a hint that our data might not be as as well labeled or our ontology might not be as complete as we want and that's actually the case um so this is already way out of date uh I I just wrote this a few days ago but we have been working very hard on uh improving our rdf representation of uh Wiki data so uh already today we wouldn't have to use this rega and substring matching and so if we run this query on wiky data we get a whole bunch of results uh here's a small sample um you'll also notice that I have the names of entities and I have dates but I don't have any information about what the event itself was so for example Mexican-American war did something uh in 1846 um maybe it started in in uh April 4th uh sorry April 24th 1846 maybe it ended we don't really know and again that comes back to how we structure our query how much information is even available for us to ask uh and so forth and as I said before even this is out of date uh and we could find out exactly what happened happened related to the Mexican ameran war oh looks like I'm out of time so let me just jump through the last two slides um this is the one I'm most excited about uh for this talk um so we when we started we talked about connecting fan to an employer or connecting uh fan to being born in the US through these multiple levels of indirection uh and this is kind of that um exploded to a a more impressive level so this says uh starting from Richard fman Traverse our graph of knowledge to find out what his field was uh who uh I'm sorry who he was employed by um who other people uh were that were also employed by that employer What fields they studied and then tell me bring that all together and tell me who worked with Richard fman and what were their fields of study and again running this on Wiki data we get some results and I'm just going to fly through these um so the the point of all of this is that uh semantic web is uh super daunting and uh there's a lot to learn but there I think it's worth the trouble I think there's um being able to uh Traverse these sort of graphs of of data not really graphs of knowledge and then build up Knowledge from synthesizing those um pieces of information is super powerful we can learn things we didn't we didn't anticipate we can discover relevances between things that we didn't know were there like oh Richard Fineman is connected to Oppenheimer maybe I didn't know that that's cool um and now I'm over so until they pull me off with a hook uh you can get involved in this um if you want to just uh improve the data that's available you can go to wik data uh and it's it's built on Wikipedia uh sorry it's built on media Wiki which Powers Wikipedia and it also feeds um Wikipedia uh the query service that we're building is built on Blaze graph so there's some URLs for those two um and then there's lots of further reading uh which I'll skip over and thank you very much um I think since I'm over um I'll take questions sort of uh out of band and let the next presenter set up unless you let me stand up here and run my mouth more I was told warn you I warned you so oh okay warning acknowledged any questions so all those natural language question that you andery statement those already template the question is how do I take this question that I want to ask and come up with this ridiculous looking Sparkle query uh and there there's this whole um pattern of exploration of a graph so all of this information is stored in basically a graph database and the way I came up with this query was not from a template uh I started from a known point I started from Fineman and then I essentially asked the graph what do I know about him so show me all of the links around Richard fan so it's like going to a web page and just looking at all the hyperlinks and clicking on them all and seeing what comes back and pruning the ones that you don't care about so here and in the first line I say hey who was Richard fan employed by and whoever that was uh who else was employed by that employer uh and of of those people what were they occupied as and so I basically just sort of explore this graph all Manas you anal that question oh it's it's all manual so there's no uh um we don't have yet any natural language processing on that question to infer this query this is all strictly manual so just want Rel the wiki data query service is built on Blaze graph yeah so Blaze graph is a triple store and graph database and not Rel toe this how do you compare with some M so there are a lot of uh databases that looked at uh when we were researching this and and we went through basically a huge trade study to to compare the pros and cons in our case we had specific requirements like we wanted uh cross domain failover we wanted certain levels of availability we knew we had certain data sizes that we needed to deal with and so on so with all those criteria we ended up actually we ended up on another one called Titan um who was in Acquired and their whole team was effectively dissolved so we went back to the drawing board and and came back to blaze graph um and so far it's been working pretty well um if you go to the actually if you go to blazegraph tocom uh they have a press release and and they mention us and and talk about all of the other uh databases that didn't get selected they're all really good um it's just this would happen to be the best for our particular needs you going to natural language um would be super awesome um but it's way above my area of expertise I think I have to get off now so um I'll I'll be around feel free to grab me thanks