SF Text: Jay Wacker, Q&A with Alexy Khrabrov @Groupon

Recording: SF Text: Jay Wacker, Q&A with Alexy Khrabrov @Groupon

thank you hello everybody I'm Alexi krabrov the organizer of SF text and here we are on location at Groupon and today we have a meetup on ontologies which are very important any kind of organization of text and here was we have Jay Wacker Anthology architect from Cora hi Jay hello it's great to have you thank you very much for having me uh maybe first you can tell us a little bit about what are the ontologies and uh why corn is this anthologist well so quora since his Inception has had a very extensive topic base and the ontology forms the organization and understanding of the these topics this is used uh very critically at the heart of uh the whole quora question and answer system used in many different facets and so I'm the person in charge of the uh the curation of the topics the structure the the philosophy and theory behind the structure and also the systems and how we're going to be building it out going into the future uh uh we talked before and I know that you are also a professor of physics at Stanford University and I used to be physicists as well in my previous life so I wonder how does one move from physics into ontologies and how what is your interest in the anthologies well I've always loved structure and as a physicist that's what you're sort of brought up to understand and to um to think in terms of what are the different structures of uh nature of of laws as a theoretical part of particle physicist I studied the laws of nature now um there's a lot of philosophy in that and you know that a lot of that carries over into the structure of essentially ideas and Concepts um and so it's uh was a very natural transition and then on top of it I was a long time core user a lover of it and knew the team very well um and so it was just a very natural transition that uh went off pretty seamlessly uh so that's great to hear because folks say that core Scientology is one of the best analogous in in industry and we all know that in almost any website you go to it has some kind of analogy it's usually you know it has sections or if it's Amazon it has different products and so forth and a lot of them are kind of at whole creations and people just you know sit in the room and they decide these are going to be the the layers so you have some kind of drop down menus on sites and you know have different categories of Industries right and it's it all seems very at hawkish so you know as a scientist I was always wondering you know is there like a uh you know is there any good scientific way so so from your experience do you think that you know uh there is kind of uh the right algorithmic way to create an anthology or is it you know is it is it possible to have different kind of antologies in different spaces and it's kind of not well defined problem so ontology from my uh reading of both the philosophy and the information science literature is that it's not a it's not a finished science however um there are structures that you can impose upon different sectors of uh of the uh knowledge space and so in quora you know we aim to cover all of human knowledge so it's a a globally uh a global ontology similar to say uh you know dbpd of wikipedias uh um derived ontology um and so from having you know structures in place we can actually uh um make the ontology predictable and understandable and roughly uh you know adhere to sort of the the the intuitive Notions that you would hope for so quora's ontology historically has been Community Driven and community-based um and so one of the challenges there is that we need to like make the ontology both complete and predictable while maintaining the community uh driven uh aspect of it the the organic nature of it so actually a lot of my work is you know interfacing with the community making sure that what we're how we're going forward is maintaining the core Anessa of quora no for a topic system interesting so um so people basically are using the Anthology to to find different things so personally so I'm often just you know starting to search on questions and I kind of find right so I wonder if you so if when you look at user Behavior Uh how many users do you see actually are descending down the Anthology how many people are browsing through the Anthology um right now we actually are don't have a great browse product so that's uh that's something that we're you know working on and uh um for the future so I don't um so we don't actually break that out right now um I you know it's mostly for the people who actually just um interested in different areas of knowledge and one of my my uh jobs is to make sure that these different areas of knowledge sort of have similar structures similar philosophy similar type systems so that uh we can understand them and use them in an intelligent manner um inside inside the Predator quora so uh this is Rousey you would kind of use them for for relating uh various questions to each other yep right um interesting so so for instance I know Quark has a lot of uh knowledgeable startups right like a lot of Founders are there and people kind of it's kind of an interesting Community where you can ask you know direct question of you know Michael Dell and he'll tell you you know how he did some you know how he found the Dell so uh and so so uh and so you have this industry knowledge there right so uh do you do you you know try to kind of make it these connections more obvious for instance can can somebody who looks at you know uh uh social networking startups be related to you know somebody looking at uh Facebook questions they will discover similar Twitter questions do you do you want to enable this so it's just you know side effect of um I think that uh having things that are actually quite that are similar ontologically have um should be uh should have similar um structures both in terms of uh the top the subtopics under them and also uh um potentially the the questions um and uh you know that's part of the predictability reliability making sure that you know we we have the knowledge space uh Phil you know um covered in a in a reasonably complete way uh so uh and uh you're going to talk about it later you know about all the technical um uh considerations but I'm wondering so what are the kind of touch points where do the users kind of interact with Anthology where do can wherever they become aware of it and so the ontology is mainly uh they mainly come into contact with the topics you know most you know we try not to make users have a mind meld with the entire topic system and the ontology behind it to be able to use quora in an effective way so um we we try to make sure that uh we use the Intel the ontology to make our system more intuitive and intelligence that we need to require less interaction from people who use Square whether it's asking a question or getting answers or Distributing uh the uh writing to uh people who are interested but you have some crowdsourcing tools right like you have tools for people to correct uh oh yes absolutely everything is available in product uh for people to modify create new topics um set the types you know to set what type of topic it is um what the uh what where the topic falls in the taxonomy of topics and so we have all that available for the community to use and and that that's you know historically been how we've done done the ontology and it's you know led to I think a very diverse set of uh of topics that really cover a lot of human knowledge and so now what we want to do is we want to make sure it's sort of uniform which you know is more which is beyond what sort of uh users are easily uh you know going to spend time you know making every possible uh you know make sure that every possible subtopic is uh um is in every parallel topic that's that's something that we don't want people spending their time on right and probably they don't have the full visibility right so they probably will help you with completeness but not necessarily uniformity exactly and and right now we're working for uniformity making sure the structure is regular setting up the type system so that we can actually act uh intelligently more intelligently about the topics so uh do you see uh people kind of arguing or disagreeing about uh topics right like so somebody can go in a lot add a lot of detailed topics somebody would rather have a hierarchical structure do you see any of these kind of interactions um I'll we do see that occasionally and you know we you know the the biggest issue usually is that uh uh topics are being used in multiple different the same topics being used in multiple different ways and so um a lot of this actually requires disambiguation because it becomes very confusing if you're using a topic in multiple different ways um to set in any one portion of the the taxonomy the uh the the topic graph um so if you break it apart into the right uh sub components um so that you're actually speaking about one entity it actually usually becomes pretty uh clear uh what the right place is just by the just going back to the definitions of what you mean by that concept uh do you have any uh things like Wikipedia for disambiguating a topic you know um we use parentheticals just of uh um we're building out a type system um but uh we use parentheticals at the end of the topics to clarify what the uh scope is so for say Captain America there's lots of things you can mean by Captain America whether it's the character whether it's perhaps a movie whether it's the creative franchise whether it's the comics or various other aspects that you can imagine the Captain America figurine uh you know that the toy that goes that comes out right they all are different concepts but uh you know you need to specify which one you're actually speaking about and if you just use the the uh the word the topic Captain America um you know children talking about uh you know what the best uh Captain America figurine and is will be you know intermingled with uh people talking about you know detailed aspects of the comics right from you know the 19 in the 70s right all right right so okay great uh so obviously you you put a lot of work into the Scientology and users uh are there any ways for you know the word at large to use it uh you know like can it be if it's a truly Universal Anthology right you can let others map things onto it is it is it possible um that is not possible right now um and there are no immediate plans to do that uh it's just um the return on investment just it's not quite there um at this point in the future uh you know it's it's possible but at this point uh um we're not planning on doing it in the near term yeah I found this that's interesting because uh I was at Cloud at some point before and you know cloud has an API which kind of you know lets you classify uh influencers by by topics so it has an anthology and uh uh Folks at Prismatic now they actually make their a topic graph available right and so they uh I think they see value basically so I'm wondering you know we have several graphs like this right it's obviously Prismatic people you know they blog about it how they created the interest graph uh put all the work into that because it's important to surface news um Wikipedia has its own Anthology uh and um obviously you know uh we have several public graphs like this so do you think in the future uh some kind of convergence of these systems you know core and Wikipedia and and others I don't necessarily uh see that uh like I I'm more familiar with uh uh wikipedias the dbpd ontology and the entities that are associated with they're actually quite different between quora and Wikipedia I mean there's a level of notability in terms of what's even acceptable for Wikipedia articles um and it's really important to uh you know like how you organize the structure is different because like the you know core is greater getting at experiential knowledge so how you organize where you can wherever you might find experiential knowledges uh different than where where it would be valuable in Wikipedia so um these are you know different systems uh you know the high level ontology you might um you might think are you know you can come down to similar uh structures but once you get down into sort of the the bulk of the ontology where you're getting into the the large number of topics um I think there will be significant differences just based upon the use case interesting so basically so it looks like the the usage defines still the kind of analogy so so Wikipedia is an encyclopedia and kind of definition of facts right and so and quarius Q a right and basically so it seems that you know the Anthology for kind of questions people may ask right will be guided by this use case of questions right and and so maybe the Prismatic interest graph will be different because it's Guided by kind of news right so this is this is interesting so they will we'll see different ontologies based on this I wonder if we can you know we can compare computer scientists who can compare trees right and graphs so I wonder and what will we will find right if you whom you know compare these graphs formally in some way right for instance okay can you talk about like the depth or maybe the branching Factor because there's this Rock Rule that they have no humans have to have five plus many plus minus two Branch branches right that's how the kind of easier remember do you have any rules like that in neurontology um so we so first we have a graph we are not a tree so you have multiple parents um we think that's actually very important for defining Concepts you know many concepts are you know intersections of uh I ideas so for instance a perfectly acceptable topic are Italian restaurants in North Beach yes um and that's you know the intersection of Italian restaurants and uh North beaches part of San Francisco yes and then Italian restaurants in and of itself right there's an intersection of Italian food and Cuisine with restaurants and so naturally you'd want to actually uh take the topics up in both directions in terms of uh the the the rough rules we're hoping that for um most of the parent-child relationships we're aiming for say 10 to 20 children maybe 5 in some cases probably not more than 20 because it just becomes very difficult to navigate beneath that with the exception of is the relationships and is a kind of relationships where you can have uh you know arbitrarily many uh types of of examples of a concept and we do encode that via parent-child relationships in a special way then in terms of parentage you know we really hope to keep the parentage down to one two three um in most cases uh um not have lots of Concepts because you really run into a sort of global problems if you have too many parents yes and so while we were very conscious of that so um yeah I think we're planning on having a strict rule of six and aiming for one two three for most uh subjects so the degree uh you prefer is between 10 and 20 10 20 children under a parent um but yeah possibly uh what some some will be five it depends a lot on the uh on the actual uh um topic and how much you um you know it's essentially a trade-off between breadth versus depth um and so do you want to you know keep on navigating down versus uh with and so you know how many levels can you actually keep track of in your head and naturally find uh so you know we're we you know we're dealing with the the trade-off of that uh I wonder can you uh measure this right as experimental scientists like we can actually lose some of this and I was always wondering you know in trade-offs like this you know breath versus depth you can actually make an AB experiment and you can you know structure this way way versus that way I guys do anything like that um we are not doing a b testing at this point on on the uh structure of the ontology um it's something that we could potentially do um you know we I think that we are not planning on having this be massively visible to uh um the average person who uses quora so it's a lot more what actually functions best in terms of getting uh um the content on warrant to the people who are interested in it yes um and so it's really the bottom line as opposed to other sort of uh metrics in terms in terms of like people understanding what the ontology is versus uh what we really care about which is having people you know write great questions questions and answers on subjects right but you still can measure the outcome right so so I had actually this idea I wonder what you think about it so we can actually say let's let's let's measure all the interactions of people with Anthology maybe indirectly by kind of seeing what kind of knowledge they navigate right and and so they can they have paths through the Anthology and let's you know say that you know the efficiencies sum of all of these Paths of all the people in in a in a unit of time and then you have your outcome which may be total page views or some Metric right of overall side performance and so basically you can have a function where you have different structure of ontology and then you know it will lead to Total aggregate of all the past travel which may differ based on the orientology and then you have your final uh objective function right so conceivably you can compare efficiency of different Technologies by by just you know looking at all the past people travel together and and summon them up um I would say that that at the same time that maybe near isomorphic to um essentially the the function that you're applying on top of the ontology how you're actually using ontology and taking the different uh different versions of that ontology and different versions of the function acting on the ontology so like how you know how much you Decay as you go up and down the uh uh the graph versus you know how you decide to go on which path and Define uh decide on relevance um they're you know obviously very complicated uh functions there and so like how much that differs between uh um the two is is really unclear to me and you know and so uh you know it seems like uh the better way to go is to for us for my first impression uh would be to uh consider us showing that there's an actual functional difference like an actual material difference between um the functions on the ontology and the uh um and the uh different ontologies whether they're if they're just remnappings of each other then then uh you know if you're actually able to algorithmically change the ontology um to actually be able to do this A B test then um then that's really just a function acting upon the ontology right right so that can actually be just stripped off and put onto the uh into the actual function that's using it yes um and so um all that is uh you know things that are you know part of just a standard way that we use uh the topics in the ontology I mean that's that seems like you know there are some experiments you know we can simply can run on this and so I'm very curious perhaps I'll do some possession so I mean they're not totally equivalent but I think if you do an algorithmic change to it it's um it's they're the functionally similar so if you like manually change the ontology then that's um very different yes and then if you take the ontology and do this transformation yes transformation upon it yes um interesting I think yeah I think I find this topic fascinating so I think there are like a lot of different uh experiments we can potentially do uh to compare this so what is uh your like what is the hardest problems are you facing with the anthological systems right now um you know the uh the hardest problem is uh you know understanding the the actual usage that's been developed over a quora in all the different areas of the ontology and making sure that uh we maintain an authentic horror you know ontology um and then making sure that we're Faithfully representing it as we go forward and not just blowing it away because there's a lot of value in it already and so this is understanding the community doing Community Management as well as uh you know um you know being a dictator at times and saying this is just what makes sense and this doesn't make sense um so that's um those types of uh um challenges are really uh there you know restructuring the entire orientology there's only so much you can do manually if it's been manually created uh you know algorithmically if it's been manually created and so that's so suspect and social system a special set of Technical and social issues around them and maintaining the system culture this is great well thank you very much Jay no thank you we're looking forward to your talk and uh always great to have you ssf text okay thank you very much thanks