SF Text: Stefan Gower, TopicScout @Lithium

Recording: SF Text: Stefan Gower, TopicScout @Lithium

that was a really interesting talk before me I mean because in the talk before it just illustrates that when we say the word topics we can mean very different things so in one case we're looking in clout is looking how you associate topics with users okay very valid very very important here what I'm going to talk about is something different a bit more traditional in the sense of giv a piece of text how do you look at that a human being does and determine what the topic is for that text now I have a a slide deck I have PowerPoint slides which I'm going to get to but typically uh talks on topic classification they will describe a system they'll describe what they did they'll tell you their processes their tools what whatnot generally speaking though you don't see too many much data on the topics themselves so topic Scout is different that uh it is so far gotten to one system of 10,000 topics and each topic is associated with distinct and relevant items for that topic we're talking everything from lung cancer to hedge funds uh to dresses uh to social issues politics you name it it's there theoretical physics whatever and so the idea behind this is that is there a way given a topic to be able to very quickly very cheaply because automation is key to be able to extract those relevant items so that you can later on simulate what a human being would do who who's really able to look at all these lexicons and say ah given this text it's about this topic so I'm going to show some data first and then get into the slides so this is from topic scout.com and it has well we've been drinking beer at uh here and this I'm not sure how much you can see of this these are types of beer this is a topic for types of beer I'm going to go over this very quickly on the for those of you familiar with uh classification on the left side we have shingles which are essentially a sequence of Words which for English have been gone through through a a stemer the Porter stemmer and on the right side which is somewhat unusual in presentations I show where these actual strings were recovered from now if you look at these things you can easily understand that there are a lot of things which are about pypes here and we can go down and see more and we can go to lots of other places on the site currently there are about 4,000 topics and I've shown about 20% of those those topics so let's just go see something for uh wait let's say strength training what we've got here okay posture okay so here a bunch of items about posture uh neck posture correct bad look posture yes it's bad hunch forward I guess I'm no expert in posture but I assume given what I've seen from topics so talking expert that these are relevant to that topic so I could go through more of these but they're currently about 10,000 of these what's important as I get into it is that they can be built very quickly very cheaply very consistently and precision on 4,000 topics of these kinds against 100,000 web pages I'm not talking Wikipedia I'm talking the web as a whole which is pretty crazy was 80% precision and with that oh by the way this is an interactive talk if anyone has questions as I go just feel free raise your hand and uh if the organizers will remind me to repeat the question I would appreciate that so really if you're trying to do topics one of the hard problems of the last 20 years is to actually do topics at scale when someone says scale they can be many different things we'll get what the scale me might mean but let's just say you know to scale you might want at least get to 10,000 topics if not 100,000 topics or more so you know a human being can look at a piece of text and long as they have enough information about it they're really pretty good about figuring what that that text is about and this is the type of topic classification I'm talking sometimes it's called topic identification but for my sense here it's more the same thing it's pretty important just because there's a lot of unstructured text around and there's a lot of applications I'm not going to go through the list of applications for this audience I think most people probably know some of them they're quite a few so what does it mean to classify something well here's a very simple example say you have four topics to choose from you have hedge funds jealousy a beverage tub data mining okay pretty simple you can't here's some data here it's not the full data from that side and it happens to be selling beverage tubs so know what was the topic well it was beverage tub that's the right answer so that's easy but it gets a lot harder because Size Matters dealing with a small number of topics not difficult at all there are lots of techniques can do that but scaling is much more difficult size really matters a lot so whether it's 100,000 thousand 10,000 million and more how much can you scale so what causes some of these problems well a lot of it's fairly common sense you know if you have topics which are really distinct from each other like baseball and lung cancer and salad Dan there's a a large distance between them it's pretty simple to tell them apart but as things get close together it gets a lot harder so as you the numbers increase you tend to have more and more topics and they get closer and closer together and you can have topics that can overlap with others there's all kinds of ways this leads to problems and so a little visual here you know that's pretty packed more packed so as you scale up you know this this stuff starts happening and so how do you deal with it so another aspect of the problem is that many systems use supervised training for topics so they'll take some topics As baseball or hedge funds and they will have some set of training documents maybe 500 documents about baseball or uh cricket equipment or hedge funds or oral cancer or you know birthday candles whatever it is it doesn't matter but you know if you have to pick all those documents out and curate them manually when you start talking about 100,000 topics it's it's a lot of documents secur rate properly and even if you did that by the time you're done the topics may have started changing and may be out of date so it's pretty hard to maintain so automation is really important here so topics keep changing some slowly so quickly so you really need to be able to keep up so automation is really important in this you really can't you really can't depend on on human beings doing very much so also when you're looking at documents uh particularly in the web it's not so bad inside the Enterprise that's pretty bad too but on the web there's just a tremendous amount of noise in documents you see I mean probably don't really need to be told this it's many web pages like they throw the kitchen sink into the web page there's clutter there's advertising I mean I don't want to use any bad language let's just say there's a lot of unnecessary stuff on web pages say the leaste uh it's really difficult to clean out the web pages so some algorithms may depend on things being clean it's really hard to keep things clean I mean there are some available tools to clean uh web pages boiler pipe is one of these uh but I found uh using this on a large number of web pages that it works well in some situations and not really very well in others so I did use it for a while then took it out uh so that's actually a problem as well I mean I assume that Google and yahooo or others have actually some good tools Maybe do this but in terms of Open Source or anything it's it's actually hard to get tools that REM remove the noise properly another problem in in in this at scale is just that you know a particular piece of taex may be segmented topics may change from one piece of text to another so how to determine that flow and when things change is also important things can overlap then scale sometimes it's just in one language that's okay perhaps but in some cases you want maybe something to scale that's going to work across you know multiple languages so that's another issue at scale you also this is not quite a scale issue but it's actually rather important is that when you're thinking of classification is there may be different dimensions to classification you may this subject like what I've shown here whether it's lung cancer or it's a part dresses or cocktail dresses or election laws but there's also things like location there's time there's sentiment okay one nice uh Tool uh it works particularly well in the USA and parts of Western Europe I think is claven which is produced by I think uh I don't want to get their name wrong they're it's quite good claven is an open source tool to help disambiguate locations I'm not currently using it but I think it's it's quite interesting and I I expect to use it in the future so is another issue in classification is what kind of Dimensions you're talking about then if you need text class now how do you tell if it's going to scale now scale may depend on your notion of what scale is I mean people may have different senses some may think scale means the number of topics and 10,000 is fine or 100 or 100,000 or scale by refer how fast it can classify the web pages but I think a lot of scale characteristics here are the same kind of scale character you see in a lot of solutions whether it's in Google or maybe Dynamo from Amazon there's certain systems that scale quite well and I think some of those characteristics are true of all good scale scalable systems so you want to be effective so in the case of text classification you want to be precise you need to be automated because if it not highly automated it's certainly not going to scale can't depend on human resources very much at all which is the next bullet point you know little demand for human resources you've got to keep it really low if it requires one person with a lot of effort to come up with a topic by putting a lot of effort in that's not going to work it has to be maintainable has to have a large number of topics with the large has to be defined by whatever your target is for a large but i' would say it's reasonable thing larger is probably larger than several thousand and probably could be much larger than that multilingual may or may not apply it depends on people's applications but if you need it then it's important uh but in the case of text classification in particular right one of the issues is that if you I've looked at a lot of vendor sites for text classification and typically what they do is they describe their processes their tools they describe their apis you don't see many topics at all which is very strange right if you go to a place which is about something you expect a lot of examples of it probably if they're so good at doing it you'd expect to see thousands and thousands of topics but that's just not what you see so for topic Scout a long time ago I came up with a rule because I happen to be down working in Santa Monica and it seemed that some people wereing trying to get a movie sort of dressed a certain way to get attention and when I was working on on this day in the early days I I came up with a rule and I call it the Hollywood Rule and the Hollywood rule is kind of simple and this is nothing to do with technology but a lot to do with uh I don't know marketing I know a VP of marketing who actually now uses this Rule and it just says if you've got it you flaunt it so here's an example of the Hollywood rule they have it and they're flaunting it so there's also the inversion of the Hollywood rule if you're not flaunting it you don't have it so if you look at lots of text classification websites you will not find lots of topics at all they talk about technology but not the topics and I think it's actually almost the most important thing when you're looking at text classification is to see the fact that you're seeing a lot of actual data that they can produce rather than talking about the apis and tools so they they talk about what they sell they don't show off thousands of topics they're not flaunting it and according to Hollywood rule they don't have it so what has topic Scout done now topic Scout uh is built on its own proprietary s set of algorithms it's really not like anything else out there it doesn't use any kind of no really known techniques for its core work uh it has a Sy one one system of 4,000 topics nearly 4,000 topics about 30 3,750 to be exact uh and it's achieved 80% Precision over 100,000 web pages and I'm not talking about Wikipedia Pages there are lots of researchers and they they like to classify Wikipedia pages but these are pages from all over the web and now these numbers may not sound sound so impressive but if you look around most systems are much smaller I mean they don't do any they don't of this size so topic Scout is showing thousands of topics and if you look at the data uh you'll see you know lots of relevant items for a lot of different topics from Health to shopping uh it has computer science in there and uh there are many other branches which are currently not being shown I have a I have a one system of 10,000 and so that's the data that topic Scout produces now how does it actually work I mean most of it's pretty proprietary but at a very high level uh every topic in the system is associated with one or more queries so if we have a topic for the for The Walking Dead uh you enter a query for The Walking Dead it goes through a search engine it gets out some number of documents from the query result maybe 300 to 500 some of those documents will be set aside for evaluation purposes later on okay to test for precision the rest go into Data Mining that produces the topic lexicon and then eventually evaluation occurs and we get Precision measurements and so there's very little work to create a topic there's not you know you don't have to you know curate uh the sources and the algorithms behind it are good enough even when the data is pretty bad it does very well which is rather important because it's not being curated the entire system has no handpicked documents whatsoever every single thing in that system on that website was done through queries and query results there's nothing handpicked whatsoever uh it is proprietary uh it's based on some ideas and information theory that occurred to me a long time ago and I went experimented with them and hoped that they would work and they produce really good results so I kept at it uh one of the different things here on topic Scout is calculating Precision so in the case let's say even like The Walking Dead example now it could be when you're doing Walking Dead you get a query that the query result for Walking Dead could be perfect and all the documents in there are just about walking dead and that's great in practice query engines you know often produce some things which really are not what you thought they should be so it's not curated so there can be some errors you know in the query results you'll get documents which are completely irrelevant or they really belong to another topic so to actually evaluate the Precision for topic Scout what I've done is analyze actual data and figured out what the bias was and what would occur is that sometimes that sometimes with fair amount of frequency topic Scout would pick a better topic than what was in the actual query set So based on that a there's a bias there so I take the raw Precision there's a corre correction and that's how I actually get the Precision in the system uh I mean this is this is about runtime speed and I mean the rest of this is about learning and data mining and slower processes that either be done in the back eventually they could go near real time in terms of actually doing the classification when you have that many topics you can have performance problems because you're you're dealing with a lot of things so what topic Scout actually does it takes whether it's five or 10,000 or larger number of lexicons and actually converts those all into long numbers okay so by the time it does this conversion there's no strings whatsoever what happens at runtime is a document is converted into a string of Longs and so by the time it's doing the scoring it's dealing with very simple data and it can run really fast and that was important to get it down you know to get high speeds I don't have hard numbers but it's probably in range about 20 to 40 milliseconds per classification uh another aspect of scale is it sometimes asked about is you know you know can can this system you know actually discover you know Topics by itself which essentially automatic taxonomy discovered so it can actually figure out topics on its own it currently does not do that uh that would be great if it could it's a hard problem in itself but it is another aspect of scale it's not something that topic Scout currently does and that is it any questions the open source boiler pipe what the the I only mentioned One op there claven uh I sometimes do it in Injustice because I believe it's spelled with a K but Som I get wrong it's either k l a i k l a v iin or c a v iin it's produced by Baro Technologies on the East Coast it's nicely done yes the last question was what was the open source Tool uh for disambiguating locations by the way so in the the first thing we showing different top top SC were those stem uh viant topics the things on the shingles on the left those are are shingles sometimes people call mams i' I've gone the terminology your question was what when I was showing the topic Lex kind initially on the left hand side what were those things that were stemmed okay and the answer to that is that they're shingles and they happen in a lot of those cases to be essentially you know you know onew or two-word shingles okay but they could be longer too the system actually can support longer shingles right now it's not doing that but it will be generalized for that it Shing topic yes so for for each topic on the left hand column shows all the shingles and then what I did was imagine that you you figured out what the shingle should be right but you're trying to present these things to Ordinary People most people when they look at these shingles they get very confused I get confused sometimes myself frankly so that seemed to be a problem so what the system does when it's finished with everything and this for presentation purposes mainly at this point it goes back over the training data and it looks for for examples you know of strings that converted into those shingles and it starts PA mapping the original Source strings to the shingle so you'll see examples like you know inform might map them to information informative informs informal even a mistake but there it is uh and that's that's what that's what's curing on the right hand side so then browser like far those the those yeah so there's a hierarchy of topics I mean someday it's going to be kind of a modified lattice but currently it's a strict hierarchy and that's right so there's relationships between the topics they have super you know a super topic and a topic has potentially a set of children topics a topic has a lexicon has been built and once it's built the Lexicon is formed as has a set of shingles and each shingle is associated at least with one source string and frequently with more uh so maybe this is part of your proprietary engine but uh where are these lexicons coming from when you given a topic like how is how is that automatic does that require manual curation it's okay completely an example it's completely automatic let me tell you that like the one manual part well two manual Parts actually for the system does not do what is called literature automatic taxonomy building it doesn't create taxonomies automatically so for example if you had something like uh an investment topic and you might put you know retirement advice or something underneath it right someone has to create that then they have to potent associate a query with that typically the query is often the same thing as the name so you don't have to do anything sometimes it's not that's the the human step that's required and it can take longer but a typical example 30 seconds to a minute per topic and the rest is automated after that okay so you're talking about the organizing the topics themselves into a taxonomy yes that that is manual and the lexicons that go with each topic how where was coming okay so I going repeat the qu I think the question the question was okay once you have the topic how do the lexicons occur okay so once a topic exists you know it's in this topic it has one or more queries associated with that okay the highle process is pretty familiar okay so since well not that familiar because most systems they don't don't go by the queries it evaluates the queries it goes to some system in this case it's typically going to Yahoo boss right it's evaluating the queries it gets a bunch of URLs back okay next phase which is the acquisition phase it goes and gets the data right you know does some massaging to the day gets that ready for data mining then it runs it through a map Produce job and then it produces its initial lexicons there's actually other things that happen even after that and finally you get and there's a little method somewhere in the system called finish it does the final finishing of alexicon and then then it's finally done and then it finally goes to its final evaluation phase where each topic then with a hold out usually it's called a holdout the set of documents that were not part of the training are then evaluated to see how well it performs against that does that answer your question I'm not sure if I have yeah there's some pieces so I guess the queries you're talking about are queries that can be run against a search a search engine just a sear a search engine and point it f it could be it could be a search a engine within an Enterprise for that matter the system actually has a little search agent connector framework if so happens mainly it's going through Yahoo boss but it could go through some other one just as easily you could mix and match in point of fact that gives you a collection of documents yes uh well then we get into the algorithms the you their things will be filed and described and I would love to talk about it I really would like to but for reasons are pretty obvious I can't so yeah there there's it's it's quite different though from other other things it doesn't fall into a you know a cat you know typical categories uh but it's probably you know it's it's a multi-stage thing as well but it's worth doing it because it's all about the automation because if you can truly automate it then you can and you can get effective results you scale I'm not sure there was a personal question but I think he's gone so okay so is that the 80% that you're reporting is that uh classifying an arbitrary document and choosing one topic for it well I I want I want the question was kind of precision I'm talking about is that classifying an arbitrary document and then seeing if it gets the right result is that well and is is that is the the problem that you're to assign a single topic to a document and then see if that matches a single label well at the valuation phase that's basically what it's doing I mean very production you might want at multiple topics but evaluation I think the the question is are you trying to get like one result and the answer is in evaluation yes and I I will say that you know it is the case that you know I'm I'm getting documents from a query result I have my holdout set okay okay so at this point I'm not taking completely random documents from the worldwide web right I have gotten documents from a query result whether it's 500 documents on lung cancer put a 100 of them aside for evaluation purposes and though then I'll run I will take those documents and when I'm doing the classification it's not binary I'm considering all the other topics so when I say you know there's 10,000 topics I'm considering all the other 10,000 topics as well and then Precision is measure if I actually get the right result which in this case is lung cancer I see but you you don't test then um yeah that I thinkes yeah I mean I'm I'm open you this is part of trying to scale the evaluations there this this was yeah this was a pretty good way to do it it took a there was a bunch of stuff that had to happen to make it happen but once it got in place and I actually even roll up the results on the topic trees so I can see the Precision go up so I have a deep tree I don't have to look at everything I get roll up and I can say okay for health you know it got 83% you know shopping got 76 and computer science got 84 and I can just tell by looking at something because at this level of you know there's so many topics you just can't look at each one you have to figure out some way to make this scale okay and if that's it for questions then we're done thank you very much