data.bythebay.io: Niyati Parameswaran, PhrazIt : Tool for automatic text summarization
Recording: data.bythebay.io: Niyati Parameswaran, PhrazIt : Tool for automatic text summarization
so hi everybody I am Nia T and this is the second consecutive talk just a show of hands how many of you were there for the previous talk all right okay so now I'm gonna be talking about a phrase it which is a tool for automatic text summarization it's an extraction based summarization a little bit about me I am a data scientist at IBM Watson where I work on conceptualizing core machine learning and NLP algorithmic paradigms and also building cognitive solutions specific for partner use cases I did my bachelor's in computer science from BITS Pilani in India followed up with a master's from the University of Texas at Austin I've been at Watson for a little over 10 months now I am consistently battling in choppy I'm on the path where I'd like to become a Bayesian ninja and belong to the group of people who laugh out loud really loud and that's a picture of me doing exactly that ok so for this particular talk the focus is phrase it now text summarization happens in two ways there are two kinds of algorithms that exist and two kinds of literature that gets published around summarization there's extraction based summarization and there is abstraction based summarization so extraction based summarization is basically when given a document or documents of text effectively what is done is that you are extracting sentences from that passage or document of text verbatim so there's no condensing that's happening in any format it's just picking out sentences to form a shorter summary abstraction based summarization however works differently we're in apart from condensing sentences in the sense that picking out the most relevant sentences you're also changing the manner in which a document is represented so that's the more true summarization framework but phrase it is an extraction based summarization so it follows two primary steps the first is where we establish a thematic or a descriptive score with every sentence in a passage of text and then we reorder all the sentences in that passage or document of text on the basis of thematic relevance you'd see across literature in extraction based summarization that sentences get ranked on the basis of three status static features which you use as indexing weights and these static features are term frequency term position and term length so while the link phrase it the idea behind it was why just work with three static features why not account for context and a contextual understanding so that we are able to associate a theme with whatever document we are investigating in order to render a summary so the second step that phrase it did is where we ranked sentences on the basis of a thematic score which was associated to them and in this regard we were able to allow for a contextual understanding so phrase it therefore introduces a unique idea of a context based indexing to resolve the problem instead of working with a context independent term indexing it's important to understand that every document that we are working with more often than not contains content specific terms and background terms so instead of just thinking of every document as devoid of the context in which it appears it's kind of important to account for that context and that's exactly what our we did through phrase it because the existing models in the extraction based summarization space they're not really able to distinguish and differentiate between terms by a sole reliance on just term weights because the term bases here are term frequency term position and term length so we use even context when establishing the importance of a sentence in a passage or document so this complete dependence on term significance is reduced heavily and now your document indexing weight is not completely independent of the context in which appears talking about the algorithmic framework that gets leveraged we are powering of our text rank which is an unsupervised algorithm the reason again for using an unsupervised algorithm is because with a supervised text somewhat supervised algorithmic paradigms for summarization you need to provide a large amount of training data and this in effect translates to having a whole bunch of documents with a bunch of non key phrases and while our supervised techniques are capable of producing what we call as interpretable rules in order to identify what characterizes a key phrase the trade-off was that you require a significant amount of training data and so we decided to move away from that into the space of unsupervised algorithmic paradigms for extraction based summarization instead additionally within unsupervised keyphrase extraction framework like phrase it it's also way more portable because it hasn't been trained on a specific domain so basically there isn't any kind of customization in the extraction process instead it's capable of learning features that are present in the text and they're able to determine whether certain key phrases are central to the text or not so this is done with text rank it works similar to Google's PageRank algorithm which I'm sure you guys are familiar with where the idea is to select the most important webpages but again like I said with text rank it is a graph based ordering algorithm and it Associates importance of a sentence in a passage or document of text by using the static features of term position term frequency and term length so we don't do that we work with also context and I'll just explain how yeah so what's happening with phrase it if we need to put it in perspective it's is that it's just a general or graph based ranking algorithm and this is how it it's going to work so basically every sentence in a passage or document of text gets represented as a node and the edges across these various sentences are what is going to this comes from the contextualized phrasal vectors which is what I went over in the previous talk but I'm just going to do a quick overview for those of you who weren't part of the previous talk effectively with the contextualized phrase of vectors I am able to get a phrasal representation for every sentence on the basis of the core concepts and the core topics which are referenced in that particular sentence it is it powers off both structural similarity as well as semantic similarity so for the structural similarity piece we are basically looking at how do I explain this in really short because that was all of the previous talk ok so we have a word to whack and we have word nets and sets and what I'm doing is that with word to work and word nets and sets in word to work I have LSA as well as Lda additionally so what work has been augmented now to be able to render phrase of variations Frasor representations of sentences as opposed to just a vector representation at a singular word level and this is possible because of LSA and Lda where LSA reference is the most important concepts in that sentence and lva are references the most important topics in that sentence so basically by leveraging this piece I'm able to get a phrasal representation for every sentence and this is what gets used in order to establish how important a sentence is to a passage or document of text because that is what is giving me the theme that associates a particular sentence given a document or passage post this kind of evaluation effectively once this kind of graph is constructed we use a stochastic or a Markov matrix to identify the ordering of these sentences on the basis of thematic relevance which sentence has a higher thematic relevance automatically gets bumped up in the reordering and that gets pushed up in the summary that gets rendered so this if we need to go into more detail basically for the Markov matrix I'm going to obtain that kind of ordering on the basis of the eigenvector that corresponds to the eigenvalue of one phrase it again has been validated against the state-of-the-art on semi well as well as the duke 2003 corpus post 2003 Duke as well as all of NIST they've been using a metric called Rouge on which I haven't done analysis just yet so we were sticking to our metrics of accuracy and F just in order to retain metric consistency across all of the state-of-the-art that we have evaluated on so here are the results around that phrase it essentially like I said it's a text rank with lexicalized Association because it's allowing for an understanding of context and here the accuracy it beats the state-of-the-art with 70 point zero two on accuracy and the F is fifty nine point nine the initial the baseline was a knife based algorithm that had an accuracy of forty point three and an f of twenty seven point eight I know that this talk is scheduled for 50 minutes but I don't know why that was it seems like the previous talk in this talk should have been interchanged on time but I'm gonna quickly show you guys a demo and I think we're gonna wrap up much earlier here so this is just a piece of dummy text it's an email conversation that it happened and what I do here is let me just show you with alchemy language because this is something that has been integrated with phrase it I get the most important concepts which are referenced in this particular piece of text so the piece of text for those of you if you can't read is basically I hope you got my email earlier on the menu we look forward to seeing you at the breakfast session on Wednesday for the more given our past discussions in your focus area I also wanted to check with you if you'd like to shed you a meeting would want to shed you'll a meeting in the second half of tomorrow let me know if either option work for you so basically by leveraging alchemy I am also additionally able to get a sense of the keywords and the associated relevance across these keywords and that is something that gets used in phrase it additionally when building that contextual indexing space here is a demo of phrase it I'm plugging in the exact same text and this demo is just to basically show you how our phrase it is able to pick up sentences on the basis of what might be most significant to a particular passage of text and what is really the call to action item so if each one of you just reads this particular passage of text there are certain action items which are referenced there and so if you basically wanted to pick out just one sentence which is most crucial to this entire email it would probably be I also wanted to check with you if you would like to share you the meeting with Gregory and us where shed Yuling a meeting is the most important action item so as you would see phrase it there's a pretty neat job and this is I like to use this particular dummy piece of text because it picks out the correct action item as the single line summary now in the two line summary it picks out the second action item which is we look forward to seeing you at the breakfast session on Wednesday it also does something neat here which is it reorders the second call to action item in position one and pushes the first call to action item in position two the reasoning behind this is that phrase it also has a cohesive summary sense so it knows that sentences that start with furthermore or not the way that idle idly summaries would get generated and so it reorders various sentences in the summary in order to get a summary which is more cohesive to read you can work with so say you have a particular document of text and you want a 20% reduction in that document phrase it allows you to do that it's just not part of this particular demo you can also get a deduction on the basis of the number of lines that you would want in a summary say you have hundred lines in your document and you want to reduce that to like maybe the top 12 phrase it will also enable you to do that there is another piece to phrase it which I have added recently which is currently phrase it at least the demo here is a single document summarization which means for one particular document or a bunch of passages of text it's going to render the offense of that passage or document of text I've extended this to a multi document summarization framework where basically you can have more than one documents and a phrase it will still pick out the most important sentences referenced across all of those documents this is done where the text rank algorithm is replaced by Alec's rank algorithm and the utility of this is apart from just picking out those sentences which are important across various documents there is a post-processing heuristic that I use called cross information subsumption and using cross information subsumption if there are certain sentences which are basically paraphrases of each other or are saying the same kind of thing we remove those and we a love for we bias for lexical diversity so that every sentence which is picked out of different documents too gets represented in that summary so that's an extension of what is being worked on now and yeah that's pretty much what I wanted to show you guys through the automatic text summarization piece I see we've done really well on time if you guys have any questions please feel free to ask at this point yes um randomness just fighting randomness I have some kind of OCD and everything just needs to be structured kind of Mira is my job where the data scientists have to find structure and data so I guess it's it's just there in life the summaries oh that's just the sizing I guess yeah yes so were to whack I'm just using that word thwack peace which is the Wiktionary corpus so there are no labels as such which is associated with any kind of data the idea is to get a phrasal representation of every sentence in a vector format basically contextualize phrasal vectors as opposed to word vector representations yeah those vectors precisely precisely yep that's what it is any other questions yes any more yes no no phrase it was conceptualized for the use case wherein we have a lot of partners who work with IBM Watson who have a lot of noisy crappy data sometimes a lot of the data is just outliers the idea is to pick out what is the essence in that data so that we'd at least have neat enough data when we are training our systems so that's what that's been done to but that is an interesting thought and should be implemented for for what yeah they're like Google's open source work back although they call it is not really part of they're not really something that they've given out but it's still it's still easily available so that's something that you can look up