data.bythebay.io: Niyati Parameswaran, SAMEntics : Tools for paraphrase detection
Recording: data.bythebay.io: Niyati Parameswaran, SAMEntics : Tools for paraphrase detection
hi everybody I'm niyati I work as a data scientist for IBM Watson and the first dog that I'm doing today is on semantics which is tools for paraphrase detection and paraphrase generation two unanswered questions on this particular topic slide the first is a little bit about me I'm near the-- i did my bachelors in computer science from bits pilani in india and then did my master's from the University of Texas at Austin I am consistently in a battle against entropy I am on the road to become what I'd like to think of as a base a bayesian ninja and I belong to that group of people who laugh out loud really loud in my capacity as a DJ scientists at IBM Watson I work on conceptualizing core machine learning and natural language processing algorithmic paradigms and also uh putting together cognitive solutions specific to where is partners this talk is going to be on semantics which is basically a spin-off on same plus semantics it's tools for paraphrase detection and paraphrase generation and it does this in for primary steps the first is by discovering rieber rings of sentences across domains then bucketing these questions on into hierarchical categories within the same domain identifying those buckets which have sparse data that's going to kick off the paraphrase generation piece so that you're able to enrich the overall training set for the deep to a system all of these pieces I will be going into more detail but holistically the entire purpose of this project is to expedite question-and-answer mapping whether it's one-to-one one-to-many many to one or many to many okay so with that let's step into the one of the primary pipelines of semantics which is the paraphrase detection I'm going to go over step-by-step about how this pipeline was conceptualized and how it functions so step one was aggregating scores using an somber classifier for both structural as well as semantic similarity that's for some time focus on the structural similarity bit you'd see that there are five algorithmic paradigms there that's Gero Gero winkler dice cosine and Levenstein now each of these algorithms conventionally they get used to compute edit distances between words at a character level so for instance if you had the word friend fri en d and you had a misspelling of the word friend say FR ein d in that case there is one single swap between these two words which implies that the edit distance between these two words at a character level is one now what I have done is I have basically augmented all of these algorithms to function at a word level so now think of it as we have two sentences sentence a and sentence be where sentence a si has five words and sentence be has six words that means in sentence p obviously there is one insertion and let's say there's one swap somewhere in the middle that would imply the edit distance across these two sentences is now to because one insertion and one swap so effectively what I've been able to do is by augmenting each of these algorithmic paradigms in an ensemble I have been able to find the reordering that occurs between two sentences from a syntactic construct level so that is variation from a syntactic construct and that's how I get my score of structural similarity let's call that score x we then move on into semantic similarity which is crucial because just structural similarity is not going to give you any sense of whether arm hold it just relying on structural similarities I'm going to tell you whether two sentences are always paraphrases of each other so for the semantic similarity same intakes powers of two frameworks frameworks which you guys would be aware of our that's word to work which is a distributional semantic framework and there's word net which is a lexical database now our with word to work because it is a distribute no semantic framework it has been constructed on the hypotheses that linguistic terms with similar meaning would have similar distributions or in other words basically words which are similar would appear in similar context so however what you need to remember with nye word to work is that there is a vector representation only at a word level it's not constructing vectors at a Fraser level because of which there is a lack of context so think of it this way um if you had the word pair cats and dogs and if you had the word pair boys and girls now cats and dogs could appear in a similar context to boys and girls if you're talking about eight-year-old boys and girls would want to fight with each other but that in no way means that cats are synonymous to girls or their dogs are synonymous to boys which is why we kind of need to take into account context when working with a framework like word to work which is why arm in same intakes I have augmented the manner in which were to work functions to a word to wag plus plus which basically enables me to generate contextualized phrasal vectors and the manner in which i do this is with an integration of LSA that's latent semantic analysis and lda which is latent dirichlet allocation now LSA basically generates a set of concepts based on sentences in terms so if you want to think of it at a high level when you're given n sentences it is going to reference the concepts present in those sentences and lda which is latent dirichlet allocation is a generative model and what it does is that it explains a set of observations using a bunch of unobserved groups are thereby establishing by some kinds of data or more similar than other kinds of data so at a high level basically when you're given n sentences it's going to list out the topics which are referenced in those sentences so the learning problem that I have is something like this where I have all of these sentences each of which are of variable length and what I want to do is I want to get a fixed n dimensional representation for each of these sentences and then if those are sentences are closer in the distribution of semantic space that would imply that these are semantically similar and so our paraphrases of each other so for everybody who is from or is familiar with deep learning think of this as a recursive auto encoder and I'll still explain this particular piece because this is what is the crux of same in Texas paraphrase detection pipeline so for those of you who are not familiar with what an auto encoder does basically and the number of nodes in the hidden layer of an auto encoder are lesser than those that are present in the input layer or the output layer so typically an auto encoder would have say to n input nodes and then n hidden nodes and then again say to an output nodes so what happens is when you're providing to n input nodes this is getting encoded in the arm n hidden node layer after which it's getting decoded to recreate the 2n output nodes the advantage of using auto encoders is also because this loves you to function in an unsupervised manner as opposed to using a neural network architecture which would do something similar but in a supervised manner so with an auto encoder typically what I'm able to do is I'm able to automatically extract features which are present in a sentence and what I do with this particular auto encode of framework is that is that I collapse an entire sentence into a phrasal representation the way that I do this is um like I said every sentence has been on by using LSA and LGA I have able to pick out the most important concepts and topics referenced in the sentence so this is now my sentential representation I then take the first two concepts or first two topics or concept topic topic concept whatever it is each of which are n dimensional vector representation so I basically aggregate them that gives me a two dimensional input which gets fed into this auto encoder right is everybody with me so far okay so it could be a concept or a topic so T is the topic yes so this gets fed as input into the auto encoder and the auto encoder on the encoding process is going to convert this one concept or topic or two topics or whatever it is you get the point into an n-dimensional vector representation now this gets done recursively so say I had four words in a sentence um I plug this to a n dimensional vector into the auto encoder so that's going to give me a hidden layer representation which is n dimensional I then use this along with the next word which could be a concept or topic and recursively plug it into an auto encoder again to get an N dimensional representation so this way effectively at the end I have been able to successfully collapse my entire sentence into an N dimensional representation of the most pertinent concepts and topics by using a recursive auto encoder which is basically stacked one above the other so now that this kind of collapsing has occurred what we have built through this process is that I have been able to render a phrasal representation for every sentence as opposed to a simplistic a word vector representation and this has been done by leveraging word to whack by leveraging LSA and lda and by exploiting arm the concept of an artificial neural network arm by using a recursive auto encoder so this is the piece where the distribution of semantic framework gets leveraged that's the piece one let's call that score that I get a now we also wanted to look into exploiting a lexical database the reason there is simple for a deeper semantic compositionality understanding basically to get a better sense of synonyms and contact specific words so for a word net there are a bunch of state-of-the-art algorithms that exists called H Sol CH and you'll find this on the word netpage less group rest and so on basically these tell you how related certain words are with others in wordnet there is a concept of sin set which basically means all of these words are semantically associated so wordnet gives you that but again wordnet gives you that at a word level so again it's become wordnet class class because I've augmented it to function at a phrasal level and that's done by utilizing a machine translation metric called blue which helps me understand semantic relatedness are by figuring out for Ngram co-occurrence comes so that's how that piece gets leveraged so let's call that score be so now for my aggregated score of structural and semantic relatedness what I do is that I way structural similarity at point 4 and I've a semantic similarity at 0 point 6 on the reasoning behind this is you could obviously way structure and semantic similarity based on whatever is convenient to you I just found the point for endpoint 6 seem to be the sweet spots and it's helping me rendered higher accuracy scores for this particular task so this was that particular piece yes it's a or machine translation metric that gets used if you get packages to basically use the blue metric yep so once I have this particular score the second step was to implement a hierarchical clustering algorithm that's going to group these sentences by threshold anon this particular score now as you guys would be familiar with any kind of clustering algorithm you require two things you require a metric and you require a linkage criterion so this was the metric that I chose to use instead of using simplistic Euclidean or Manhattan distance or so on and the linkage criterion that got used was a centroid linkage criterion which basically tells me that the distance between sets of observation should be studied as a function of the pairwise distances between those observations so by you centroid linkage for that and it's a agglomerative bottom-up clustering so basically when you have a whole new g of sentences it's going to group these sentences starting bottom up on the basis of syntax and semantics relatedness I substantiated this class files ability to recognize hierarchical clusters within the same domain tested this on the microsoft paraphrase corpus and on semi well data for standard corporal validation and also compared the performance against the state of the art so here are the results as you see the baseline is by on the healthy a cosine similarity with tf-idf waiting is what they use the accuracy was 64 65.4 and an f of 70 5.3 semantics beats are even the last paper that came out in 2008 in this particular space of unsupervised are clustering by intent or unsupervised paraphrase detection this is not any kind of semi-supervised or supervised algorithms none of those are mentioned here the accuracy of semantics is seventy four point eight and hits an f of 82.6 um the reasoning that I would attribute as to why are we were able to beat the state of the art is primarily because semantics has the ability to account for both structural and semantic similarity which like you would see in the prior literature none of those papers work with and semantics also is arm useful and distinguishes itself from other papers in this domain because it allows for a contextualized understanding without any kind of hand designing of representations so that's why it works as a superior algorithmic framework which is why it was able to beat the state of the art I'm gonna show you guys a demo Simon takes has been used often with a bunch of our yes that's correct absolutely but a spelling correction like I said it's happening at a word level right so certain characters are misspelled here i'm not talking about it add a character level insertion deletion replacement or swap i'm talking about it at a word level so how different are it is a sunny day sunny day it is although it's not grammatically correct but there is a syntactic reordering there and that's what gets captured okay so um i'm going to show you guys a a demo so basically as simon tix has been used with a lot of our IBM Watson partners because we have a service call the natural language classifier which is basically the deep key way that Watson is no no for question answering and as training data for this kind of question answering system we kind of need the consumers or the partners to put together something known as a ground truth where they have a bunch of questions that the users of the application that they want to build would be asking and they have to label these questions to an intent obviously as you can see there are overheads associated with time and computation to do this so this is where say matix get leveraged because it clusters questions on the basis of the intent associated with the questions because it works as a paraphrase detection tool so I'm going to work with dummy data because obviously partner confidentiality I cannot show you the other data but like you'll see the input here was basically a bunch of 18 questions and on an analysis what's happening is that questions which have a similar intent or basically a referencing similar concepts or about the same thing semantically even if they are varied in structure they get grouped into the same cluster bucket the next question that you guys would probably ask me is there any a--they the cluster description names don't seem great and yes that's true and the reasoning behind that is because same in takes is an unsupervised clustering algorithm so right now the cluster descriptions are basically brute forced to pick out the most pertinent announced from one of the queries in that bucket so like you would see for cluster ID 1 itself what adds to shoe comfort when running um it's brute force to pick out shoe comfort for instance as a cluster description that's the space where there is scope for improvement and the scope for improvement are comes through by making it a semi-supervised introducing for supervision basically if you have a customized ontology for your data obviously the manner in which the cluster descriptions get rendered would improve with training but this is a demo of that do we have any more questions of the matter is that it still beats the state of the art even if there weren't are that significant and improvement which is which is all that I can say for this the reasoning behind it is that we are using a contextualized understanding and a phrasal representation but yeah I don't really have anything else to say as to why there wasn't as much of an improvement that you'd like sure metric GN Jayceon is a word net similarity are it's one of those algorithms are that's provided by word nut that's that's a vector-based this is a vector-based plus a deep semantic compositionality based framework yes yes right yes absolutely yep all of these sentences are of different lengths so I'm not working with their paraphrases so they're not needed to have the same length so oh no I haven't done that I haven't done that so you're saying basically is there a change in accuracy scores in accordance to the length of the sentence no I haven't done that kind of validation test but it might be interesting is something to look to yes so I I don't have that slide Whitney here because I knew the right to get this done in 30 minutes but I do have one separately and we can talk post the doc yes yes mmhmm mmhmm yeah so I actually did a comparison against my ensemble classifier just for structural similarity versus that and they perform comparable or so so I didn't really replace mine with the new s that's come out because they are doing comparable the reasoning behind that would probably because be because i'm using an ensemble as opposed to just using one single algorithmic framework which is what they're doing but in addition to that we have a semantic similarity component and so overall our accuracy and F scores are higher yeah these are all of these results are on the MSR our PT and the semi well our data set together for is it for unsupervised clustering because i am validating this only across all unsupervised clustering frameworks for paraphrase direction okay because all I have is from the semi well and the MSRP be up to date and this is the maximum but i'll be happy to look at that paper post the dog okay I I look at that what year was it published okay all right so uh yes yeah so I with the semi well data set in particular they do have they have a reference they've manually created certain things which are paraphrases of each other or not so you basically have to check whether those sentence IDs are present in your paraphrase detection once you've run the entire framework so basically whether it's a hit or a miss it was a binary assessment of whether it's been captured as a paraphrase of a particular sentence and placed correctly in that cluster or not yes correct yeah yeah that's that's exactly what I'm working on right now to figure out if there is that kind of variation that causes the difference yep okay so I have another piece to go over really quickly which is the paraphrase generation so with the paraphrase generation this initially started off as a template eyes approach but as obvious when you're using a template eyes approach it's heavily constrained on syntax so for instance if I gave in it is a sunny day it's going to come back with it is a bright day but this is true for most data-driven approaches because while they might be able to find and capture more nuanced substitutions from a context sensitive data perspective it's still these kind of algorithmic paradigms are not really very adept at introducing function words and following like general principles of grammar so the idea was to not just rely on syntactic constraints but instead conceptualize a surface réaliser which is able to generate paraphrases that are syntactically variant yet retain semantic meaning so this is done by following these four steps first is where you take in word level alignments of two sentences which are paraphrases of each other as input you then project those word level alignments into something called logical forms which is effectively an automatic parts of the sentence arm dependency parsing is what's getting used these projected and lineman's or logical forms are then converted into disjunctive logical forms or dl f's and this is the piece that I've tweaked because I've converted the disjunctive logical forms in both directions and these disjunctive logical forms basically represent alternate choices at a level of semantic dependency so let me explain that a little bit essentially for English we assume that there is structure in language only when we read from left to right as in its uni-directional now that I am speaking you might be able to predict the next word that I am going to say right so that's because it's uni-directional but obviously for a machine there is structure in language even when it's interpreting it from right left so what I did is um conceptualized a bi-directional sofas réaliser which is able to tap into structure of language from left to right as well as from right to left so think of it this way if I had a bigram model given the current word i'm able to predict the next word say for left to right and let's do the same thing for right to left so given the next word I would be able to predict the previous word so if i use a bi-directional bigram model I'm effectively working on it as a trigram model except that I'm conditioning on the middle word and I'm able to predict the next word as well as the previous word so this kind of usage of a bi-directional bigram model which then got fed into open ccg for the N best realization enables me to get a more varied paraphrase generation result and I'm able to basically even find grammatical alternatives to the original sentence and also able to mix and match content across the paraphrase is provided as input open ccg is for those who are interested it's an open source java library to which i've made some tweaks open ccg was something that came out of jason baldridge one of my professors back at UT he contributed heavily to this effort but yeah that's pretty much about the paraphrase generation piece I don't have a real-time demo for you guys since this takes a little bit longer to run but here is an example so if these were two sentences that I provided as input to same in texas paraphrase generation i get a whole bunch of potential outputs which are all paraphrases of these sentences you can see that apart from there is that mix and match that I spoke to your bath so is there a way I can shut my account there is is there a way for me to close my account there is introduction of some kind of function words and the basis of the grammar of English and in this manner we are effectively able to enrich our ground truth and gain more variations of the same sentence with variation in syntax that is still retaining semantic meaning and yeah that's everything around Simon takes as a paraphrase detection and generation to questions and thank you yes correct Trey so you're absolutely right in that place because you're looking for a word level substitution generally data-driven approaches are better because then you you don't have something general across various domains right so for instance this can be used across various domains because it understands the construct of English as a grammar with what you're talking about is a in in banking or say in the real estate sector there would be certain words which are replaceable and so that's context-sensitive and then you can have word level substitutions so I am NOT are a linguist so I don't really have actual algorithmic paradigms for you that or approaches that would do that for you but that's where you should start to look if that's what you want is all any more questions yep all right thank you