SF BA ML: Alex Minnaar, Using Word Vectors to Evaluate the Human-Interpretability of Topic Models
so my name is Alex menara I'm a research engineer here at nitro and my talk is about yeah my talks about using word vectors to evaluate the human interpretive human interpretability of topic models so this is just a lightning talk so I might not go as in-depth into some concepts as I probably should but I'm available after if you have questions so topic models are a collection of unsupervised learning algorithms that try to uncover the hidden latent ematic structure of large document collections so they can be very useful for tasks such as searching and browsing and summarizing these large document collections and most popular topic modeling algorithm is a late interior latent dear slay a location or Lda so this is at a high level lda represents topics as distributions over words and represents documents as distributions over topics so here's a here's an illustration of lda here so on the left you have your global topic distributions and here is an example document in a corpus it has a local document proportions distribution and each word is assumed to be generated by by a topic which is or sample from a topic which is sampled from the documents topic distribution rights o el día has a free parameter K which determines the number of topics to be learned and often if K is chosen to be too small or too large the topics can be of poor quality and here's an example of a good topic and a bad topic and it should be obvious why the good topic is good as human interpretable and the bad topic is not so in order to choose the optimal k we have to have a way to evaluate a topic model and lda is an unsupervised learning algorithm so that's that's not straight forward at all so early on people had been using statistical evaluation techniques so basically what they did was they got their corpus and they they divided into a training set and a test set and they train their lda model on the training set and then they computed the predictive likelihood on the tests that given the parameters that were learned in the training set so so the idea being that a model that is able to predict a set of held out documents well would be a high-quality model and vice versa but this this was sort of contradicted in a paper in 2009 called reading tea leaves how humans interpret topic models and basically this paper described an experiment so it was a large-scale Amazon Mechanical Turk experiment where a topic model was learned and then the top 5 topics by sorry that top five words by probability work for each topic were taken and a-6 intruder word a random word was added to each of these topics and then the turkeys were asked to try to identify that intruder word so if the turkeys were able to consistently find that random intruder word then this means that the topic with is human interpretable because obviously the the topic words contain some sort of thematic oh here it would be easy to to pick out the random one and vice versa so the surprising result was that topics that had a high predictive likelihood were not correlated with topics that were shown to be human interpretable and in some cases they were actually anti-correlated so this result caused people to sort of move away from the predictive likelihood evaluation metric and and and move towards trying to evaluate topic models by trying to model this human interpretability so the intuition is that topics whose high probability words that are semantically similar to each other should be topics that are human interpretable and vice versa so this sort of topic coherence general formula was was was was created so basically you take the high probably high probability words within a topic and then compute the pairwise semantic similarity between each pair so there were a few papers that that did this one using point wise mutual information to assess the semantic similarity between the words one using word debt so basically if two words fell under the same word that concept then they were said to be semantically similar and so this sort of topic coherence approach performed much better in the word intrusion task than predictive likelihood so sort of the main point of this talk is is how to use word vectors to compute topic coherence so basically just a broad overview of word vectors so they're produced from unsupervised neural network models that are fed large amounts of unstructured text data and these vector representatives vector representations of words are produced and these vector representations capture the semantic information of the corresponding word so it turns out that if you take the cosine similarity between these word vectors it's an effective measure of the semantic similarity between the corresponding words so for example if you google provides some pre-trained word vectors using the word tyvek algorithms and if you look at the word vector corresponding to the word France the most similar words according to cosine similarity are the ones you see here so words like Spain Belgium Netherlands ones we know are in fact semantically similar so basically we use the same the same idea just to take the high probability words within the talk topic and then compute the average pairwise cosine similarity between the corresponding word vectors so the best of our knowledge this this approach hasn't hasn't been tried before so we haven't yet applied this to the word intrusion past but we have tried this with our with our own corpus that we ran an Lda model on so we ran a 100 topic lda model and then we computed the topic coherence for each of the resulting topics and on the left are the top 5 topics by word vector coherence and on the right are the bottom 5 and I I think you can agree that the the topics on the left are much more human interpretable than the ones on the right and this also this topic coherence idea can have extensions to the online lda algorithm so basically the the online Aldi lda algorithm learns the topics in a sequential manner so basically mini batches of documents are fed into the model and the topics are updated in in this way rather than then in batch with regular lda so one of the benefits of online lda is that we could actually see how the topics are changing after each mini batch so if we can see how the topics are changing we can also see how the topic coherence is changing and and it's interesting to see how this actually works I mean our intuition should be that the coherent should increase as we see more mini batches the idea being the more data you see the more the more coherent your topic should be so we tried this Armand and this was the result so as you can see on average there is a coherence increase but it's definitely not monotonic in fact some mini batches actually decrease the overall coherence so each data point here is the average coherence over all the the topics learned so even though there is a there could be an increase in coherence on average there still could be some decreases in coherence in some topics so this sort of gave us the idea of what what if we could sort of simply track the most coherent version of each topic that we've seen so far and basically only updated if the next mini batch increases the coherence of that topic so this this led us to this sort of greedy coherence online lda and and so this that green line there shows sort of this these these greedy topics that are monotonically increasing in incoherence by design and as you can see at the end we we end up with a set of topics that are much more coherent than if we had just run online LD a lonely that's it so for that last part so we're actually not influencing the model with that so basically we're just running the sorry we're basically just running the online Lda normally it's just in the background we're just keeping track of the most coherent version of the topic that we've seen so far so we let it run normally but we just sort of record the most coherent and we basically don't update that version unless we see a more coherent version we're not actually influencing the algorithm at all no no yep yeah so you you'd have to have in memory the the current topic and the most coherent topic so far so it's it's double uh yeah it was it was trained on yep um good question so if it was trained on about 30,000 documents wait yeah yeah that's a good idea I mean I've read previous papers where they've done this with with Gibbs sampling they've sort of biased the the probability of sampling words that are semantically similar to each other but yeah if I had more time I'd like to explore that well we're actually not we're not sort of changing the original algorithm at all we're just sort of keeping track of the most coherent ones we've seen so far so we're not actually replacing the the we're actually not updating the actual topics that are being learned we're just keeping in the background sort of the best ones that have that have been seen so far sorry can you be back ready i'm not sure i'd have to think about that we could talk with good ok