Chris Moody introduces lda2vec

Chris Moody introduces lda2vec

Recording: Chris Moody introduces lda2vec

[Music] thank you Alexi for the awesome introduction thank all of y'all this is a ridiculously large uh coming out I didn't think anyone would be that interested but clearly I was wrong about that um and thank you Dave a lot for organizing and of course Nitro for hosting the space so for anyone who's played around with text and NLP you can you know that it's a really messy affair because you have to teach a computer about the irregularities of the human language you have to and lots of ambiguities and you have to teach this sort of like hierarchical and sort of sparse nature of English grammar and vocabulary but at Stitch fix when a client says that she's in her third trimester we know that she's pregnant and when she says that she used to wear scrubs we know that she's in medicine and when she says that taking a trip we know that she wants a fix for vacation clothing and so this is sort of the power and the promise of word vectors is being able to remove a lot of these ambiguities and make all of these different concepts a little bit more similar so so before I start I want to talk a little bit about myself uh I'm Chris Moody uh that's my Twitter handle what was the uh what was the the hashtag that we were using for this machine learning SF machine learning is that it is all right so please feel free to use that uh I was Caltech physics uh I went to PhD I did a PhD in astats and supercomputing at UC Santa Cruz uh so not too far away uh I'm a psychic learn contributor I wrote the the the latest version of T which is a dimensionality reduction technique and it's makes really pretty like uh visualizations for high dimensional data which is sort of like this top right diagram over here uh and now I'm at data Labs at Stitch fix uh and so this is an awesome job because Stitch fix tremendously values high high risk High reward research uh and highrisk high reward work and that means that I get to play around with lots of different stuff like gum processes tensor decompositions um I'm a contributor to the chainer Deep learning framework as well uh but anyway Stitch fix is an really really awesome place to work uh and if you guys uh think so too certainly like U be happy to chat about that afterwards cool all right um say what okay so you guys can't just hear me okay all right great so I'll talk into this then cool so today I want to talk a lot about uh a few things one I want to explain word DEC and I'll try to go through it a little bit quickly because I feel like a lot of folks already know a lot about that I want to talk a little about LDA which is a similar text algorithm uh and then at the end I want to save a little bit of room for some series experiments that I made over Christmas that I've been calling LDA DEC that tries to mix and hybridize a lot of the ideas that are inside of word DEC and LD into a single algorithm um and I don't think we'll have time for this but it's a bunch of fun stuff out there uh if you see this presentation online later on you can look at it but there's all kinds of really cool applications for word on on graphs not at all having to do with text uh for recommendations so if you've if you use Spotify the Discover weekly that comes in every week it gives you this recommendation for a new playlist that runs on word deac right but it doesn't actually use text at all which is really interesting um but anyway lots of cool stuff uh but lots of good good nuggets to go read about later on all right uh before I start really want to give credit to all of these folks Tomas mikolov uh David blae who invented word DEC and who uh discovered LDA or wrote LDA I should say respectively and then Chris Ola is a Peter teal fellow he has this really awesome blog these really really cool visualizations of neural networks of word representations how lstms work really goes nuts redim rur uh wrote The Python Jensen package which is a really nice way to get into word DEC in Python Omar Lev and Y of Goldberg are emics and they connected this great big body of work with RC back to sort of more traditional NLP statistics like PMI we'll talk a little bit about later on Richard soer is the CEO of metamind and he has this awesome NLP Stanford class Shin Rong and Tim Hopper both give these really Lucid explanations of of how LDA works and the the sort of like the technical details the mathematical uh derivation of the gradient behind word DEC really awesome stuff all these things are links uh if you do find the talk online Okay so this this this example right King minus man plus woman equals Queen and this is what made word DEC famous and this sort of blew open uh certainly my brain when I first read it because a computer can actually do this right so a computer takes the word King subtracts the word man adds the word woman and when it evaluates what that answer evaluates what that answer should be and it gets Queen now so you start to think that you know maybe the computer actually understands what these words mean maybe you can actually solve these analogies cleanly right like it understands the biggest difference between uh between king and queen is the same as the difference between man and woman namely the gender so that's really really remarkable because we didn't have some table that was like oh king that's man or that's like a male and queen that's female and like women that's female we didn't have a list of attributes we just learned this from a bunch of raw text right and I think that sort of Simplicity uh gives this algorithm a lot of a lot of Leverage and a lot of power um great and a few other things uh people like to say that this is like a shallow algorithm it's not deep learning okay fine um but it's not super big difference it's just sort of semantics uh and also a really nice thing is that it comes pre-trained uh you can actually download the word vectors you can use them on your own corra you can calculate differences between those words yourself which is a really nice thing to do okay so I'm going to talk a little bit about how the word DEC algorithm actually works so like a lot of machine learning algorithms it's you're going to set up some objective function you're going to randomly initialize all of the parameters and then you're going to do gradient descent Trying to minimize your objective or maximize your objective in some way okay so the challenge here is that I'm going to actually try and not ever mention neural networks once throughout this entire talk right um too late too late well after that um and so so sorry so what we're going to try to do is learn this one word Vector from its surrounding context and if you're thinking about NLP Maybe thinking about NRS and you're thinking about transitions between engrams or maybe you've played around with TF IDF and you've built a matrix of terms and then what terms are in what documents and you've tried to factorize that Matrix and now it's LSI okay this really isn't that right uh this is really we're going to try to learn the embedded representation directly with no intermediates and we're going to try to update it with every single example so let's say that we give it the sentence the fox jumped over the lazy dog what we're going to try to do is maximize the likelihood of those words given the word over so we're going to try to maximize the likelihood of thee given over and fox given over and maximize the likelihood of jumped given over and so on and so forth and so this is a very much a bag of words model there's no state that's going in from word to word to word that's remembering where I am in the sentence that remembering that oh I just had a noun maybe I'm expecting a verb to come up next so those things are more like more like Markov models or crfs or like lstms it's not like that at all all of these words could have been jumbled up inside of that window and it's not going to pay attention to that order at all so it's a pretty simple assumption because all you're doing is trying to predict one of those words given one of the other words and aside from that there's no other like really big parameters that go into this all right so what should that kernel be what should Fox given over beam so first of all you're toas mov and you're trying to write word de so it's going to have to depend on word vectors in some way uh and so you're going to say okay Fox given over is going to be the same thing as the vector for Fox given the vector for the word over and we're going to randomly initialize them we're going to have a small twist we're going to have two vectors for every word and it should depend on whether it's in the input context or in the output context I'll explain what that means in just a second so in this case the input the context Vector is going to be the word over and our output word is going to be the word the and we're going to try and maximize the that seeing that pair of words and we're going to try to maximize the probability of fox given over and jumped given over and the' given over and lazy and dog and so this is the innermost for Loop inside of word DEC all right uh we're going to have this Central pivot word and then we're going to go back a few words and we're going to forward a few words and within that window we're going to take all these pairs and we're going to try and maximize the likelihood of those pairs okay now there's one for Loop just outside of that and we're going to switch from the word over we're going to now start training on the wordthe and we're going to say maximize the likelihood ofth given the and fox giventhe and jump giventhe and over given the and lazy given the and dog given so this is now the double for Loop right inside of where right so I start with one pivot I have some window around it I look at everything in that window and then I move a window over one okay so that's that's that's kind of it so this kernel is kind of important but that's that's that's why I think it's so weird uh when and a kind of a disservice whenever we start talking about this big giant neural network that doesn't I said it twice now sorry uh you know I think it's actually a little bit simpler to understand it it's just sort of like a double for Loop and it's maximizing these two little things okay all right so what should this kernel actually be what should V out given B and B so now we've defined the sort of highle update path now we just need to talk about this little kernel inside of it um and it's going to boil down to what's the difference between V in and V out we want to make them as similar as possible whenever they show up a lot together and we want to make them really dissimilar if V in and V out really never co-occur so in this case we're going to use the inner product the cosine similarity if it's normalized it's not too big of a difference but you could have imagined other things you could have said hey maybe I want to minimize the ukian distance or if I'm getting really fancy maybe I want to minimize the mahalanobis distance or other things but in this case we're going to use the inner product so to remind you the D product has these sorts of properties when the vectors are really really similar in this case they're kind of pointing the same direction they have kind of the same magnitude they're going to be around one at least if it's normalized and so that means things like Canada and snow which are maybe related words they're going to have an inner product that's kind of like one on the other hand like if you have words like Canada and desert maybe the inner product is going to be kind of like zero because those things aren't that similar in this case the vectors are going to be slightly orthogonal and if they're like actively dissimilar like very negative in the same Dimension then you're going to have uh an inner product that's closer to negative one and so that means the do dot product is always bounded from negative 1 to one at least when they're normalized but we'd like to measure a probability in some way right like this I can't have negative probabilities but I have some sort of measure of similarity so how do I get from similarity to a probability and that's where the softmax comes in this softmax function is very critical to lots of other like machine learning algorithms and so I want to dive in a little bit into it so the softmax uh is really really really similar to a logistic function you might have heard of a list logistic function before and that Maps you from a scalar to a binary outcome that says given some score how do I map this to a true false softmax on the other hand is given some score how do I map that to a categorical outcome to one of n outcomes and what are my n outcomes in this case the N possible words I could have chosen so given this input word what are what are all my other possible choices everything in my vocabulary and I'm going to score each one of them by the inner product of those two things and so softmax is so that Vector in product is giving us similarity and that soft Max is mapping from similarity to probability okay so here's the actual form of the equation this is just proportional to this is the top part of the equation um and so it's just going to exponentiate the inner product of those two things and so that inner product was going from NE 1 to one and so now that exponential is bounded to just positive numbers okay whatever uh but now I'm going to normalize by all of the other possible things it could have been so now in the top hand side I have the two word vectors I saw and then on the bottom side I see all of the other possible choices it could have been and so this has two effects one is that top side is is the numerator is trying to push those two word vectors closer together it's going to push those two word vectors that I just saw try to push them a little bit closer try to make them almost the same right so try to push those word vectors to be aligned on the other hand I really want you to push all of the other word vectors that I didn't see farther away and so that's what this softmax function is doing and so that's going to be the probability of V out given V in and so this is the kernel inside of word DEC and every time I have that double for Loop every time I have a pair of words I'm going to try and measure this thing and try to measure this probability and so what I'm going to do after I measure that probability I'm going to do gradient descent on that particular probability and I'm going to say try to maximize that probability move the vectors in a direction that will make me make my probability a little bit higher and so I'm not going to go through the derivation of that gradient it's kind of lengthy and kind of boring but like it's that's generally the idea and what's really cool is that this is actually relatively simple right I have a double for Loop and I'm optimizing one thing between two words um and it's relatively simple and what's amazing is is really fast and fast enough that you're reading billions and billions and billions of words in every day and so that's actually one of the most powerful parts of word DEC is it is the fact that you can cover so much data that uh that you start to learn all of these you have lots and lots of observations for every word Vector okay so I did lie a little bit um there are actually three different algorithms inside word DEC uh and I'm going to talk about just two of them this is the second of those and this is the skip gram negative sampling algorithm um and this this case before we were doing a soft Max decision we were saying map this score or whatever to one of n different um different words in this case we're going to want to make a binary a logistic choice and the question that we're asking of our data now is did this come from our data set did this pair of words this context and this other word combination did that come from our data set or did we completely pull that out of thin air and so what's what's really interesting is that this thing is uh is actually like sorry trivially uh Sol like solved by just having the context Vector equal to the word Vector so that means that if I only give you examples from the data set then this algorithm will tell you yep all of those things are from that data set and you'll be like okay well you were never wrong and you perfectly optimized that loss function so what you need to do is give it some negative samples give me some samples that like didn't appear in that data set and so that's when we add another term to this equation and we're going to say okay draw I'm going to still use the same context word I'm still going to use that Central pivot word but I'm going to now randomly draw a bunch of other word vectors from anywhere in my vocabulary they didn't appear in that window and I'm going to say make sure that you can discriminate between my fake samples over here and the real ones that came from the data set so this is a slightly different than the question we had before before we were like okay given my word Vector pick out of like the 10,000 100,000 other word vectors make sure that like I picked that one what's the probability of that in this case I'm asking a slightly different question in this case I'm saying given those two word Vector given those two words that were in my my that were observed make sure I can discriminate them from my random sort of adversarially drawn negative samples and so one twist that happens in real life is that uh you actually have multiple negative samples uh and just one positive sample okay and this is sort of a parameter you get to pick in in the word de model now what's really really really really really really awesome and what Omar Levy and Y Goldberg showed and this is what what I think made them really really famous and sort of blew open a lot of like nips work is that this uh this skip gram negative sampling model is completely equivalent to Matrix factorization that means that like if you've done PCA before if you've done SVD before uh if you've written if you played around with Matrix recommendation algorithms before you've probably done some amount of Matrix factorization which's really cool is that algorithm we just finished talking about that was trying to discriminate um things that were from my data set from things that were definitely not from my data set is equivalent to this right and so that's really really really really exciting because that equation just comes from sort of more traditional NLP this connects word DEC this makes it feel like it's not completely out of the blue this connects it right back to a lot of like traditional academic work okay so PMI is this pointwise Mutual information it's fancy but it's the actual equation is actually super super simple and it's just an information theoretic m a measure of Association like how Associated are C and W in this case so let's let's look at this for a bit so it's PMI of a matrix and your Matrix is a co-occurrence counts so in this case how many times did the word dog show up in the same context as the word uh Fox you're going to write okay I went through all of my Wikipedia Corpus or whatever Corpus and said oh eight times they were in the same window uh and then you know I guess in Wikipedia probably more than eight but like you know just for illustrations how many times the dog and Siberia show up in the same context oh once maybe dogs are really rare in Siberia and maybe those two words are really really like unlikely to show up in the same context foxes on the other hand they're running around Siberia everywhere and so like you put it in a four and so you think that those things are really really similar and so this is so and so this is what we're doing with the PMI we're just going to take this Matrix and we're going to say all right take the number of times Fox and Siberia showed up in the same context and so that's that top the number number of times Fox and Siberia were the same and then normalized by the number of times I've seen Fox everywhere and the number of times I've seen Siberia everywhere and so that that numerator or that denominator is really just controlling for the popularity of the word Fox and controlling for the popularity of the word Siberia and that's really really really important thing to do because words follow this power law distribution like the fir like the word the and the word and that's like every other word right so you need to control for the popularity just because the wordthe show up there doesn't mean it's really related to the word Fox just because they co-occur frequently you need to control for the fact that those words are very popular and so this is really just getting right back into how much are those things like once you remove popularity how much are they really related to each other uh and so that's what this measure is right and it's actually relatively simple and so it's really really um it's beautiful to me that like the algorithm we wrote before that performs so magically well can actually be simplified into just these kinds of co-occurrences and connected to this great big body of work in NLP now what word only does is it actually adds this little term hanging off to the right hand side and it's a minus log K it's really just a bias it's a constant it's really kind of unexciting and it's kind of remarkable that like you shift all of this thing over and then that's then that's word to back and so what this is effectively doing is it's down weting really really rare terms and it's making more frequent words uh so that they're more weighted and again that makes a lot of sense because words follow this power law distribution don't don't don't necessarily spend so much time so much like computational effort so much algorithmic effort trying to model like the really rare words you really want to get the words that you see all the time correct and so that's what this is saying downweight the things and it's this waiting that fundamentally makes word de different from all of the other algorithms that come before it uh okay well this is sort of a mess of a slide and know but it's really pretty uh that you can actually show this proof in just a few lines and I won't do it here but um you guys can look it up later if you're super curious all right so qualitatively how well does word DEC actually work uh let's let's look at what are the words that are really similar to Redman and word is in the bottom row and the Redmond is that first or second column and previous models they were like oh Conor and lck and Keen those are all the words that are similar Redmonds like I the thing at least to my eye those things don't have anything to do with Redmond but if you go down to word DEC oh it's other Redmonds Redmond Washington okay fine other spellings are Redmond but Microsoft also shows up there and that makes sense because Microsoft is actually located in redond all right well what what are the words that are similar to Ninjutsu you get Ninja you get martial arts you get okay that's great those are obviously like very very very similar sounding words uh okay fine but maybe you don't really believe qualitatively the results like let's breathe let let's start looking at the quantitative results and the way you do this you start by testing analogies you do you ask questions like King is to man as Queen is to what and the aners woman and then that in that case it would be a semantic analogy and so that's the third column from the right but you also have syntactic tests in which case you're asking questions like walk is to walking as run is to what and the answer is running in this case you're testing like did you learn the gerant form of these words uh and so it's a little bit more grammar and you're not asking things like gender anymore cool all right anyway it does it does remarkably well it certainly beats the pants off of everything else before it um and you know there's like things that like have come and and improved on word DEC but like this is sort of like an impressive benchmark for a very first thing so what's actually happening when you add King and man and woman together so I kind of want to try to drive some intuition about what word vectors are and how they mean and and how they sort of interact with each other so let's load up all of our word vectors in this case I'm just showing just four of the words but we have a 100,000 words and I'm showing just two of the dimensions and there's 500 Dimensions so this is kind of like a as if we'd effectively PC it and we're going to take the difference between man and woman so remember that we're taking king plus man minus woman so we're going to take that difference and we get this Vector we're going to load up the word uh King and then we're going to take that Delta and add it head to tail right kind of just like how we did in kindergarten or maybe not kindergarten but and we're going to Res have some resulting Vector in this case it's going to be 3 and 0.9 live somewhere in that space just a set of numbers we're going to do we're g to have our other 100,000 word vectors that live out there and we're going to try and find the closest word Vector to that one and of course voila we're going to find that the closest word just happens to be the word Queen right and so this is how we're solving King minus man plus woman equals Queen uh and so Queen just ends up being the resulting Vector now what's really really really interesting is that this red Direction always encodes gender right so the difference between man and woman looks exactly like the difference between king and queen and that means it's always feminine and it's not just for those four words like you can go out and out and out find it for other words right so the difference between aunt and uncle is exactly that Vector to the difference between daughter and son is that Vector to and so that means that like you know after a while you start to see the look of this pattern it's like okay whatever that direction must always mean that we're going to move in the more feminine Direction cool so we have hundreds and hundreds and hundreds of different directions right we have a 500 dimensional space that means at least another Direction might be higher status right that was the difference between man and King for example or the difference between woman and queen but we have another 4 98 Dimensions so what do they encode let's start looking at some examples right so you know we live in this Vector space where operations like addition and subtraction aren't just sort of like mathematical Concepts they're like meaningful in a lot of different ways so check this out right like Czech plus currency you get coruna you get Czech Crown that's perfect right that is definitely the Czech currency uh Vietnam and capital you get Hanoi right German and Airlines you get Lanza how awesome is that right like so Lanza like you start to really feel as like a perfect mixture of just German and Airlines it's really cool to think of these vectors as just being additions of other ideas and of other uh of just vectors and ideas being sort of interchangeable and that you can fact that you can add and subtract these ideas to create other Concepts cool so I yeah uh I'm not sure I could do you oh yeah so the question was uh why do you have multiple words uh and so it goes back to how have you pre-processed your document uh in this case they've applied a pre-processor that takes U lots of commonly co-occurring words and then transforms them into one token like you can kind of think of it as just like taking those words and then like adding underscores in them saying that yeah that like uh like hoochi Min City is secretly like one word right um which in and of itself is its own algorithm and an interesting thing how do you actually go from four words to one Tok but um we'll talk about that yes you add random and at the distance to the nearest nebor in your Corpus um those words so if the words have some kind ofical relationship to each other you're G to have to say that again so you comine say uh I don't know check yeah are just words that we don't expect okay just combine words at random uh yeah so you oh yeah absolutely because there might not be a word that lives at that point in space so the question was uh if you take uh you know Vietnam and add completely like ridiculous word to it like Vietnam and I don't know Scala right you're like I don't know what the combination of Vietnam and Scala is and certainly you'll calculate a a resulting Vector for that and then you'll try to figure out what the closest word Vector is to that point and you might not find anything or I mean you'll eventually find something you'll eventually go far out enough that you'll find something but it might not be that similar it might not be that close yeah that's a really good point uh the question here is like of like pmy uh like how do you uh what happens when one word has the same morphological form uh but has different semantic meanings uh for example you could imagine that like the noun form of this is very different than the verb form or maybe that it just has multiple meanings um part of that is answered that like you you live in this very high dimensional space so you can actually point in lots of different directions at the same time that takes care of some of it but uh another really interesting thing is that you can actually go through and pre-process this and change every instance of one token with like oh I recognize that it's actually in the noun form and I'm just going to say uh currency just in a noun form and then uh just literally add like underscore noun to it and then learn a new word of learning on that which is actually a blog post that came out by S by I forget his name now but the Spacey IO folks who actually talked about sens Toc uh like last week or two weeks ago it was really hot on like Hacker News but if you're interested in that I would definitely look up sens ofc they try to take care of exactly that kind of problem but otherwise it's not explicitly handled okay so I'm going to keep moving on uh definitely like running a little bit slower than I wanted to but definitely save your questions for the for the end from now on sorry uh okay so what I really want to do is now I've learned these word vectors let's let's start adding them together with stuff that I have uh at Stitch miix right so in this case I want to take item 3469 which I don't expect you guys to know it looks kind of like this and I want to try to add the word pregnant to it right and so people write back they get particular items of particular pieces of clothing particular pieces of fashion and they write back their reactions to them and in this case they might say you know I really love the stripes in this and the cut around my neckline was amazing uh but you know someone else might write something about it being uh gray and black maybe they like that maybe they didn't but either way word DEC is going to start picking up on that subtlety and that Nuance in the language and it's going to start associating that with the token for this item and so you have lots of this this sort of interaction of order a Wikipedia amount and far too much to actually sort of manually annotate but this is perfect for like a word deac algorithm that's going to just run over all of this text um okay so fine so we add these two word vectors and the result is we get these other word vectors are really similar okay but when you look at what items those represent you get you get get this and I was sort of a maze right like this actually worked really really well I found other items that were black and white and gray and Stripes but like you can see like the model is wearing a maternity bump right so these are explicitly made for our maternity line and they're safe for expecting mothers and you can see even the item on the far on the far right uh you're going to have sort of similar tones and similar colors but this time it's flowy and it's great for expecting mothers okay so um there's a bunch of stuff I didn't really mention I'm going to skip the slide uh but it's a lot about how to actually do the stuff at home um there's a lot of problems with dealing NLP it's still really messy you got to clean it up databases kind of suck at like looking at like word vectors and calculating differences and doing linear algebra a database it's kind of terrible and you can do it but it's not fun um and you know like any machine learning algorithm you're going to have false positives so now I want to talk a little bit about LDA uh and so this is uh this is LDA run on our data set right so we have lots and lots and lots of written about a particular every single item and so some of our items we have more words than than the collected works of Shakespeare right and so we just know the crap out of all those attributes right and and so that means that and so if we take that space and we project it down to a two-dimensional space so this is a tne uh dimensionality reduction diagram uh you can look you start to get clusters in it and so this is one of them right so this is the sort of the clunky jewelry cluster and somewhere else you'll see a dangling delicate jewelry cluster and so we aren't looking at the images at all here right like or rather the algorithm didn't look at the images at all these are things that clients describe like in the same sort of like most cohesive fashion like it's like sort of contrast y big clunky jewelry uh another cluster over here is going to have topics on patterns on Styles so this cluster is going to be um tops that are all similarly described as being sort of high contrasty and with colors that sort of Pop um it's really cool that you can see the space right uh you know no longer are these things like completely disparate like items they actually live really close to each other um you know maybe like you want some bright dresses for like a warm summer night uh and when I was writing this we had just launched our maternity line so this is why I'm talking about it so much but this is a maternity cluster right and so you can see that people who maternity talk about it in a very specific way because you know uh when you have a baby your clothing needs to change very very very quickly what's really neat is that it's not just visual topics not things that are really readily apparent like zippers or like um or like colors or different kinds of fabric you also get lots of topics about fit right so and that's very very very very important right like how someone describes how their neckline how it fits on their arm how it hugs their midsection or maybe it flows out really wide is really important and it's very much catered to every single person's body it's going to be completely different capturing those topics is extremely important to us and is how we try to deliver a personalized service okay so I love both word DEC and LDA but I really want to talk about how how actually they're really kind of similar but they come from completely different communities right so when you look at the word of paper you get this sort of like big giant neural network diagram it's got like a bajillion different parameters going on and lots of different uh and you know you look at this you're like man that's why it's so awesome it's got so much crap going on and then like uh and then like the LDA like graphical model over here is the complete opposite and the basian love to do the complete opposite it's like look this is the simplest most elegant model I could have possibly written and uh you know it's got a whole five letters in the whole whole thing right and both cases you have no idea what's going on but uh but you know it's I think it's kind of neat to you know they're actually more similar than they are separate or different cool so I want to talk about how word DEC is actually a local algorithm and it treats the whole world like it's one very very long text string no end of documents no end of sentences and it's just this window sliding across words right so I have one word predicting another word but you know your te your text probably looks a lot like mine it's actually organized in some way I have some client comments column uh and you know know I probably have it indexed by something and in fact what LDA going to is going to do is it's actually going take advantage of that index to try to predict all the other words inside of it so in LDA you have documents that are globally predicting words inside of that document or sorry you have a document vectors that are trying to predict all the words inside of it and so this makes a really big deal when you get to things like less so like client comments like this but much more so like when you get to legal documents or medical documents which are tens of thousands of words and then the difference between word DEC and LDA is huge another big difference is how you actually represent the vector word of is going to be all real values you're going to have numbers like minus. 75 and plus 2.2 um the LDA Vector on the other hand is going to be is all going to sum to 100% And one of those values is going to be 78% maybe another 11% here a 0% a 9% to visualize that word is sort of looks like this right it's like bar chart's got some things going up something's going down but the LDA Vector is a lot simpler to visualize because it's got very very few components the way to think about this is the word defect Vector is more like an address it's kind of like 200 Main Street 200 Main Street is not twice that of 100 Main Street right but you can figure out a lot about 200 Main Street by looking at the neighborhood around it something similar is true of word DEC right you can figure out words that are similar to that hold on I'll I'm running over time I'll get your questions at the end sorry so so it's much easier to say to another human being that this topic Vector is 78% of something but it's it's really hard to say that in a word of vector that it's you know this word Vector isn't plus 2.2 of anything and it isn't minus 1.25 of anything else but the LDA Vector is 78% of something and it's a lot easier to interpret that and this this this starts to get a really big deal when you go to High Dimensions right and word effect is going to be very high dimensional and LD is going to be uh just as high dimensional but it's going to be sparse that means that like if you can kind of barely see it it actually has like some some mass at 9% some mass at 78% some at 11% and the most of those other components all 97 other components are going to be zero or really close to zero word on the other hand is going to say oh it's a little bit of this a little bit of this little bit this and another hundred like little bits of this right so it's really hard for human to sort of interpret what all of those numbers actually mean on the other hand it makes it extremely flexible right you can be similar to another word Vector at least for a 100-dimensional one in aund other ways on the other hand LDA document vectors have fewer ways that can be similar or separate but they can be far more interpretable and I'm going to argue that that's because of two big critical properties of algorithms one is that it's a mixture and two that it's sparse sparse means that it had most of those elements being zero mixture meaning that you can decompose it as a sum of other things okay so the question is can we actually do both can we try to combine the best ideas in LDC or uh the best ideas of where DEC and LDA in LDA de so this is a bunch of experiments I ran over Christmas um and it was super super promising uh and and it's been really fun to talk about them since but definitely take them with a grain of salt just experiments so this is our typical table at Stitch fix we have word DEC it's trying to learn some word from some other word nearby ldan is trying to use that document Vector to predict all the words inside of it and but you get these really really awesome topics out of it right so if she says uh you know I love finding new designer brands for jeans you know that that document is 80% in the high fashioned topic and some other one says you know I love love love love the color the pattern and the flowiness you know that that documents 60% the style topic that's really really awesome because then I can go back to my CEO and say oh look like the number of like people talking about high fashion has been changing a lot over the last few months we really should order some like more High fashioned clothes or something like that right this is the kind of like impact right like this is how you can sort of steer a ship and steer a business by having interpretable machine learning models something that like is a lot harder to do with word not impossible but hard and this is especially important for things like fashion because Trends and styles are rapidly changing and it's a really fun problem to try to stay ahead of them all right so again word DEC is predicting locally in this case the word awesome is trying to predict the word top LDA has some document Vector trying to predict that one word this top so the question is how do we actually combine this right can we actually try to have predictions features coming from the document vector and from another local word vector vector and you know I think think the answer is yes and we're going to do this by uh just saying like Okay predict the the output word given the input word plus some sort of like document Vector right so we're going to have we're going to combine that input pivot word with some document vector and that's going to be nice because that document Vector is going to capture these sort of long distance dependencies and that word Vector is going to capture very much the short distance dependencies this is actually really really similar idea to paragraph vectors and Doc and that actually means that it we actually know it works and uh unfortunately though that document Vector is extremely uninterpretable you know it kind of looks like this and it's about as interpretable as a hash right it's kind of just a bunch of numbers it's kind of an address um but it's hard to tell on the other hand what I really want is an LDA Vector that looks like this most of its math is in two or three components the other 90 some components are zero and now I can actually come back and say to my CEO like look this document is this percentage and this top topic and you have 30% of this topic and 70% of that topic and I can start to analyze that I can start to do all kinds of like bi on it great uh and so what we're actually missing are things like mixtures and sparsity and we're going to do this by taking this document vector and we're going to try to make it into a mixture and it's actually pretty simple to make it into a mixture you say hey that document Vector it's actually going to be decomposed into a bunch of other word vectors and so the intuition here is that earlier we saw that you know Hanoi is Vietnam plus Capital right so we know that we can tear apart Hanoi and tear it apart into two at least at least two other word vectors Vietnam and capital and we know that Lanza can fall apart into Germany and Airlines so what I'm really hoping here is that that V topic one will be my Vietnam and that V topic 2 will be my capital and that if this document is about Hanoi I can decompose it as those two kinds of things so we know that document vectors can be additions of word vectors when I did this uh you can start to look back and say okay well fine I figured out what that topic Vector was um what's it really similar to and you can start calculating that oh it's similar to trinitarian baptismal Pentecostals u b which be possibly which turns out to be a monk schismatics uh excommunication oh this must be the religion topic and I ran this on the 20 news groups data set which is sort of a canonical NLP data set that comes into lots of packages okay let's look at what the other topic Vector is similar to it's mosovich absentee Indonesia lebanes Israelis oh this must be the sort of like a politics or maybe like International politics topic and what you get at the end of the day is that those individual topic vectors look basically like word vectors and then those purple coefficients the A and the B there are going to tell me how much of that document is going to be in the religion topic and how much of that document is going to be in the politics document and once you interpret what those topic vectors are you can start to figure out what almost all of your documents vectors are and so that's why mixture models are so powerful for interpretability so to be clear each one of those documents is always going to have the same sort of topic vectors the same components fixed but the weights for each one of those documents is going to change so everything's going to have a little bit of like religion and and everything's going to have at least a little bit of politics in it but you know if it doesn't actually have that much politics it's going to be pushed down to like 0% or like to 1% or something now the first time I did this I failed to make the those coefficients sparse and so I was getting things like oh it's plus. 75 in the religion Direction and minus 1.25 in the politics Direction I was like what the hell does it mean to be minus 1.25 in the politics Direction the math works out right like I can add and subtract word vectors but I can't you know intuitively it's really hard for me to figure out what's the negative of politics right what's the negative of Canada what's like the negative of like what other random word right um and so when you when you start to look at this and if you have a hundred other topics those individual coefficients start to look kind of like this uh you kind of just have this big mess of numbers and it's really hard to figure out what's actually going on it's hard to figure out what's happening when you have more than just a few coefficients so we can there so this is a trick that we can steal from the basian we can say hey look I want these coefficients I want to place a prior on them I want to regularize them in some way I want to say hey look these coefficients should be drawn from a derish lay distribution I'll explain the details in just a second but basically what this is going to do is it's going to make everything sum to 100% so right now these numbers are likeus 1.25 plus whatever and they should all be percentages so between 0 and one and it's actually going to do another nice thing for me it's going to penalize lots of nonzeros here so instead of having this distribution of coefficients looking like this which is trying to say minus point plus. 75 in religion minus 1.25 in politics and then there's 98 other topics it's actually going to say something like this right so now I have tons and tons of zeros and now I can go back to my CEO and say hey look this document could have been in a 100 different topics instead it's like in these four or five topics and nicely enough these four or five topics add up to 100% right so that means I can say yep it's 20% in religion 40% in politics and you know 8% cars or something like that right and so this is what it means to me anyway for it to be interpretable this is a lot easier to figure out what it means this is a lot easier to do analytics on it's a lot easier to share this it's easier to put this in a database uh all these nice things okay so now I'm going to go and dive into like the technical details so we're going to start off with the the skip gram model so remember way back when we were saying hey this this pair of like word and context did it come from a real data set or did you pull it from like a randomly sampled like random word and what we're going to say is like we're going to start to decompose that context Vector that's the C there and so that's the pivot word and we're going to say it's the the addition of that pivot word that's that first W plus the sum of all these components and like the projection onto those components so the context is the green thing the bases are like the purple vectors the weights on those bases are like the black coefficients here and all these things are going to live on the Simplex they're all going to add up to 100% And so this is actually going to look kind of a little bit different than gum so if you guys are actually familiar with a lot of like neural network models this is actually really really similar to an attention model uh attention model that's like a little bit more explicitly like uh durish lay and sparse and so the idea here is that like that V Zer is going to be religion or politics or something uh and then those A's are going to be disly distributed and so this is actually a ridiculously simple loss function right this loss function basically when you add it makes your stuff look a lot more like it's going to add onto a Simplex it's going to make it a lot sparer and and it's so simple that I really think you should that we should be doing this more often in deep learning right because it yields highly interpretable results and it basically transforms you know my nasty sort of like high-dimensional like vector that has mass all over the place to this thing that's like very sparse has a little bit of mass in a few spots and in those spots I can interpret those spots and tell you what it is great so did we actually accomplish the objective that we wanted to so in this case we have like back to the this database table and I have all this really interesting information right I have all these words they're sort of grouped up by document word DEC is only going inside of that one field LDA is reading it like using the fact that like these things are grouped by document uh and LD is going to give me uh pretty awesome results here right like it's also going to tell me that like oh this document is high fashion it's going to tell me just like LDA that this document is 60% in the style Direction but what's really interesting is that we can start to add more and more and more features just by adding terms to our contact Vector so in this case I might have a zip code like feature right so that makes a lot of sense right like this client is maybe in Texas uh you know maybe maybe they're like in Vermont uh but if they're in Texas you know they're like I should I should be able to predict the next word coming up is probably going to have something to do with hot climates if they're in Vermont there's probably going to have to do something with outer wear if they're in San Francisco maybe I'm going to start talking about hoodies all the time right so I want topics all of these over all these over all of these different zip codes and it's as simple as just adding another component to like the context or what I'm calling context right so in word DEC context was just one word but in LDA DEC I can take that one word and add a document vector and I can add a zip code Vector to it and then I can decompose those vectors to give me topics over all of zip codes and I can decompose that client Vector to or sorry that uh that document Vector to give me topics over different documents doents so we can keep going and going and going on and you know like maybe the next maybe the next column in my my database is client ID right and so uh and so here's a question that every business wants to answer what are what are the different kinds of clients that I have right I would love to know that client ID 5977 is like sporty I would love to know that someone else has is always really interested in casual aware and this is where if someone says that you know if she's in her third trimester we can identify future mother right and if she says that she's in Scrubs we can identify that she's in medicine and so this is as simple as just adding another feature to our context Vector what's really neat and something I don't have a ton of time to talk about is that you can also make sure that these topics are supervised so they predict some sort of other outcome normally we're just trying to predict the next word in the sequence but we can also make try and make uh make it predict like some outcome like uh like whether that item was sold or not or maybe uh maybe Revenue went up last month and I would like to know you know what these topics how they correlate with revenue and how much they um and so that helps us align topics with our expectations right I would love to be able to come back to my CEO and say hey look uh you know last last month like Revenue went up or went down maybe because uh FedEx screwed up all of their packages you know if I don't have any FedEx data inside of my database how am I going to learn that I'm not but you know what all of our clients are writing back and they're telling us systematically things like they telling us like what went wrong what went right all we have to do is listen in a quanti in a quantitative way right so we can do that now right we can now start to like to understand that data we can start to understand oh there were a bunch of package issues in January maybe that's like what caused Revenue to go up or Revenue to go down and that's what it means to have really interpretable machine learning algorithms we can action off of them easily okay so LD ofc is developed out in the open uh you can go to ld.com this is some documentation it's showing that it's like out there in a l uh it's not great documentation it's uh it it has a lot of like reference and sort of API references there's some examples up there uh but it's not it's not totally uh it's not totally it's not all right so I'm gonna I don't I don't actually have this on my screen so this going to be a little bit difficult to actually look at but one of the things that I think really holds oops what that really holds back NLP in general is the inability to visualize it it's really really easy we have a thousand different ways to uh visualize numerical data but with text it's really really hard to show different uh visualizations for it and I think this gets us a little bit closer so this is like one of the products that you get out of uh out of LDC uh and it's actually this Wonder wonderful wonderful package called Pi LDA viz and just and it's also an R called LDA viz and so like a let's look at some of these uh let's look at some of these topics and we're g to we're going to definitely push my ability to read at a distance um all right so let okay here we go so here's topic 28 right and ahead of time I've picked that there should be 50 topics and oh I forgot to actually mentioned that this is actually run over The Hacker News client uh comment Corpus right and so uh so let's look at like what the most popular words in uh in Doc in topic 28 it's like oh it's declarative programming procedural Programming type classes St type classing oo languages FP language pass okay this must be like a programming topic of some sort like you can imagine like new articles like about schola or about like uh Julia or pick your favorite language they might be classified into this topic right so like let's pick uh let's pick another topic uh let's go okay here we go topic number nine good recruiter Equity more Equity option pool total comp current Sally exploding offer okay all right well I that's definitely about like you know late stage interviewing and you've gotten some sort of offer and like there's lots of people on Hacker News that are like you know worried about that uh what's really neat is uh this tool lets you actually slide between like extremely like rare words that show up a lot on that topic uh and balance that out with like extremely common words that also show up on that topic and so you'll actually switch between this is really important for things like medicine where you have extremely rare words all the damn time like some random medicine that you've never heard of and if you change this lever it'll make a little less sense with Hacker News but like if you change this lever from medicine you'll go between these like very very esoteric medical words that only show up in that topic and you'll change it more to like symptoms and conditions and things that like you and I can actually talk about so it's a really useful lever to switch between rare but extremely in that topic and like more common but a little less concentrated in that topic so what's neat is that you'll get a diagram like this every single component right so you'll get this over all your ZIP codes over all your client IDs over all of your documents right you get one of these diagrams for each one uh and so that lets you understand the rest of the features uh in your sort of in your data set all right let me try to get back cool all right so that's all out in the open it's on GitHub um there's some examples there's API docs uh it runs on the GPU it's uh up until yesterday all the unit toss were passing so that so um have that as you will uh but I think here's here's here's the big takeaway right so if you want really human interpretable document topics use LDA it's a great model it's it it's in virtually written like in every single language you can it's there's lots of like scalable algorithms for it if you want machine usable Word level token level features use word de uh and if you've exhausted both of those those things uh I would use LDA DEC right uh you know I would I would definitely use LDA and word DEC long before LDA DEC uh but if you really really really want topics over users over documents over regions and you want to sort of like supervise those features and you have a GPU and you want to waake for a few days uh then I would use LD back um quick example here is you can actually do this very very quickly it's like only four or five lines of code um I realize youall probably can't actually read that but it's instantiating the model and then it's declaring the fact that like you're going to have a bunch of these extra categorical features you're going to have features in this case is The Hacker News model uh that you're going to have features over different stories you're going to have features over different author IDs um so on so forth um okay so this isn't this isn't where this the story stops either uh I think you can actually do this for LDA to lstm so in this case everything we were talking about today was a big bag of words model but it' be awesome if I could just say give me a sentence that's 10% about religion and 80% about politics right and then just the lstm just goes forth and spits that out if you guys have seen like lstms and deep learning neural networks type stuff they're really impressive really cool and they can spit out really cool sentences um another thing is that we're working on our team um at data Labs at stitchfix is TJ Torres who's a good friend of mine he does a lot of like uh like deep learning on images right and so fundamentally the only thing that we really did in LD was compress a vector uh into a particular set of like very sparse States you can do the same thing over images and wouldn't it be really cool if we could say this image is like 10% uh stripes and like 5% this and then like another 25% like zippers or something although maybe that's a bad example but like you you know it'd be really neat uh and he can actually generate these images right from from uh from a particular Vector so it'd be really cool to just generate new images with slightly different characteristics and I think that moves us into the domain of generative fashion right we can start generating new clothes uh we can start saying oh I want I want something that's kind of like this maybe the same colors maybe the same fabric May the same patterns but I want it to actually have more stripes than the last thing or fewer stripes or not Stripes at all and so I think it'd be really cool to do interpretable models with uh within like this deep learning like variational Auto encod or images framework cool all right you guys have been an awesome audience I apologize for keeping you guys so late um and I'll I can try to answer some questions of it all thanks [Applause] St on the right I guess and LDA algorithm how do you determine the number of topics yeah determining number of topics is always hard it's um it's it's basically the same game as you have in LDA you know maybe you do some maybe you do some elbow plants maybe you measure like perplexities um perplexities are like how you sorry the the question was how do you measure the right K the right number of clusters the right number of like components in LDA uh the problem with like things like perplexity measurements is that they don't actually correlate with how humans perceive the different topics and label so there's a really awesome paper of uh where people uh look at the LDA scores that come out you can say oh this model with more components fits better but uh but but different humans will disagree with that notion despite the fact that the score is getting better so there's definitely something to be said there about um actual LDS score is not working that well which is the usual proxy for determining what K should be yeah what's your approach to INF are you able to hand um I mean you handle autov vocabulary words this the same way as you do in word de which is not really um you know like you can continue you can you can basically try to restart the algorithm and add and inject some new tokens and then try to train them from your um I mean they're trained jointly so like the word Vector in in an LDA component is the same as the the word component yeah I'm not sure oh no no so in this case you're going to just use gradient descent the entire time so it's neither uh yeah this is at all like not at all like a yeah I gu so so LDA is usually solved like through em approaches or like variational inference approaches uh this is not that at all like uh this is setting up an objective function and then minimizing it yeah uh that's a good question the question was like you add more and more context vectors uh do you find that the word vectors change in some way um it's a good question I imagine they do um oh yeah I would definitely I would maybe I don't know actually I haven't actually looked uh Adam have you thought about using have you thought about using this with andimar DET uh that would be cool so question is should we word of X been applied to like these graph algorithms so you can do this thing where you usually have a window sliding across a sentence and you say predict this word given the next word but deep walk as Adam's talking about um is more about like let's make a sentence out of individual users who are connected in say like a Facebook Graph right so I might be friends with Adam and then Adam might be friends with Dave and Dave with Alexi and so now my sentence is Chris Adam Dave Alexi and as I go across it I'm moving that window and I'm trying to make those word vectors for Adam and Chris and Dave all be really similar to each other uh anyway it's a really really cool algorithm but the question is that like you could probably add context in some way to those things uh and try to figure out different communities it would be really cool because the communities you would find would be sparse and they might be disly distributed and that might lead to like sort of interpretable communities so maybe you would find I don't know like the young people community in Facebook and like the old people or like maybe you if you ran it on just mine like you wouldd find my high school friends and my college friends but I don't know it'd be neat to go do that it's a good idea oh Jesus all right uh okay I I'll jump to this side of the room yeah of your previous examples we were uh labeling topics with words from ourpose yeah uh one detail that I didn't quite get is are we constraining the topic vectors to be lar combinations of word vectors from the Corp we um yes so there's two questions there one is uh are we constraining the topic vectors to be a word vector or comination or liar combination so no you the the topic vectors live in the same space as the word vectors and they are essentially other words in my vocabulary uh and then my document vectors are going to be linear combinations of those topic vectors is that is that your whole question yeah Vector combin a so a document Vector is a linear combination of topic vectors A document Vector is a let me see sure I said that right a document Vector is a linear combination of topic vectors and those topic vectors live in the same space of word vectors but are not necessarily exactly the same as a word Vector so for example like if I went back um right so let's go back to this guy right so in this case um I call I'm calling it religion but I made up that label right so initially it was really just where did these Victors come they're learn jointly with everything else yeah yeah anies uh that's an easy question uh no no I haven't tried I I I really should uh I haven't really tried like a lot of quantitative analyses of the stuff mostly because it's kind of hard uh the analogies is is pretty good a pretty good test but you know like maybe it would maybe it would be as good as word DEC or maybe slightly better or maybe slightly worse that's not really what I care about though what I care about is like having multiple contexts and like learning about those contexts like learning what this ZIP code should be is it like in the hot climate topic is it or learning about this client is she sporty is she client and so you can run LDA over those individual client Com or individual Commons individual columns but in this case you're learning them all jointly and you can add all kinds of extra things like supervised stuff so the analogies is good for word DEC and it's totally amazing but it's less useful for me trying to inform people in a business that's anle yes that's right yeah no you're totally right like it' be a good thing to check and see if like the the word DEC things still pop out yeah so slide maybe thats you show like how you ially follow sure so um let's pretend like we never placed that dally prior on this equation right so now we're just at this um and this is really really similar to paragraph vectors to Doc to V to document vectors and so what happens here is that those coefficients the a z and the A1 JS is that they're constrained to live anywhere in the real space so they can be negative they can be plus 93 and a half they can be - 17 and what happens is that when you let the model fit something like that it has a lot of freedom right and it finds that like and it finds that the really good model fits are like a combination of of of lots of different components of a little bit of this and a lot of that one a little bit of this one a lot of that one and it doesn't try to say it doesn't try to concentrate the components in just a few pieces now what happens when you say okay I want to take those A's and I want to regularize them in some way and I I'm going to do this with this fancy derish likelihood right and when that happens you're going to instantly penalize most of these things you're going to say look like if you are anything but zero my loss function is going to be huge and so that means that like when you get the results at the end of the day you get something looks like on the right hand side now that does mean that it you know you've added things to your loss function and that might make it a lot harder to fit the model but when you do finally fit it it's going to be a lot uh it's going to be a lot more interpretable because it's only in a few pits and pieces does that make sense yeah I'm about just in in terms of the approach the way that it sort of tries to push words that occur together to be similar similar vors intuitively that would me that you group words that Co together in the same yeah why I'm missing why I'm missing the power of being able to do analogies like I I don't I get I get what the algorith is doing I'm missing that's and that's a really good question uh the question is um great we saw this whole algorithm we saw how it works um where in that did we start solving analogies that is that roughly right so so what's what's interesting is that because you have these let me skip back to this is that somehow because of the because of the loss function that we set up we managed to have these linguistic regularities so something about that soft Max something about decomposing our currence Matrix something about saying these two things are similar in some way helps us get to these kinds of regularities and once we have these kinds of regularities we can solve analogies but the link between soft Max or skip gram negative sampling and these kinds of like linear regularities isn't totally clear to me isn't totally clear to me like I don't see why the this thing completely falls out of the loss function that we set upic um yeah some of it still does feel like magic yeah second question I does have to beor model so I don't no I don't think it had to be the question is does wec have to be a neural network model um I mean like the way I tried to explain it was it's really not like a neural network moded at all but you know I think like you can recast lots of different things I bet you you could recast LDA as a neural network model you can think of LDA as optimizing a document vector and word vectors too right you can certainly think about it like that if you want it's it's not usually how people use the language but you can okay so I know I'm way over time so I'm actually going to stop it here um and then we can uh actually have questions sort of like after strata winning the strata winning qu oh okay oh that's rightand how big do the Corpus need to be for uh that's a good question um so I try so I've actually gotten pretty reasonable results on small data sets like uh like the 20 news groups data set I haven't tried things smaller than that but that data set's like 2 or 3 megabytes of text I think maybe right around that um and it's worked reasonably well right so like basically well reasonably well is like there are 20 news groups in that data set you know they exist there if you don't tell the algorithm about it does it recover those topics and does it recover those usit groups and the answer is like yeah pretty much um you know you get some religion topics you get some politics topics so it's worked out reasonably well there um and then I've tried to scale all the way up to like The Hacker News Corpus which is like several gigabytes and it seems to work well there okay all right now it's last last question right okay awesome thanks guys a