data.bythebay.io: Chris Moody, lda2vec
Recording: data.bythebay.io: Chris Moody, lda2vec
awesome thank you thanks thanks a lot welcome and thank you guys and thank Neil and Alexi of course for organizing and having me here and inviting me here again it's awesome to be back so as we all know NLP can be the sort of messy affair because you have to teach a computer about the irregularities and ambiguities of the English language right so you have to teach the sort of hierarchical nature of grammar the sort of like sparse vocabulary but stitch fix when she says she's in her third trimester we know that she's pregnant all right and when she says she wear scrubs we know that she's in medicine in some way I mean she says she's taking a trip that means we can distill that into a fix for vacation clothing so before I start I want to talk a little bit about myself i'm chris moody that's my Twitter handle I talk a lot of a lot more about this stuff I also really like stuff that isn't weird to vac and lda like Gaussian processes I contributed the TC to the latest version of scikit-learn sorry all right we'll see how well I hold this at two inches away from my face I've also sort of active and in the deep learning community I'm a contributor to the chain or framework and a bunch of other stuff it's really neat so anyway any of the stuff piques your interest i'd love to also just nerd out on this stuff not just weird avec an Lda and right now i'm at data labs at stitch fix which is the most awesome job ever because i get to work on all this stuff in the context of like fashion cool so I'm going to talk a little bit of a word davec I'm going to talk about lda and then ultimately I'm going to try to mix the best parts of weirdo back n lda to make el día de Becque so why do we even care about weird avec there's gonna be a lot of word about talks today and I'm going to try and dive into the details of this algorithm and rip it on try to explain it as simply as possible but this example right here king- man + woman equals queen sort of blew open the whole NLP world and that's because you start to think that computers actually understand what words mean right so it understands that the biggest difference between king and queen is the same as the difference between man and woman named leave others gender in there right and how did it learn that I just learned that from a bunch of raw tax right we didn't then have some table in there that said yeah king is male Queen this female and we didn't look up attributes for that we just learned it from a bunch of words you know seen in like in a big context just reading through Wikipedia I think that's awesome and that's really really powerful I think that forms the basis of like more complicated systems later on great so the way we're going to work with weirdo back is we're going to set up some objective function we're going to randomly initialize all of the parameters and then we're just going to do gradient as I'm kind of similar to like how most machine learning algorithms go so and I'm going to try to do this without ever mentioning neural networks once right so I'm a physicist by training I don't know anything about neural networks and I think we can actually do this without having to invoke these giant architectures I think we can talk this much much more simply so let's talk about training you want to learn this word vector W from its surrounding context if you worked with NLP before maybe you're familiar with like n-grams and maybe a transition probability between this and Grandma this other end grammar maybe you've worked with tf-idf and you're dividing a bunch accounts together or maybe you've worked with lsi and you're thinking about these sort of co-occurrence matrices it's not any of those things or at least not explicitly any of those things instead we're going to try and learn this word vector directly from just the context around that and we're going to update just this word vector no intermediates every time we see some example and so one of our examples might be the fox jumped over the lazy dog and what we're trying to do is learn the vector representation for the word over given the context of the other words in that sentence so we try to maximize the likelihood of the given over and jumped given over and document or and lazy coming over and so on and so forth and this is a pretty simple assumption right this isn't some this isn't some like recurrence relation there's no state right it's a bag of words model and it's just going to take it and all those words could be shuffled around in their order and it would still wouldn't affect how word davec is being trained so what should this little kernel which should this fox given over actually be so we're tomas make law and we're trying to invent weird effects we should probably depend on word vectors in some way and we're trying to learn them so we're going to randomly initialize them and then we're going to try to maximize that likelihood given those those does word vectors so we're going to extract pairs of context pairs of context and target word vectors from that little window so if we have in this case we might have the word over as our context word and we might then say the given over and fox given over and jumped given over and the given over and lazy given over and dog given our so this is the innermost for loop inside of work tyvek there's going to be a second full of it but this is the innermost one right so in are so scarred I literally have like 14 look like those over every word in our corpus it puts that little pivot word right there and then goes a few words back if you word for word and then extracts all those pairs inside in the middle and then in the next iteration of our second floor loop are we're going to move our context word so before it was up over and now it's going to be centered around though and now Oh instead of saying you know the given over now we're going to say that given time Fox given that and jumped given the and over given them so this is at a high level what word davec is doing right it just has this little window that's moving across the corpus and it's going to extract pairs of words in that window and then it's going to try just train on just those pairs and those pairs are called skip grams right so a unigram is just one where to buy gram is two contiguous words or two consecutive words but this has the Skip gram because i'm kind of like skipping a bunch of words in-between my pivot and my contacts work and so that is sort of the high-level update path of word to back right it's just two for loops so that's why I think it's like a little disingenuous to have this big giant neural network it's like okay like it's a kernel and it's two for loops right we can understand this in a much simpler way all right fine so like let's talk about what that Carl is what should this like what should my probability of fox given jump to be you know what how should I measure the loss between w and seeing it boils down to trying to measure the difference between those two things we want to make them as similar as possible when we see them in the same context and we want to make the mess dissimilar as possible if we never see those two words together okay so in this case we're going to measure the difference between those two things as just a dot product between those two things you could have chosen you cleaning a distance if we want to get really fancy pitches and things like how I know this distance but it doesn't really matter here in this case we're just gonna use the dot product and I'll product is some pretty nice nice familiar properties if those two word vectors are very very similar like Canada and snow then that now products going to have something close to one if those things are orthogonal that dot products and be close to zero and if those things are really really dissimilar that dot product is going to be close to negative one at least if these vectors are normalized okay so that thing that dot product is going to go from negative 1 to 1 but what we really like to do is measure some sort of probability right we can't have a negative probably that's just that's just ridiculous right so we're going to pass this through a sigmoid at some point in time we're going to take that dot product to pass it through and what that's going to give us is when these two word vectors are highly aligned like Canada and snow or like fox and cat that's going to be on the far right-hand side of this diagram right I could be very similar and that means that the output of that dot product is going to push to +1 on the other hand if they're very very dissimilar like say Canada and the color red or something like that they're going to be very very dissimilar down here and so that's just going to take our high vector thing turn it into one scalar that goes from plus 1 s 1 and then turn that into another scale that there's goes between 0 and 1 now intuitively what this is really saying is did those two word that pair of words come from my data set or not and that's it's just a logistic it's a binary choice given this these two vectors I need to make a binary choice whether this came from my data set or not now unfortunately this doesn't work all right this doesn't work because I only show it except positive examples right like and if I only show a positive examples it's just going to label everything as a positive example and so I'm going to be wrong because I want to show the positive things but that means that things like Canada and desert and Canada and you know light are all going to be the same word vector they're all going to be really similar and obviously that's a wrong model and it's a wrong decision so what I'm going to do is I'm going to try to add some contrast and I'm going to try to add some negative examples and so this is the second part of like what weirdo Beck is actually doing it adds this extra term at the end of that loss function and it's going to say okay I saw that you had the example of Fox and jumped but I want you to rate that thing higher than Fox and say another random word like career right I want you to distinguish and discriminate between something that occurred in my data and something I just randomly pulled out of nowhere right so distinguish between Fox and jumped and distinguish between Fox and career that's what that negative sign says and that word negative and that w- is coming from somewhere in my vocabulary now in practice we might actually pull a bunch of like extra negative samples so we'll have Fox and jumped on that far left side but we might also have a bunch of other negative things like I really don't want you to really don't want you to emphasize that Fox and Korea are similar they're not I don't want you to emphasize that Fox an Android are similar or Fox and sunlight or similar so you draw all these extra negative samples and that gets you a lot of contrast and that forces you to discriminate between things that are similar and things that are not simply on the other hand and so we're going to compute this L function and then we're going to nudge all the values of all those word vectors such that this thing is maximized so that such that this likelihood is maximized and so that's all skip gram-negative sampling that's the pia the most probably the most famous word of a calgon there's three other actors to other algorithms and so we're going to take this loss function and we're going to optimize it okay so so it turns out it's actually kind of simple but we can actually even make it even more simple and this is a really really cool paper by levying gold break in 2014 and it's one of the most cited NLP papers of 2014 and what this is actually showing is that it's actually extremely similar to matrix factorization all right and it's really really really powerful because all that double four loop stuff turns out we can just write down one matrix and solve that one matrix instead of solving as stochastically as I sort of like streaming in an online sense but I want to sort of unpack what this mysterious PMI matrix actually is it's really cool because it's similar to traditional LLP so weird of X sort of came out from left field although the NLP folks I had are more familiar with things like PMI which is the point wise mutual information and it's this sort of information theoretic thing that's associate that measure the association between wnc like okay all right what does that actually mean it's actually super simple it's this right it's the number of times I have seen Fox and jumped together that's that top that's the numerator that's that C and W divided by the total number of words I have / and the denominator now the number of times I've seen fox by itself and the number of times I've seen jumped by itself okay so it's a bunch of counts / each other and the sort of mysterious k i'll come back to that but when I actually see this equation what I actually really interpret is something like this that top thing is just saying how popular is the combination of Fox and jumped okay now divide that by high popular is the word Fox and how popular is the word jump some words like the word the are super super popular right like that is like every other word right but somewhere it's like deoxyribonucleic acids going to show up like one in a million times right so if I see the combination of the and Springs like it's probably not because then springs are particularly well associated just because the word the shows up all of the time right so in this case this ratio tells me how much am I over-indexing basically how much is that the word though or the word Fox of the word jumped how much is that pair of weird sort of above average right and so if that ratio is above one and those two words have like have a kind of must be associated in some way if those if that ratio is equal to one it's like yeah those both of those words are pretty much that common and I would kind of expect them to show up that like frequently and if that ratio is below one then you're like oh those words are like sort of negatively correlated having that word over here means that that other word must never show up and so that's what weirdo Beck is doing this is what sort of more traditional p is doing but what all the only thing that really the most important thing that weirdo Vic does differently is it adds this K all right that is a critical critical k and that thing is saying i want you to emphasize really really popular words that's very very important because words are sort of zip flawed distributed right so i have lots and lots of words that only show up a few times the top 1000 words like if i restrict my whole vocabulary to just the top 1000 words i can probably explain like ninety-five percent of like what I'm saying right now of like Wikipedia right there it's just those top 1000 words but i really want to model all of like the rare words and what this is saying is like look if i've only seen that word 20 times i don't really want you to pay that much attention to it right it's a kind of a noisy word but if i've seen that word millions and millions of times yeah amplify that word by a factor of K and make sure that you model that word extremely well and so that's what we're tyvek is basically doing it's just giving more weight to frequent words and less weight to infracore its and that's why this method is so that's why I like casting it in this matrix factorization ways so powerful because you immediately start to see that over at the back is really just like a really sort of like a naive noise model on top of my words and that's what makes it so very powerful so the last few years I've been suggesting oh you should use Jensen you should use the spark word de Becque libraries but recently in the last few weeks I've started changing all of our word divx stuff at stitch fix over to just being count based stuff and so I don't know this is a bunch of code and I actually expect you to read it but you know now that 99% of word avec is just counting I can you know sequels really good at counting stuff right and so here's the query that I actually use at home to count how many words show up and in what context so here's the critical part right count how many times I saw a fox and jump together count how many times I side just fox count how many times I've just jumped right and so that's the query and then this is the whole math forward debecker right you can take that i initialize some matrix that's of the size of like number of word by number of words and then I'm going to plop in every single element i'm going to say how many times it fox and jump come together divided by the total number of words and then what's that ratio i'm going to divide by how many times i expected to see the word fox and how many times I expected see the word jumped and then i'm going to multiply by my magic constant k which is going to emphasize frequent words over in frequent words and that's it right i save that i take the log of that and then I throw that into SVD and any linear algebra library in any programming language worth its salt is going to have SVD in it and so that's it right so now this is awesome because I've just gone from having sort of like this big like pipeline of word divx stuff down to like fitting one matrix in five minutes on one core on one machine in like four lines of code right it's super simple it's awesome it works really really well and that's why that paper is so cool it's so important and one of the most highlight certain things okay so I've talked a lot about like how the word of a Calgary the works i'm giving you two different ways of solving it I want to talk just tiny a little bit about the results these are on the top three rows here are different word of act like algorithms that came before were due back and you had and so that top row is is sort of like a target query word like Redmond one of the words that are really similar Redmond and the caller Bert model that's that second the top row is saying Oh Conyers and Lubbock and Kane are really similar to Redmond okay I don't really get it I don't know what any of those words mean but like maybe but Redmond is similar to read when Washington okay it's cool that's like a slightly different spelling and read in Washington get in Microsoft okay Microsoft makes sense it's actually in redmond what about ninjutsu ninjutsu similar to ninja and martial arts insert okay awesome that's great that's super cool weird avec is really working so what's actually happening when we add all these weird vectors together we add King and man and woman so we're gonna so I'm going to try to show an example of what this math means it's going to help us build an intuition we're going to leverage that intuition later on to build el día de Becque so in this case I'm showing just two out of like 500 dimensions is basically as if we'd like PC ated we're a load of all of our word vectors i'm going to only show you four of them but maybe there's a hundred thousand here alright so i have king and queen and man and woman i'm going to take the difference between man and woman with a load of those weird vectors calculate that difference vector and it's going to be just man- woman and you load up the word vector for king and then I'm going to add head to tail the word vector form and woman I'm going to end up with some extra point I don't know where it is but I can mechanically add the elements of those vectors together find some viewpoint I'll now look through all of my vocabulary to figure out what of those nine hundred thousand words is actually closest to this thing and I might get the word queen all right so it didn't have to be true but it was and that's really really remarkable that Queen was like the resulting vector now it's really really cool is that direction is that direction between man and woman between Cain king and queen right in that direction encodes gender and it doesn't just encode gender for those four words and encodes gender across all of my words and encodes gender for between daughter and son it encodes that gender between aunt and uncle between men and women king queen and so you start to get the notion that like this is actually a regular linear relationship right that direction always means gender it always means that the more I move in that top left direction I get to be more feminine and of course I don't have one dimension two dimensions i have 500 dimensions so in fact one of the other dimensions might mean something like higher status like that could be the difference between man and King right and so this is sort of how I'm encoding or how weird noveck encodes words as these sort of regular relationships in five hundred different ways so it's really cool so we live in this vector space where addition and subtraction are semantically meaningful so i can take check and currency and get the corona the check crap that's awesome that's definitely the check currency right hanoi in fact the vietnamese capital lufthansa definitely a german airline right so this is super cool because now you get the idea these vectors are really just mixes of other ideas and mixes of other vectors that addition and subtraction of vectors isn't just like a no linear algebra thing it's like something that conceptually means something to me ok so one thing I really want to do is use that at stitch fix now stitch fix is a sort of personalization service you sign up on our website you tell us a little bit about yourself and we'll send you clothing and we'll send you fashion that we think is relevant to you in some way and you might like some of it and you'll buy it you'll keep it and you'll return the rest in a box and along the way you'll give us a bunch of feedback and you might say things like oh I really love the stripes and the cut around this really other stripes and the cut around my neck line is really amazing and someone else might write that this thing is like grey and it's black and so we're de Becque is starting to pick up on the subtlety and nuance in that language right and that really works really well because for some items we have several times the amount written for them then we do like the collected works of Shakespeare so we really really understand some of these items so okay so I want to take the word vector for this item and i want to add the word vector for the word pregnant all right so we had just launched our maternity line and we might get a bunch of other items like this and when and you know I don't expect you to know these item numbers but if I go back and look up like what they look like they're in fact stripes they are black and gray except you can see that the model is wearing a maternity bump right so they're actually safe from eternity they're great for expecting moms they have similar tones and are flowy just like the previous item so it's really awesome that I can do that kind of math on my own corpus with our own concepts with our own ideas so talk a lot about window Beckman I want talk a little bit about ldi because it's my second favorite algorithm and LD is very very good at summarizing documents in ways and so in this case I've shown up most of our inventory and shone like lots of images here and what I've done is shown that lda is effectively clustering these items just by using how people describe these items and the people will call this cluster for example is a clunky jewelry clustering somewhere else there's going to be like a dangling delicate jewelry cluster you know this was one on you know you'll 'extract op Ock's on patterns and styles in this cluster is similar described because these are all sort of high contrast tops that sort of just have these popping colors to them and that's how and I like this a lot because this is how our clients perceive our clothing right this is not how we think about this this is not some metadata that we build or some big table with lots of hierarchy and attributes this is how our clients think about our clothes and our clients perception of our clothing is much more important so you know maybe you want a bright dress for like a warm summer night and there's a definitely an Lda topic for that right so el dia helps us model these topics over documents in a very very interpretable way so I talked a little bit about what the differences between lda and weird avec are so one of the biggest differences is that they come from very very different communities all right so a machine learning side on the word de Becque side you get you get you get diagrams that kind of look like this right it's this big giant neural network with a bajillion different parameters and it must work because look how much stuff isn't it right the Bayesian is on the other hand are completely opposite paradigm they're like look how beautiful and elegant my model is there are only four letters in the whole thing right so I think it's a really big difference between these two communities but effectively they're really actually doing a lot of very similar things so we're tyvek for example is local it's trying to predict one word given another word nearby right it's not trying to predict aware there's a hundred things nearby it's trying to predict in other words with it like four or five things it treats the whole world like it was one very long texturing no end of documents no no sentences or anything like that and we just have this window that's moving across those words but if you're like me you probably have a database at home or at work and it looks a little bit like this you have a column and there's definitely ends to those comments they're definitely ends to those strings and you probably have it indexed by something right there's a document index and so what lda is doing is it's effectively using that index to predict all the words in it you're going to use that structure that document structure that you already have to try and predict all the words inside of it all right and the difference between lda and where'd of it gets very very big for things like legal documents or medical document when you have tens of thousands of words another really big difference I want to spend a lot of time on this is how they're representing the word vectors so typical word davec vectors are going to be sort of real line distributed I'm going to have numbers that go between minus point seven-five and it was a mistake clearly mistake to try and open up water bottle and talk at the same time and the LDA document vectors are all going to add up to they're all percentages right they're all going to add up to a hundred percent and I think this is actually a really really really big difference because it means that one is a little bit more interpretable than the other right it's much easier for me to say it another human being that this document is seventy-eight percent in one category than it is to be it's plus 2.2 in this and minus 1.2 5 something else where the is a little bit more like an address right it's like 200 Main Street that 200 Main Street isn't 200 times one main street but I can't figure out a lot from its neighborhood right like 200 Main Street might be in New York and I know the demographics of New York and I know that the word vectors around those word vector can be similar in some way on the other hand LD is more of a mixture model right it says that it is seventy-eight percent of some topic and eleven percent of some other topic and so that's a critical critical ingredient right and that's going to help us like interpret things and the difference between these things gets exaggerated when I go between five dimensions and say a hundred dimensions right now my hundred dimensional weird of a factor looks like this I can't go to my CEO and say yeah this is what clients are saying right that's what are you giving me it's like oh yeah it's plus point six in the first dimension and plus point two five in the second dimension and right like I can't really do that but I can totally tell her that it's seventy eight percent in this direction could've been in a hundred different categories but it's only in two or three and it's mostly in this one and just a little bit in these other categories it's really really really helpful for her when she's trying to run the business when she's trying to tell us like what are our clients saying is it shipping issues is it this other thing I want numbers on these things and that's how we can actually steer the ship and we can steer a business is through things that are human interpretable so the word about vectors can be similar in a hundred different ways right because they're hundred dimensional and they're very very flexible but the LDA vectors can be similar in far fewer ways right because most of those things are going to be zero and that lets us be a little bit more interpretable it's a lot easier for me as a human being to read two numbers instead of a hundred numbers and it's a lot easier for me to read percentages instead of like floating point numbers and I'm going to argue that if we want to mix these two things we're going to try to add mixtures and we're going to try to add sparsity and that's what we're going to try to do with el dia to make so you should try to take this take this this is a series of experiments so take it with a grain of salt I did them over Christmas I just recently submitted a paper on it but you know this is not this is not like super production grade work right it's but it's more like prototyping and playful and fun things so weird avec is predicting things locally right so if i have a sentence like Lufthansa is a german airline i'm going to take the word german and i'm going to try to predict the other word there are the words around it I'm extracting those pairs of pivot and target words and a moving window that scans across my corpus now what we really need though to make it more like lda is we're going to need to add a document vector and this is going to be really helpful so if we have the word German we might predict other words that are similar to German like French or Spanish right but this document might be about airlines so we're not talking about other languages and if we add and if we know before that we can add word vectors together right and if we know that we can add the word German and we can the word airline we're going to get the word Lufthansa and now my predictions are going to be a lot more accurate because I'm combining some sort of local context of the word German and sort of this global context that this document is about airlines in general then I'll be able to guess much more accurately that the other words around they are going to be closer to lufthansa there closer to arrow Lloyd and these other sort of German airlines right so we're going to have this latent document vector and it's going to be randomly initialized for every document in a corpus and if we do this we're going to get a very very similar layout to sort of Dokdo back into paragraph vectors and its really really useful it works really well it's good for training sentiment models it has really great sort of perplexity scores but it's not interpret right if I look at that word vector it's going to look or if I look at that document vector it's going to look like all of my other word factors and you know this thing is you know bios interpretative rule is a hash right it's not that helpful all right so let's make this into a mixture and to make this into a science and sparsity so this is how you make this into a mixture you're going to keep adding like a few more like layers to this you going to add a few more steps to this right and so now I guess I hey let let's say we had three topics and I'm going to have a document wait and you're in that document way it's going to be a composition of a bunch of topics and then waits on those topics so in this case it might be that my document is point 3 4 & topic 0 and minus point 1 and topic to that's the second column and at topic matrix and then the point 17 and topic 3 that's the third column and that topic matrix on the far right side so before we had you know a hundred degrees of freedom for each document but now we only have three degrees of freedom all right the word vector space is still 100 dimensional but we only have we have far feet we have it's far more constrained right and so the model better choose really good topics because i only have a few ways to summarize the whole document and so i want to talk a little bit of what those topic vectors are actually doing so if I rip out that first column of that topic matrix it's going to be my topic 0 and I'm going to compare it to a bunch of my other word vectors right and and I'm going to figure out which ones are the most similar to this topic vector I might get for topping zero that a most similar words are Trinitarian or baptismal Pentecostals schismatic sand excommunication okay all right I get it so like this must be the sort of like the this must be like the religion topic or maybe the Christianity topic or something like that okay so let's let's let's move over from column 1 2 column 2 so I moved over once and that matrix and now now my most similar words ARMA losev it's your absentee Indonesia Lebanese is ray okay this must be like I don't know that like the politics like a topic or something like that right and so that's how these topic matrices are working and so when I send earlier let this that this document weight is point plus point 3 4 must be plus point 3 4 times religion and minus point 1 times politics that's still kind of uninterpreted like what does it mean to be plus point 3 4 and religion and what does it mean to be minus point 1 politics like mathematically that works out but is that intuitive oh I don't know what it means to be negative politics I have no idea and on top of that that word vector is really dense it kind of looks like that all right so what we're going to try to do is going to try to make it we're going to try a softmax this thing and so that soft max is going to move everything from negative infinity infinity to proportions now we're actually back into percentages land or getting a little bit closer to el día like representations and so now instead of being point three for religion we're now forty-one percent of religion and now we're instead of being negative point one of politics where twenty-six percent of politics okay so at least that I can kind of interpret but now the problem is that like first time I did this I still got it was very dense right and so I would get that this one document vector is you know two percent in this direction three percent is other one it's eight percent of the seven percent of that five percent for a hundred times right you're like okay there's a zillion categories and it's really hard for me to interpret what this document is when it's spread out in so many different ways so that's why it helps to you know make it sparse so that's the this like the 3rd and final ingredient of aldi a Tyvek is to is to add sparsity so it starts off kind of like this this one document is spread out a bunch of different places but at the end of the day we're going to add this little like loss function and we're going to make it we're going to make it be very very sparse at the end of the day so that we ultimately only choose that every document can you know Beal and as few categories as possible and it turns out that that likelihood functions kind of written down the top on the bottom right is actually super super simple to calculate and I think it's actually so simple that if anyone's designing their own machine learning algorithms it's pretty simple to just toss it in and you get a lot of interpretability out of it all right you can take a really powerful algorithm and make it interpretable and that's something that's kind of lacking a lot of the time in the machine learning or especially the deep learning community like most of the deep learning results are not something that will help me understand the science and themes and topics of my business right they will help me give really good predictions but they're not really going to help me understand like what happened in December like why did the revenue go up or why are the revenue go down but interpretable topics might write or at least I'll give me like some insight into it and so the idea here is that will start off and it'll be you and you can see the sort of the pink or the red document proportion vectors I'll start off very very homogenous but what this loss function is going to do is it's going to encourage the sparsity and by the end it's going to be ninety-nine percent in one category and one percent in something else and then zero percent like and everything else and that's why sparsity interpretability and mixture models are so powerful okay so at the end of the day you're going to end up with something that's quite a bit more complicated than just weird effect by itself but it's going to achieve our goals right it's going to mix weird vectors and with these sort of sparse and interpretable document representations so now I can use some here's a fun part of the talk I can start to give you like results on sort of a hacker news comments so i downloaded the hacker news corpus it's like one or two gigabytes of just comments and text and you can find these kinds of really really fun relationships right so you get that you know you get very word tyvek like relationship the California plus technology you get Silicon Valley digital plus currency you get Bitcoin okay it's definitely a digital currency I think it's really cool that JavaScript minus browser plus server you get no J just you know server side note I mean still JavaScript but it's sitting on the server Mark Zuckerberg and Jeff Bezos both the CEOs and Facebook and Amazon that's pretty cool NLP minus tax had to do an LP example lp- tax plus images computer vision Snowden Assange both whistleblowers one of them the US one of them mostly in Sweden so it's pretty cool right like it's pretty cool that it picks that stuff up but you know if you had just done weird effect by itself you would have gone a lot of those results so I want to I want to talk a little bit more about like the sort of like the different topics see if I have enough time yeah so all of this code is up on github it's all and you can look actually look a lot of these notebooks as well and play through them and this one's pretty fun like this is the this is the ldo bike model run on on the hacker news comments so like we start to look at some of this and it's i'm using this really awesome library called pi LD Ava's which allows you to sort of interact very quickly with a lot of the topics and a lot of the most popular words in each topic so what's going on the on the on the left hand side here are I have 40 different topics and I want to try and figure out what those topics were earlier you're saying oh this must be their religion topic because it's got schismatic sand Trinitarians and Pentecostals well this is kind of how you figure out what the label of that topic should be you start to look at the most popular rights oh this is housing costs rent okay Oh housing affordable housing housing prices gentrifying okay this must be this must be something about like you know housing prices and rent prices have sort of skyrocketed lately must be something about that right let's look at another topic let's look at topic aight bitcoins mount gox denominated capital gains tax liability liquidity index funds deflation dwell again this must be sort of like a finance and like money topic right okay how can you love to talk about Bitcoin and probably talks about you know other sort of stock markets type stuff let's look at another one topic 16 so neutrinos tau quantum mechanics kind of computers in the black holes space-time pie calm aerospace right so this must be this is a little closer to my heart this is must be like a physics and science topic right that's cool okay all right here we go here's here's maybe a little more interesting one Scala jas julia rust generics go routines optional typing right said that this must be like these must be documents that are about programming or programming languages or computer theory or whatever right so this is so this is why lda is so useful right like because we can actually go back and we can start to like extract patterns out of these topics and we can start to understand like the themes and all of our and I love our data so and that's that's exactly what I do later on in this notebook let's go look at some cool examples oh here's a good one so I plotted to topics and I can show I know when the comment was made I know it was made in 2008 or if it was made in 2013 and so this is showing that like this Green Line is housing social issues affordability and rent and you see right around 2013 people started really talking a lot about like rent and what about how unaffordable the San Francisco Bay Area is starting to become so that's really cool right so like if I'm running hacker news like as a business like I really want to know what people are talking about it I really want to know what overall like arching trends are I really want to know what's focusing what's dominating that community and in this case it might be housing and ran right and so if i were really thinking about that like a client base like maybe i would try to engage with that community a little bit more or maybe i would offer like more housing specific like options to hacker news right so that that gives us a way to quantify and understand in a very interpreta bowei what's going on in the text of like the hacker news community that's there's a few other ones here oh here's a cool one all right so I plotted six or seven different topics all sort of about social issues like civil rights is one of them technology and in society there's one about privacy FBI wire typing this this the spike right here has to do with terrorism surveillance and constitution stuff and it's spiked in the middle of 2013 knows anyone know what happened in the middle of 2013 that would throw such a wrench into the how can use like community Snowden yeah exactly nailed it yeah Snowden Snowden released a bunch of released a bunch of documents at that time and so as a result you can see there's an incredible surge of interest in a huge amount of like a huge surge of just topics about that right so again if I'm running how can use like a business and I suddenly see a bunch of topics pop up like this allows me to understand it and and it's not clear to me how you would use word davec alone to do things like that right but lda and ldo devack by extension will give you things like that you will be able to understand of like the comments in your text a lot of your business cool let me go back so there's a little bit of documentation on el de tyvek on the web it's mostly just sort of API and reference documentation it doesn't have a great number of like narrative documents and documents but it's mostly because it's still sort of a fun prototype toy project but there are a lot of examples it runs on the GPU pretty effectively it's got a lot of unit testing although a lot of them are failing right now so take that with a grain of salt and so here's the question that a lot of people ask me about LD a Tyvek and it's like should I actually use it and most of the time my answer is no right like from the most part like lda lda runs certainly runs like most of the business at stitch fix and weird avec runs like the other big fraction of it right and that's because both those are extremely powerful algorithms both of them have very very robust implementations and spark and Python and are whatever but you know if you do want to sort of tweak around with your own topic models if you want to add a topic over different parts of the region maybe you think people in Vermont speak slightly different than the than Texas they certainly order different clothes right or maybe you want to add maybe want topics over every single client there's something that like everyone's very every business is always very interested is understanding their clients like how what are some clients saying are their stereotypes for each one of my clients or their archetypes rather can I understand them sort of more crude broad strokes okay maybe I should add a sort of like document vector feature on my clients right and so if you want start tweaking topic models i would start saying hey maybe use lda tyvek it's kind of cool it'll be very quick to like try and prototype new models but otherwise these other technologies are great and the technologies that are going to be the backbone they're going to be driving ninety nine percent of like your analysis cool all right so with that you guys have been an awesome audience again I'm Chris moody that's my Twitter handle where I talk a lot more about this stuff i'll be posting a blog post about this pretty soon on the multi-threaded stitch fix blog series anyway awesome you guys are great thanks you still have time for questions or am I out you guys are take a long time save you yes to both of those yeah so the question was this ld8 avec question was does LD a Tyvek take a long time to run this yes and does it and I would basically only run it with a GPU right another reason why I wouldn't actually use el dia tyvek anywhere close to like production without like really really vetting it without like knowing what you're what you're getting yourself into right I mean it could it could be useful it could be cool like once you've built like your model that's like specific to like your domain really addresses like your problem is very very well but I would not try it before first triangle da word avec because those things are highly optimized beasts by now right there's a really good infrastructure behind it yes it does take a training trait for example training the hacker news model took like two days on the Titan GPU right it's not it's not come out of the yard where it takes a while I would not like but if you like playing around with stuff yeah sure play around so if you see a little bit more about how the loss function for increasing sparseness for the document peckers Eric Idle grand versioning and yeah like in practice they score four goals without that sparsity parameter no you do not get the same results and in fact one of the ways that you measure things like word davec or word models in general is by measuring things like perplexity and so perplexity is basically how well did I predict the word given that the word over the word foxes nearby if you don't include that sparsity primary you'll still do pretty well on perplexity test you still do you'll be able to take those document vectors and you'll be able to Train sentiment models on them but they won't be interpretive all right and so and that's possible that's basically because you're moving from this kind of representation down to this representation now I think the other part of your question was how sensitive is too oh yeah so like with le a you know depending on the documents near corpus right if you're looking at a bunch of things that are like survey are those things like your actual corpus and um and so you can kind of twiddle this you know I'm a brand good sort of either increase or decrease so this is sort of work in the same way it's sort of a very proud yeah so this is literally the Dervish Lee likelihood of seeing those proportions so it's actually gonna be very very similar to the hyper parameters in el día so there's that alpha in there that's the concentration parameter that goes inside of like a normal LD a model that alpha is above one you in fact in fact don't encourage things like this you encourage things like this if that alpha is far below one usually people set it to one over the number of topics then you encourage like lots of sparsity but yeah the number of topics is important the number that the concentration is important in practice you you just run a bunch of different things figure out like kind of quality it's an unsupervised models unless you happen to have like supervised data to check it on and if you do have supervised data then don't bother using lda just train a normal thing classify around it yeah might we see something beyond stitch fix for preferences noce districts is the ultimate answer yeah I think like ultimately like preferences and things like that are inevitably very very long tail it's a lot of the conversation we have bats districts is how do we address personalization how do we understand it better and to me the way I answer this question is text and how people talk to each other is extremely efficient at like communicating those personalization preferences right it's it's not like a dialog box where you punch in like a bunch of preset options it's a very natural it's very fluid but it takes a huge amount of effort to get even like the most crude representations from what you're saying like this so ultimately you know maybe stitch fix is like the right answer for personalization but more importantly I think like text and text representations are actually like the most powerful way to understand personalization to understand your personal style understand very very long tail events so yeah like motivated long as you've really Olson and it makes the Alphans are very terrible but also like all kind of distribution models they are simple so some kind of liked insidious things it sometimes does select one fit as an example is you gave an example we're trying find similar words or synonyms and sometimes they can actually to like antonyms being soon samatha I a tasty pizza hey disgusting pizza like those two words are actually use personal way so they're like how you having to deal with things like this especially I production when you rebuild that you model your pipeline does educate employers production automatically do you have like it's a really good question so a question is that things like like weird avec they don't necessarily differentiate between things like sentiment they don't capture the valence of a lot of words and your example I think was like this pizza is really tasty here this pizza is really disgusting it will actually put tasty and disgusting really close in the vector space because they have very very similar contexts they appear in very similar content in food contexts I don't have a really great way of saying how to separate those things aside from like give it some examples and then train another model on top of that all right so like for us we know whether an item was bought or not Bob so we can kind of like hack it together and say oh maybe the words in that comment are all negative because it wasn't Bob it's still kind of crude another thing that we actually do is and didn't really go to talk about it today is that we do a lot of dependency parsing so this is actually building like a full grammatical tree saying this noun is being modified by this adjective in this verb and this is like the dependency tree over here and that word actually relates to this one over here and in those kinds of cases you can actually get negations and you can capture like whether something was being negated or whether something or what adjective was modifying what word whereas right now we're def echoes kind of just a jumbled up bag of words in those cases that you can start to add a little more subtlety a little bit nuance of like the sort that you're talking about now what's interesting is that in el día also kind of sub bears like similar problems we will get one of our one of our topics is it's interesting and one of our topics is you know the stylist really paid attention to me really well or the stylist like didn't write and so it's a and so it's either about you know it's definitely about attention about whether we're listening or not but it doesn't differentiate between good and bad right and in that case like I don't have a great answer like we just end up having to go back and saying like well was it bought or is it not bot and that's basically our label in this case and then we can but those topics by like yes like it she see she's clearly about being interested or being a you know she's clearly reading like the clients notes and everything else but we need to differentiate between good and bad basically just by weather and bought or not so most of the most of the time the answer starts long-winded answer when was the time my answers are like you have external data in some way