A Word is Worth a Thousand Vectors

Text By the Bay 2015: Chris Moody, A Word is Worth a Thousand Vectors

Recording: Text By the Bay 2015: Chris Moody, A Word is Worth a Thousand Vectors

welcome thanks for coming thanks for having me thanks for inviting me texts by the day great so as we all know and hope you can kind of or at least it's my opinion that kind of NLP can kind of be a messy in a difficult affair and that's because you have to teach a computer about the the irregularities and ambiguities of English language and this sort of hierarchical and sparse nature and all of the grammar but at stitch fix we use word vectors to help us learn from the raw text in customers notes and that helps us sweep away a lot of the problems a lot of these issues and so that means that when we're trying to deliver this personalized experience to our customers and our clients she says that she's in a third third trimester we know that she's pregnant and you know when she says that she used to wear scrubs to work that she works in medicine in some way and then when she says that she's taking a trip that we can get her vacation clothing right so that's sort of the promise and the power of word vectors and I want to start off by going diving in very very deeply into the theory of word Tyvek all right so now we've heard a lot of work to vex but i'm gonna try and take a wildly different approach to it and I'm going to not mention neural networks at all in the entire time so as a physicist I sort of tried to take things like as simply as possible and I get a little bit lost when we had lots of layers so I'm gonna try and take this in a very different direction and then I'm gonna talk about use cases at stitch fix so we use Lda and we use word effect both and then at the end this is my favorite part of the talk so if you've you're tired of hearing about where Tyvek you should totally perk up for this last part this is totally fun experiment so this is playing word to Veck having nothing at all to do with NLP and nothing at all to do text right so this is a birth defect on social graphs again no words word devack on Spotify playlists all all kinds of crazy things so it's really weird that you can take this algorithm you find deep connections to the rest of other fields but anyway before we start I want to talk a little bit by myself that's my Twitter handle Kristy moody and then of course if you do tweet anything please do hashtag TV TV I went to Caltech in physics I did a PhD in Astro statistics and supercomputing afterwards and now I'm a data labs at stitch fix and so stitch fix is this phenomenal company we do it really does this sort of high-risk high-reward research right and so for me that's great that means I get to play around with this fun stuff like Gaussian processes teason e-word devack tensor decompositions in factorization machines so you know aside from doing all this work Tyvek if you guys are interested in any of those techniques I'd like super love to grab some coffee and just talk about that stuff with you and of course we're hiring so if you guys are also interested in this kind of stuff talk to me about that as well great so do want to mention that large swathes of the talk are sort of taken from other presentations and from other focuses papers and these are much much these are you know established NLP guys tomash Mikhailov wrote word Tyvek and if you guys look at the deck after the talks you can click on these links and they'll go to stuff Christopher Ola is a teal fellow and he has this beautiful blog with these interactive visualizations Redeemer Herick who wrote jensen and the word Tyvek packaged in it we're leaving your Goldberg these guys are phenomenal these guys are academics that have tied the algorithms behind word Tyvek to sort of a greater body of sort of more established and LP techniques richard sojer who you heard a few hours ago has this great NLP class and Shinran who have these very lucid explanations of the word of a gradient and what it's actually doing okay but why do we actually care about weird effect and I think that top example which by now you've seen many times but just sort of blows everyone out of the water right so it's King - man plus woman equals Queen and that really means that the computer understands the biggest difference between king and queen is the same as the difference between man and woman all right so it actually understands what it means I can solve analogies and we're not treating words as blocks we're really instead modeling relationships between those words and it's great because it learns that pretty much from the raw text so we have to tokenize some stuff but it's not like building a dependency tree or all this like grammar we can sort of do it for different languages almost just out of the box and these are distributed representations and then that forms the basis for more complicated deep learning network so the other thing I want to say is that this is not a deep learning algorithm this is a very shallow algorithm and in fact I would even hesitate say that it's actually fairly simple right and that's a powerful thing that powerful simplicity let says go super fast over lots and lots of data and that's actually sort of like what I think is the crux of where Tyvek and so if you've been following the deep learning world you can get a lot of mileage out of pre train networks like Kaffee or coffee cafe I'm not sure you pronounce it or imagenet and you can try to do the same thing with word Vivek you get weird vectors that are already pre trained and if you don't have a super specialized vocabulary you can get a lot of mileage out of just using those libraries ok all right so the way that I'm going to spend the next few minutes is I'm going to set up this objective function for word devack we're gonna randomly initialize all the parameters in these cases the word vectors and then we gonna do gradient descent basically the same stuff that you do for a lot of learning algorithms great so the objective for word divock is we want to learn as a word vector in this case V in and we want to learn it from its surrounding context right so that's basically the model that we want to think about and if you're thinking about SVD or LSA or LSI you have this sort of this this co-occurrence matrix and you take that co-occurrence matrix and you try to compress it in some way using SVD or if you're thinking about engrams you think of this transition probability matrix but in this case we want to learn this vector directly so no intermediates no big matrices we want to learn this this guy immediately we're gonna do a by randomly initializing them and then sort of doing gradient descent on the objective so to give an example let's start with the fox jumped over the lazy dog and in this case we want to maximize the likelihood of seeing this context given the word over all right so the over is gonna be our central word here and we want to maximize the probability of Givet of seeing the word the given over of the of the word fox given the word over and the word jumped given the word over and so this is fundamentally what we're due back is doing at a very very small level like the most interior like you see a bunch of for loops and then right inside that last for loop is this thing right here all right so the and it's a really still really simple assumption you see right it's just given this thing try and predict this other thing so there's no other sort of secret parameters there's no other conditioning it's basically just this guy all right so let's think about this what should we actually try and make this probability we should make it we know we want to find the probability for the word Fox given the word over okay well we know that we want it to represent we know that we want it to be a word vector in some way so fine we'll rewrite this in terms of P V Fox and then V over and then we'll randomly initialize this stuff but here's the small twist that we're due back we're gonna have two vectors for every word and it's going to depend on whether it's the word that we're conditioning on or whether it's the word we're trying to predict whether it's in the input context or whether it's in the output context and so we're gonna have a context window around every word so if you go back to our example of the fox jumped over the lazy dog we're gonna start with the word over and that's going to be our V in and then our V out is going to be the word though and so we're gonna try and maximize the probability of seeing these two words and then we're gonna maximize the probability of seeing fox given over and then we're gonna maximize the probability of jumped but remember that in this one we're using V out for jumped and we're using V in for over and so okay great so we go scan across our whole context window and so that's one of our four loops and then you know we move over and now we're gonna train instead of over we're gonna train the word though and so in this case we're using the word in the input vector so it's a little bit tricky to think about this but we have two vectors for everything and then we scan across the whole sentence again and we say try to maximize the probability of the Fox or Fox given da and then jumped given da and then over given that so this is at a high level essentially what word do Veck is doing alright so those are the two fundamental for loops inside of a word go back alright it's not it's really not that crazy and that's why like why I feel like it's a little disingenuous to talk about all of these like layers I mean it's a neural network but like this is like I think a fairly simple like two for loops is a lot simpler to me okay great so how do we actually talked about this probability out given VN and we want to measure the loss between V out and VN and we want to make it such that V in and V not occur all the time together we want to make it as close together as possible so what are the different ways that I can compare two vectors I could think of sort of like the Euclidean distance I could subtract them and take the norm but in this case we're going to take the cosine similarity between these two guys and we're gonna say that okay great so that's going to give us a number between negative 1 and 1 ok but we want a probability and obviously we can't have a negative number there's a probability so we're gonna throw it into a soft max all right and so what is a soft max do at least it's gonna get us a number that's between 0 & 1 all right great so that's a little bit closer to what a probability is but what's off max really is at a conceptual level is it's making a choice out of n items all right so if you're familiar with logistic transformation or logistic regression you're trying to transform you're trying to predict one of two things in this case you're trying to predict one of n things namely the word out of your whole vocabulary right and so if you're a statistician the softmax is just a multinomial but essentially it's just a sort of an appropriate mapping given that you could have picked all of these other words so that's what it is at a conceptual level but at an algebraic level it's this guy okay so the top the numerator is an exponentiation of those two words again the cosine similarity but that bottom part is a pretty gnarly thing where I have to sum over all of the other words in my vocabulary all right so this is happening every time I do one of those comparisons when I do foxed and over and when I do jumped and over this guy is happening right and so this is a sum over all of the other possible words in my vocabulary and so that is essentially the probability that sits right in the middle of word debecker right that's it right this is the kernel of work to Veck there's not as you see there's like not that much else to this right there's we're just going to apply this operation again and again and again and again and then we're going to update the vectors so we're going to just do normal sort of gradient descent so it's a gnarly looking equation but basically we take the derivative of that probability and say yeah move that probability and whatever makes those two vectors closer to each other all right so I'm not going to go through the derivation of that Richard Souter in his class does it and wrong does it you can look it up that's it right like the rest of this is just going to talks on some it's going to tack on a few hacks that make a little bit faster but like it's really not that complicated enough if you ask me anyway okay so let's talk a little bit about the performance so that's just one the inner part of my for loop but of course that denominator is pretty gnarly it's of order my vocabulary and that means that you know for every single update I'm going to have to do V operations and then I'm going to have to go across my whole context but now I have to go my contacts Windows 10 words and have to go five words back and then do it you know and then go five Ritz forward and I'm gonna have to do that see number of times okay and then after I do that I'm gonna have to move one word for word and my corpus and if I have n words in my corpus my order is now VC and n okay all right well that's pretty gnarly howhow is that supposed to be some better than something like SVD right like it's a sort of sort of similar order of complexity okay but as any computer science student will tell you if you have a linear problem you can build some sort of tree and now you can get a logarithmic problem and you just have to pay the upfront cost of building that tree okay so this is this is now sort of getting into like refinements of word Tyvek and this is the hierarchical softmax right so that's all this is so now you build a tree that looks a lot like this guy and you have dark nodes and you have light notes and those dark nodes don't represent word vectors at all they have vectors inside of them but they're gonna represent a left or a right decision and what I'm going to want to do is I'm gonna start the top and then that word vector like say W 2 in this case might be the word that might might respond to the word jumped right and I'm gonna have to eventually find the path that gets me to the word jumped and I'm gonna calculate the probability of having taken that path so I'm gonna start at the top and I'm going to say okay well jumped like I know I was the bat is where W 2 and so if I want if I'm starting at the top and I need to end up at W 2 I need to take a left at this node okay all right so I'll write down that it's the probability of going left at node 1 and then I know that the next step I also need to take another left at node 2 and so each one of these steps is another sort of softmax step but in this case I'm not using a word vector I'm using this node vector and updating this node vector and then at the last step I need to take a right okay and so I've sort of glossed over some of the details here but the crux of the story is that this is now logarithmic a number of steps right and so if I have a vocabulary that's 50,000 words that's like 10 comparisons right it's not that bad now if I have vocabulary of yeah those 50,000 words this is 50,000 comparisons this is why you really care about building a tree would you rather do ten things or would you rather do 50,000 things so this is a lot faster and so yeah I've written out this so basically the the denominator of that softmax that we did out before would basically mean and multiply everything with every single one of those white nodes in the bottom and figure out like what the total sum is and normalized by that number cool all right so great so now our performance is substantially better and now we can essentially scale up to 100 billion words right so that's roughly like the highest I think corpus I've seen come out of word to think okay so that's the real trick of word of act that the fact that it can scale and just consume enormous amounts of corpora I mean ever anyone will tell you that more data is way better than a better algorithm and this is essentially just taking that and scaling it out and to put a hundred billion words in context a book is about ten to the five words a hundred thousand words and this thing can read 10 to the 11 words so you can basically read a million books in a day right so it's a lot and with that comes hugely which which which comes with just very well-trained word vectors right and so here's sort of sort of qualitatively asking like how good are those vectors and so you can see some of the other models colaborate model that's on the top left took two months to Train right compare that to the word vector that took a day right and when you look at this like what are the most similar words to Redmond and you can read down the column well they say it's Conyers and Lubbock and Kean and that makes no sense to me at whatsoever but then Redmond forward Tyvek is related to Redmond Washington okay great that's maybe almost trivial but it's also related to Microsoft okay well Microsoft is located in Redmond it makes sense and then you can keep going Havel is related to the Velvet Revolution he was the president that sort of presided over it and ninjutsu is related to ninja and martial arts and serbs okay great this is like not quite synonyms but it's pretty damn close and you can compare it to all of the other things in this table and it really doesn't like the other models just don't come that close like it's kind of hard to see why they would be that great okay fine but maybe you're not convinced by qualitative things and you can see that this is sort of a quantitative answer like let's throw some some and some syntactic tests in it and in this case we're divot blows everyone out of the water the all the word Tyvek numbers is at the bottom two rows are two or three times higher than everything else you see on that table and so there's two types of tests going on here one of them is semantics that's basically asking King is two queen as man as to what and its woman and in this case semantics means that you have a gender relationship and you can compare that to a syntactic tests which means that you have tests like run as to running as walk is to us walking in this case you're not testing gender you're testing something like a gerund form and so that's what a syntactic sense means and so you actually have two different kinds of worked effect models and only one of them is very sensitive to semantics things and it turns out skip gram is you know pretty good at both of them but it's phenomenal that word avec is actually picking up both grammatical meanings and also semantic meanings okay but I do want to dive into this like king- man plus woman example and so this is this is one of us is what we put up on our blog like a few months ago how do you actually do this kind of computation so you load up all of your word vectors king and queen and man and woman and so we're just showing two out of the dimensions so there are potentially 500 dimensions or however many you train with it's a choice you have ahead of time and you would get something like this if you PC ate it or did some other sort of dimensionality reduction okay so we start off by taking the difference between man and woman and I have the vector here reversed but for pictorials sake I think it's just fine and you take that vector and so this is now the difference between man and woman this is sort of codifying something that's like gender and then you take the word King and then you add that difference to it you get a new point out here at the end and now you search for all of the words that are near that point in voila the closest word of course is going to be Queen it's kind of by coincidence that it actually learns this kind of stuff right and so when we go forward and we look up great so we get the answer Queen and it just turns out that this direction is consistent and it's a regular thing across the whole space of all words so if we look at things like daughter and son and aunt and uncle they're also in exactly the same direction moved over by exactly the same amount and remember them we were doing this softmax thing earlier we didn't say hey look this axis that's going to be the femininity access right we didn't do anything like that at all it just happened that this crap just popped out by accident right or almost by accident and it's a really wonderful thing about we're Tyvek that actually does this okay great so that's like the more feminine axis but of course we have 500 dimensions in this space so now we have you know the higher status access maybe this is like the biggest difference between men and kings some sort of like more status kind of thing and so you can start to imagine that all these dimensions are codifying things like oh is this an adjective or is this a verb or all kinds of other relationships that can be present in the word vectors great so what's really interesting are examples like this check and currency let's add the word vectors for check and currency and you get the word Karuna right so which is the check currency or the check crown and and so when you look at those vectors it actually makes sense like they're mostly made up by these two other words right and so we actually living in this vector space where operations like addition and subtraction are meaningful right and so and so you can get things like German Plus Airlines you get Lufthansa or French and actress and you got a bunch of other French actresses so you can see that these vectors are really made up by other ideas and those ideas are being encoded in vectors in some way and it's really cool to think of this stuff as being mixes of other of other ideas and vectors so here's an example of this working in industry and in stitch fix and so we have a lot of text from our clients so I should probably rolled back and actually explain what stitch fix actually does we are sort of a personalization company where we have clients that come up they register on our website they say hey I'm this size and how anna stylist goes through your profile figures out what would work great for you and sends you a box of your own items or items of clothing and then the customer writes back and say yeah that fit me great I love the stripes you know but like the arms just didn't quite fit me totally well and so there's all kinds of subtlety and nuance in this kind of stuff but we get 20 or 30 percent of a Wikipedian in terms of text so it's a ton of text and so we have it associated at the item and so we can do things like okay take that text about item 34 69 and try to make it I don't know try to find like the pregnant version of that right and so to give you a little bit of an idea item 34 69 is is this top it's a black and gray top and when we try to find other things that are have that are more close to we're pregnant we get things like okay well I've list of other items but if you take that list of items and we sort of look at the pictures for it we get this right so we get these outfits that are perfect for maternity right there they're black and white there's they're sort of soft and they're flowy and they're safe for expecting mothers and this is this is has nothing to do with the images whatsoever right it's just how people are talking about these items so and it's great right we haven't really we haven't had any sort of like hard metadata we haven't had to like like manually annotate anything this is purely searching with the data that we already have and in fact like you get lots of different flavors for words so this is an example of looking at all of the words that are related to the word vacation so our clients are requesting vacation clothes all the time and weirdo vac is learning from that and so it actually comes in a lot of different sort of categories and clumps and it may be a little bit hard to see and it shows kind of awful colors here but you can pick things like there's you can see a whole thing of wedding words Bachelorette rehearsals holiday events you can see birthdays brunch and Christmas and Thanksgiving and spring and summer and these are so seasonal words and so each one of these things is sort of going off in its own direction and weirdo Veck is sort of encapsulating it yeah the vector for item 34 16 I never hear you or any of these items yeah so when when they when the clients leave back their feedback they leave it at the item level and so we can do something just as simple as either just dr. Beck or we can literally write a sentence where the very beginning of that sentence literally has the word item 34 69 written right in front of that sentence right in front of that comment and so then it just gets you learn from the context around that that item 34 69 has something to do with the comment that follows it so he learns like stripes or gray in black and white yeah and then feel free to interrupt whatever - that's okay so we don't just use word Tyvek we also use LD a in our production systems and so it's worth a little bit talking about when you would want to choose word avec and when you would want to choose LD a & Weerd of X I think is great for learning weird vectors and document vectors and in general is great for feature generation for machine learning models so if you're interested in trying to figure out if someone's gonna churn for example it's probably a great signal somewhere along the way because it'll have codified inside of that vector in some way that you know she's unhappy or that she's saying things that correlate with some sort of unhappiness or something like that LD a on the other hand is gonna give us a topic distribution for each document and it's gonna be great for tags and that's exactly what I want to show you next so whenever we have clients they might write in saying you know I loved every choice in this fix great job ok so we have the tags great style isn't perfect and they're this this kind of data is great for analysts that can measure the health of our business and measure sort of keep in track metrics but really our business is a lot about personalization and about delivering that very curated personalized experience and it's all about helping the stylists do what she does best so in this case we have the body fit tag and the customers has written my measurements are 36 28 32 if that helps I like wearing some clothing that is fitted I've very hard for me to find pants that fit tight and so as a stylist is trying to look through all of our inventory and trying to figure out what is appropriate for her she might wonder you know I have no idea if this item is gonna fit her great or not and she has to look through all the customer tags and try to figure out all of the customer text and try to figure out just the right appropriate amount of text and she can zoom right into there because she knows that this little snippet has to do with body fit and so it helps our stylist immensely it helps their job go much much faster so in this and here's another example I really enjoy the experience and the pieces sizing for the tops is a little bit too big looking forward to my next box so there's one of the tags comes up is just excited for next and we also have a sizing for tops attack so it's again like very very useful for getting just the sizes correct another sort of where we can get into really really granular forms here is that when she's right so it was a great fix love the two items I kept and the three I sent back really close okay so this is really really useful for us because on the back end she didn't send those three items it looks like an outright three rejections right but it wasn't right we got pretty pretty close here and so in fact like we just need to take this fix and sort of just tweak it just a little bit and that's what it what a stylist is good at doing but we just need to surface that to her in an intelligent manner and that's what Elia helps us do so LDI in fact is it can give us an incredible amount of structure and detail information right so now this is a TC diagram so this is we've seen it a few times today but it's essentially a projection down to two dimensional space on Lda on all of the item comments and so in each one of these cells is a picture of that style or of that item that we send out and what you see here is I'll zoom in on the clusters later on but what you see here is that there's a lot of structure a lot of clumps right so had you seen like one big blob like I wouldn't have said there's a lot of detail there's a lot of structure but instead you see a lot of little things everywhere and you can start to investigate them right so if you look at the cluster up there you see it's a whole thing of jewelry like folks talk about these things in the same way right so they're talking about this jewelry as if it's clunky big jewelry there's another cluster somewhere else where it's like thin delicate pieces this is not that question right in fact there's something recaptured we're capturing a lot of that granularity in the language I mean go somewhere else you can look at these tops right and so these are all sort of described in similar ways right so don't look at just the color but look at like the size the color the fit the style of it and these are what our clients are talking about and we find that these items are similar in the same language and the same is true for these dresses these are sort of bright summer and spring dresses and this is all sort of like very very showing a lot of like the structure and a lot of the detail that we can get out of text and of course we have sort of a minute we just launched our maternity line which is why I'm showing so many examples this is a this is sort of the maternity clustering and so stitch fix is a lot about doing sort of commendations that mix machine-learning with this human stylist component living at that intersection and so we sort of have this very net flexi sort of problem of trying to do recommendation engines and what you see on the left is a teeny diagram you're actually seeing the same thing on left and right but on the right hand side have replaced all the points for items with their images and what you're seeing on the left hand side is learning from just from the ratings matrix right so a zero to four rating right and then on the right hand side it's say okay now that you've told us zero for what you think about this item tell us like write us a paragraph what you actually think about this and so you can see there's a tremendous amount of detail in this right-hand diagram that doesn't exist in the left hand one even though there's sort of fundamentally capturing the same kind of information just the text is helding a lot more than individual little ratings okay all right well we're a recommendations company we would love to use this some way in our recommendation engine but but but I will move on to talking about sort of triggering actions from customer text instead all right so you know this happens to all of us we can get disappointed customers and clients to items were not received I know you folks do an excellent job and serve a lot of people typos happen all the time serve a lot of people some mistakes will happen just thought you'd appreciate the honest feedback what we do and we can take comments like this and we can vectorize them with dr. Beck and that gives us training features to then sort of train on and then stylists they're always going through every fix but they don't usually have the time to do all this stuff but they'll give us training labels and say oh something like this you may need to send that to a customer service agent who will figure out what went wrong like was it UPS that screwed up or FedEx or was it something in our warehouse can we give you a coupon all kinds of different things we can do after that but this just lets us like surface it at the very highest level somewhere along the way and so those machine learning models just build queues for our customer service agents and they just work down those queues alright so there's a lot of things I didn't mention and I'm sure that you guys are very familiar with a lot of this stuff you know you kind of need a lot of text so if you have a specialized vocabulary folks talk about the clothing in a very particular way and if you need to sort of have your own specialized vocabulary you need to have hundreds of millions of words to sort of start training with Veck you need a thousand books roughly and two that's roughly 500 thousand comments on a blog post or four million tweets right and in your own sort of like language for whatever your domain is you need a high memory a high performance multi-core machine it could take those several hours to several days that but you kind of don't need to train that often so it's not a big deal to just grab an ec2 instance and go to town but also you can just use the pre train vectors most of the time and this is an issue at all and then also sequel bases databases really aren't well-suited for doing vector math and for doing additions and subtractions so there's this you have to rely on things like annoy which is this awesome spotify library for searching and high dimensional spaces or locality sensitive hashing and of course like any sort of machine learning system you're gonna have false positives so make sure that you build a system that's robust to results that a computer thinks that will be relevant but a human expert would never do that okay all right cool so this is my favorite part of the talk this is where we're gonna take word Tyvek and we're gonna sort of change what words mean and we're gonna change what context means we're gonna change what sentence means and we're gonna sort of like this is why it was so important to cover the theory at the beginning of the talk or we really going to screw around with these ideas and see where we can go with it all right so what about summarizing documents right so we've talked about words so when we were training word to Veck we're you know we might start with this example if we have extend and sort of the input context we have all these other words around it but you know we're just going back five words and four or five words or however many words a paragraph vectors really you know just expand the whole context and instead of training the word extend or some other word train and brand new word I'm going to call it duck 1347 and just expand the window around it right so it's not a totally crazy idea just kind of train it and then like leave the word afterward you never see it again because you never see the paragraph vector again and so that's essentially the idea of behind paragraph and document vectors I'm using paragraph and document as if they were the same thing okay so this is us using it in production right so this is trying to find comments that are similar to the word pregnant so the first ones are I am currently 23 weeks pregnant okay alright the word pregnant shows up literally it's maybe not so exciting but not showing too much yet alright so this is not I'm showing you a house this is not I'm showing you a presentation this means I'm showing I'm pregnant right so this is like the sort of subtlety and nuance that you can still get out of this I think that's pretty amazing get baby bump of postpartum nursing breastfeeding okay are these are these are also sort of pregnancy related words and these are good candidates for our maternity line essentially okay all right so if none of that stuff has blown your mind I'm sorry but I think this totally will all right so this is machine translation right this is going from English to in this case Spanish and so what we can do is run word avec on the English Wikipedia right and so we'll get things like you can plot you know four and five and three the words four or five or three and two and one but what's unique is like what we talked about earlier we have those linguistic regularities so we knew that these axes that status and gender are going to be like 90 degrees apart or whatever and they're gonna be 90 degrees apart in Spanish - all right the difference is that like you don't know what which way those axes were actually pointing and voila you can just get a matrix rotation that will move you from this 500 dimensional space in English to this 500 menschell space in Spanish okay so my mind's being blown right now for it seeing this right so like we have these all these linear algebra operations right so we can find similar vectors and that means like a thesaurus search we can add things so that means mixing two ideas we can subtract vectors that means removing some concept out of some other concept and you know now we can do matrix rotations and that means moving from one language to another language and I love the idea of having these linear algebra operators translated into sort of linguistic operators I think that's such a beautiful idea that now we can sort of have this robust framework and the tools for doing science on words so here's a different sort of experiment that Omar leaving yoga Goldberg did so there okay well your context I see your context usually going back to where's your going forward to words or whatever let's build in that dependency tree that we started off by avoiding so in this case the word Australian is modifying the word scientist so it doesn't have anything to with the word discovers and we're gonna try and train the word discovers and you know discovers is acting on the subject scientist and it's being in some way modified by star and telescope okay alright so that's going to be our context now and so this is these are the kinds of results when you screw around with the context instead of making this flat rotating window you kind of start to respect respect to the grammar and so what are the words most similar to hog words so on the left hand side you get bag of words model this is the traditional sort of worked effect model and so you get things like Dumbledore how those half-blood okay alright so these are all Harry Potter words alright great got it and the when I saw the words on the right I did not make any amount of sense to me right I was like Sunnydale Collinwood Cal arts green de so but if there's any parents in the room their schools absolutely and not just schools they're elite schools in the same way that Hogwarts is an elite school so these guys are calling this functionally similar as compared to like topically similar so it's really cool that Hogwarts fulfills the role of being a elite school in the same way that these guys are filling the role of being an elite school that's a good catch all right so fine everything there was still dealing with like NLP but this is word too vague applied to a social graph so no texts whatsoever and so in this case we're gonna call words of vertices in my graph so if I'm friends with Sandeep and Sandeep is friends with John and John is friends with Fred our sentence now becomes the sentence of Chris Sundeep John and then Fred right and then you know you can replace those with your Facebook IDs or whatever you want and that's it right no text no English no grammar nothing and what's really cool is that when you look up the vectors for Chris and Sandeep and John and Fred or whoever those vectors explain a lot of the properties of the graph and of your neighborhood that's around you in the same way that PageRank explains a lot of the relationships that you see in a graph Network and spectral clustering explains a lot of those relationships and it does really well on all of these performance tasks right so right now we're Dovahkiin from like this NLP algorithm to sort of being a general sort of sequence learner or a set learner alright so for you guys when you go home you guys should be thinking about or work or whatever you guys should be thinking about what sets what lists what bags a stuff do I have like can I just throw weirdo back at is if it were just a bunch of items and having nothing to do with NLP so Eric Bern Hartson at Spotify I said yeah I can totally do that so in my case my sentences are going to be playlists and my words are going to be song IDs right and the idea being that things like jay-z songs are going to be pretty similar to notorious b.i.g songs because they always appear in the same sort of rap playlist together and so he has this metric and I'm sorry you can't really read the text that well but the word de Becque is that second to the top and it performs admirably and so well the question was what's the performance and related artists like who is actually related to WHO and this is a lot like asking like a thesaurus lookups or like the synonyms that we've looked at weird effect before so notorious b.i.g has an a vector that looks very very similar to the Jay Z vector because they appear in the same playlist so that's a totally crazy use case of word effect just applying it to you know playlist instead of instead of words themselves okay so by now you've probably kind of figured out that we're gonna do it exactly the same thing at stitch fix and so now our sentences are fixes so fixes are five items in a box and then words are the items that go in that box and the idea being that those items aren't independent they are in fact gonna be picked out for you because a stylist thinks that you are a classic dresser or thinks that you are an edgy dresser you think so you are boho dresser so those items go co-occur together because they have sort of a coherent style in some way okay all right so this is another sort of TCE diagram of it we have lots of structure that's always a really good sign and we start to zoom in on parts we start to look at things that look a lot like as if our you know this is this is basically like looking into my wife's like closet for example or my Kelis wardrobe this is all sort of like consistent clothing that of one person would wear and if I look at a different part of this graph then this is sort of like oh this might be something that her friend wears or her mother wears it's not going to be totally her style but all of these items sort of form a coherent ensemble altogether cool all right so I have a few minutes left but I wanted to recap just that those paragraph vectors sort of these are all the weird experiments you can do the words back I totally encourage you guys to think about what kinds of bags and baskets and lists and sequences you have at home and work because you can throw where do I got it having nothing to do with text I know that we're in an NLP conference I totally encourage you doing it okay but you can also do all kinds of other things so paragraph vectors just screwing around the idea where the context window is dependency context changing that window dramatically and then sort of they have the social word avec where now my sentence is like walking around on this graph random walks on that graph and Spotify where they're like yeah my sentence is now a playlist and this districts we're like yeah my sentence is just five words long with these five items so anyway I think that this is a like a super creative like application of word Tyvek and just totally takes us out of like the MLP domain but anyway you guys have been great and you take any questions yeah please we'll start from left I guess yeah absolutely I mean those pre-trained vectors will probably do you pretty far you know like in the in the example that I showed earlier and let's go way back here right so like pregnant when they're using the word pregnant it's pretty much the same usage of the word pregnant in general English now showing that might be like a more subtle nuance that shows up in our database a lot more whoops sorry but in general I think that yeah you can you can get pretty far just by using the pre train vectors I would totally encourage that yeah like and then if one of the mother things that we do is in fact combine our corpus with the Wikipedia corpus so so that's yeah mmhmm yeah I mean basically the only thing that changes is you run along the corpus is the learning rate so when we were calculating those gradients there's a coefficient that goes in front of that gradient and as you go through your corpus it just gets artificially turned down lower and lower and lower linearly as you run towards the end of your corpus but you know you don't have to do that you can just kind of set it to some baseline and just keep adding text if you want so I mean once you do that you know you kind of go into no-man's land and no one's written papers on this stuff but it probably works just fine I not enough an expert to say quantitatively that no one has done this I haven't actually seen anything like this so what this strikes me is like you could try to do sort of like clustering on all of the words you know clustering is okay and you could kind of do that another way you could do is like if you know you basically want to take all your words and sort of project them onto another basis of a few other vectors right I don't know how to learn those vectors R priori you could try to do some sort of like tensor decomposition which is what I've been thinking about playing and so in this tensor you have a document and you have users and you have words in that document and you have a third dimension because it's a word vector that's in that document and then you try to decompose that into like a lower like dimensional space you could try to do something like that I'd be totally interested in seeing if that works but otherwise that could be super interesting and I think that would be a cool idea because then it mixes sort of a lot of the concepts behind LD a while using word vectors and word vectors are super useful because you get to have this the referencing back to like this giant corpus of like human language whereas LD a usually restricted to just your domain so it's effective like sort of like mixing the best of both worlds I'd be super interested if anyone does anything like that holster problem when you have to do items coming into yours so instead of needing to get lots of reviews back to talk about this is kind of easy strength I don't like it have you thought about too difficult yeah yeah so the the question is essentially cold-start and whether we can what can we do when we have very few words from the client so we kind of cheat at this and we have merchants and they give us a lot of description text about items so you can imagine if you're an Amazon you have like the manufacturer writes something about their item that's not quite the same as client text but it gives you somewhere that some of the way they're so that kind of helps us a little bit but in general isn't probably like helpful one of the other things that you can do is maybe use image vectors in some way yeah we've been thinking about how to map word vectors to the resulting image vectors I think that would be super interesting I played around a lot with coffee and the image net stuff and so yeah I mean you can imagine cooking up some sort of neural network that has the input this document vector over here and as the output the resulting image vector over here and just find some way to hook them up and that could be a cool useful way of sort of solving cold start in this in this way it'd be super interesting yeah so I can't say the numerical results of this but I can give you the conceptual ones and and it's basically encapsulated in this diagram so that left one is a TC of the collaborative matrix right so this is a TC of user item and then rated it basically bought it or not bought it and you don't see that much structure right so you basically see three clumps and you know what we'd like to see is like oh there's a clump for people who are highly correlated about genes there's a clump for people who are highly correlated about sweaters or clunky jewelry or boho Styles and it's hard to extract that much data out of something that's so blurry here and that's because we were just basically just asking zero or one right compared to the tax where you get you know a ton more data so anyway I can't can't tell you how much RA you see is improved by but they didn't improve dramatically whenever we started building it into our recommendation engines yeah