AI at Stitch Fix

ai.bythebay.io: Chris Moody, AI at Stitch Fix

Recording: ai.bythebay.io: Chris Moody, AI at Stitch Fix

[Music] great awesome well thank you thank you for the awesome introduction thanks for having me guys actually why don't we go ahead and give a round of applause to electing the organizers for putting on one of the best conferences in the day thank you guys so today I want to talk a little bit about how to get some level of interpretability out of your models and how to understand what your model doesn't know with variational method I think interpretability is one of these things that divide scientists in academia from scientists and industry think when you deep inside of machine learning and deep learning you become focused on that benchmark and the first thing to go is interpretability and I think that's the wrong approach and I think it's especially the wrong approach in industry I think academics tend to focus on the science of algorithms what are my bottlenecks what are my abstractions has convergence working how's regularization working but an industry the value is in the data and the algorithm is a tool just extract value of that data instead of the science and algorithms I'm actually very very very interested in using algorithms to do science so this talk is about how to do that and understand what your model is doing and also reason about what your model doesn't know so my name is Chris moody it's my twitter handle I went to Caltech I did physics did a lot of supercomputing back in the day scikit-learn contributer a teener contributor and I'm starting to move over more into PI towards land aside from deep learning I also really love things like Gaussian processes and tensor decomposition so if that's up your alley I'd love to like nerd out about that stuff as well so at a conference about two weeks ago Brenden phrase the CEO of deep turn omics as a start-up trying to solve health problems with deep learning so you know we have the full genetic code you know what's the problem can't we just look at it of course you're saying this rhetorically and of course this audience is going to stand what the problems are with that stuff right it's going to be big data you're going to hold these weird patterns inside of the stuff it's harder to causal inference on just that data set but one thing that stood out coming to me from what he said there and it was that you need to be able to trust your model right you need to be able to break that model into components that humans can actually understand and that's the desire that's president you know in every human Enterprise from health care all the way down to like fashion which is what we're doing at stitch banks and so I'm going to talk about three different things that will help you build help you understand what your models are doing one of them is teasing me the second is a KS VD and the third is gonna be L di de Becque if we actually have time to talk about that and to start with while this talk is super general I'm going to start with a very specific example just to get us kick-started and this is going to be a very very simple co-occurrence model and i really and it's going to be off in an LP land and I like it because NLP can be the sort of messy affair if you have to teach a computer about the irregularities and ambiguities of the English language and as a result when you actually find a model that works it can be extremely gratifying to the scientist and it can be really useful to the business and application right and so this is a client comment about a particular item and I've artificially injected this token item 92 in front of it and that token looks a little as that that I don't look a little bit like this and so the clients you know written I think this fabric is wonderful it's a rayon spandex I really like the lace and embroidery accent and all the deep learning is you know is running this sort of text sort of code currents models behind the scenes anyway so this is just going to be a really simple bag of words base line on top of it and it's going to give you but the important part is it's going to give you vectors that aren't immediately interpreted and we're going to try interpret it after the fact thing and so this is a model is nice and it's going to embed both words and items in the same space and we're just going to start off by counting right so we're going to count how many times a died and my students show up around the word expand I think show up around the words appendix how many times it's fabric show around spandex and wonderful and so on so forth all across some number of ways before that word and some number of words afterwards some window around right and so we're just cutting up pairs of words around the word spandex and we're putting them into my little matrix acts great we're gonna do that for the word like and we do it for the word lace and embroidery and accents and so and so forth so X is just as 2d matrix where each element is just the count of how many times I saw that pair of words near each other fine alright so so far super easy if a minor modification from things like glove and words of act like algorithms I'm gonna take the that counts as one who logs because words are not distributed and I'm going to break it up into three little pieces one of them is going to be how frequent that first word is so if I have a tuple like like and embroidery and I count how many times like showed up around embroidery maybe that pair of words is really hot has a really high count just because the word light is a really really freak of word and there's any sort of special relationship between the word like embroidery it's just that life is a common way so that first time is accounting for that second term is how frequent is the word embroidery show up you know and then the third term that's the most interesting one that's going to say is there anything special between the word like anything special between the word embroidery if so the dot product that they're vectors is going to be a little bit higher if not the top products may be a little bit lower so things like like and embroidery not that related but words like spandex and rayon both materials show up in the same sort of materials same sort of items show up in the same sort of context well they're going to be similar and they're they're dot product and they're vectors are going to be similar and they're going to push up those counts up a little bit higher so anyway so there's a super simple model to implement but I want to take a moment just to have a little call-out here stitch fix is like a very is very heavy and to NLP and very having two recommendations engines I want to draw out the similarity just for a moment here that this is exactly the same sort of model that one like the Netflix price right so some matrix factorization approach except instead of saying how popular is the word like and how popular is the word embroidery are saying how frequently does this user like buy stuff how frequently is this item bought so sort of just the marginal probabilities of those things being bought Plus interaction between those two things how similar are the words rayon it's Brand X how similar is this user to this item same sort of model very different application alright anyway chickens pop this into like whatever your favorite great needs of gradient descent framework is right so to be PI torch could be chain of hell even vocal lab that will actually do this for you with a few command-line arguments right in fact this actually also is really really similar to what fast text is doing too if you supervisor if you supervised this model and also add a hash trick so this is a really nice way of thinking about it a lot of different models anyway so if you have a lot of time this can actually take a lot of time to converge and back in the day when I was first dating would actually take days to converge and I would use it a little bit faster but if you actually just take the SVD of this you can get the same answer in ten minutes it's the same answer every time it's no fuss you can do it on one CPU with limited memory and it's fantastic right so you can take the SVD of just how frequently those two words showed up together that's that was my X matrix divided by how frequent those individual things are I take that CD of that I get deep learning quality results right so this is the CEO of Goldberg paper and love you more paper that say this is a really simple way to get word vectors out no neural networks I think that I can be done in ten minutes on the simple CPU with vanillin like linear algebra libraries using a vanilla it's not like sci-fi and numpy will totally be able to do this for you just by Kenneth oh okay so random aside whatever I filled out my model I get my tokens I get my vectors I may have held it as VD I may have done it with a gradient design whatever my vector looks a little bit like this it's a bunch of numbers and there's no way I can actually start to understand these numbers until I start like building a model around that to you know maybe I'm doing an image thing in which case maybe you're thinking about vectors in images maybe they have like maybe you've run a document modeling you're thinking about vectors for documents but like let's just go back to like our words model and let's start by looking at what the nearest neighbors are for each one of those things so for the word spandex it might be that the nearest tokens stretchy fabric great awesome Rand X totally a kind of stretchy fabric if no one knew what the word spandex was I would totally describe the stretchy fabric in some way great but this is still like a very local view of what my model is doing doesn't really give you a sense of scale of what you're what I was doing and so this is something that so tease me is a way of actually sort of getting a global picture of what your model can actually do so this is a picture of all of our Styles as our client described them and what we're doing at Citrix is we're actually selling artwork so I'm really not interested in how it's categorized inside of our internal database right it's art like there's no way that some random columns are really going to describe how our clients internalize it so I'm really really interested in that internal description of how those clients describe our Styles I mean so do you think this is students to you straight distributed stochastic neighbor embedding which is a total mouthful but what it's actually doing is it's explicitly modeling pairwise relationships in that high dimensional space so this is very different than things like auto-encoders PCA dimensional action techniques because those things are trying to rebuild and reconstruct the original signal TCA is not really interested in doing that these things trying to reconstruct the pairwise relationships between points and does that by doing kernel matching it has this high dimensional input kernel so it's how far apart are these two pairs of points in the high dimensional space well that should be really really that should be very analogous to how close they are in this two dimensional space and exactly on that plane with funny definitions for what distance means and for funny definitions for how they actually compare distances but again it's all about the pairwise relationships not actual input signal anyway let's look at a patch of space here's one that's like devoted to just scars here's another that's like just jewelry and you can see this thin dangly jewelry on the lower left and it's going to pendant and statement pieces up towards the top it's not a peaceful that summer dresses may be color blocks towards the upper right now there are problems with tease me about right and so this is this is Neil Lawrence and he's one of the is probably one of the biggest minds behind Gaussian processes as he put it you know nonlinear methods are like drinking whiskey straight not for the uninitiated so here's what he's referring to I wouldn't blame you if you thought that this distance right here was actually twice the distance as this right here or you'd be wrong that distance is completely meaningless right that global structure is completely lost there's also a bunch of other bonds to teach me the random noise in here might not be random there's all kinds of strings and random shapes that like form there's no categorization so a lot of this stuff it's kind of like reading tea leaves but there are some tantalizing details here I can look at this patch of like skinnies and I can move to up Tiller's toes the top left and start to get more exotic sort of skinny jeans so does have seem to have some local structure in it I can move it and look at these jeans either go towards the top and get more colorful things as I move to the bottom and get lighter jeans and so for me this is the holy grail right learning these directions they tell me how to characterize my business this is how our clients see my business and this is what it means but at the end of the day what I really want and how to actually read these TVs are they actually real now we get hint that these directions do exist outside teach me right so if we go back to where defectors this is obviously the very famous relationship between men and women being the same thing as the difference between king and queen same consistent relationship across aunt an uncle and daughter and son and this is when we talked about linear regularity this is what we're referring to that direction always means it's like the more feminine axis and of course we have hundreds of other directions including hundreds of other ideas that might be it might be a status direction write to me those are the directions I want to find in my model now we're probably not capturing status and gender since I'm on a clothing dataset but what are the directions that we're capturing what are the interesting directions in my space teasing is scrambling most of the directions only keeping these local deviations but what are the key directions in Mike's place and now does this generate if you think this is just forward factors you'd be wrong this is actually generic across a lot of different models if you're looking at faces there's a there's a really awesome Twitter bot called the smile vector thing which would automatically take an input image and add a smiler vector to it and slowly make it smile more so these kinds of directions exist and it happens in models where the latent space is distributed under the dot product R and that's that's what's forcing these linear regularities in the latent space anyway this is where the KS BB comes in really really handy so this is another trick from the old world that brings interpretability a really useful way before the chaos BB my CEO would come up to me and say hey what are Katrina would ask you know what are they our clients saying about this particular style and I would just hand her back this dense vector and she was like what am i paying before there's no way I can actually do anything with that vector but after the chaos video it will actually return you back saying you know it's plus point five three and item 23 direction it's plus point 16 and atom 95 now atoms and the K SVD are basically the same idea of eigenvectors and SBB components and PCA factors and nmf topics in Lda k svvv calls and atoms but whatever they are they're just directions in this space so you can think of them like my stem Ihnen axis or my status access exit axis in word effective in word today now with a little bit of elderberries I can figure out that atom 23 is really like the tank top Adam and Adam 95 is really like the exposed shoulder Adam you figure that out by looking at like well what else was really big into atom 23 and what else was really big into Adam 95 and I figure that's like tanks often expose shoulders Adam one is all about dresses Adam 14 might need sort of this urban and bohemian Adam and on 22 is about like statement pieces Adam 75 about like ring and drop earrings so this is to me this is great because I start discovering a lot of directions that live in that space and I would highly recommend checking out this blog post which is on the bottom left-hand side here of decoding the thought vector they push this technique into all kinds of sentences techniques all kinds of like other image techniques super interesting stuff I don't have a huge amount of time to talk about lv8 avec but le8 avec is another way to actually build sparsity into your models and learning about not just word word co-occurrences but topic 2 word and document 2 word co-occurrences and really nice things you get visualizations like this where you can learn what the topic space actually means critically though instead of being plus 2.5 3 atom 24 year now 90% in the tank top topic and instead of being plus 0.9 whatever you're now 10% in other topics or sparse to be constraint is that everything has to sum to 100% and so just being positive or negative anyway so this is an interesting way to add interpretability and Sparsit appear model all right now don't really want to talk about variational methods these are really really exciting techniques that you can add to your machine learning models so most machine learning most people inning is about optimizing a point estimate on your model parameters if you've never seen a client you should probably really really uncertain about that client is the way of doing this and off you can have alternative modes of like regularizing your model you can think of this as an alternative to things like drop out but it has very very very practical implications for things inside of the business right so if you've only if you trained a model that's only seen images of say denim jeans I've never seen any jackets well when you see a denim jacket you should be certain of the material but may be uncertain about like the category and Department and variational methods are a way of applying that sort of idea within the rest of your model now what's interesting is that these things are actually extremely simple to like very little effort to write code for these methods will totally sit in wheat and just in case you don't believe me here's like the whole tweet that will actually do variational inference for you in just a few lines of code I would you know this is using Auto bread but it's not very different in say something like PI torch and I want to take a moment to take the next few minutes to unpack what's going on here so let's go back to our favorite model over here very simple variational word vectors and we're going to make it we're going to make a variation all right and we're going to do it in two steps one by replacing point estimates with samples from a distribution okay and then we're going to regularize that point instead of regularizing that point we're actually going to regularize that distribution and say that distribution should look really really similar to my prior and we're going to use a little bit of pipe torch code to do this great so if I wanted to rip out that one vector corresponding to the word style to the word like or word embroidery this is how we do it inside apply torch I declare one matrix of word vectors I pull out a particular row out of that matrix of a particular index and that row is going to be my word vector now super simple to do in PI toys and I here's what it looks like with the variational mode right so now I'm going to have a mean vector I'm going to have a log variance vector two so instead of one matrix I now have two matrices instead of one vector have two vectors and what these two things are doing is that they're parameterizing one distribution for me right so that vector unit vector LV well that's going to set the mean of my distribution is that vectors a little bit larger it's going to shift that distribution to the right at least in this figure if that law is variance is larger it's going to make the distribution fatter that logs bare hands is smaller it's going to make that distribution skinnier and I'm going to trace samples from that distribution right and it's going to be stochastic so every time I run this I'm going to get slightly different answers no longer we're going to be deterministic and this is going to allow us to get a different sample every single time and every time I backprop through this it's going to affect both my mean and my variance in contrast to networks before which are just pushing one point around I am now pushing a whole distribution around now the cool thing is is up until 2013 doing exactly this kind of math and algebra required all kinds of formidable integral tricks all kinds of specialized map look at like the LDA paper way back in the day which is solved using variational methods it's incredibly impressively it's an impressive feat of like a variational calculus you don't need to do that anymore this is a beautiful representation trick and it unifies a lot of approaches and it's at the core of Beijing deep learning methods and what's more it's super elegant it didn't have to be elegant but it was and it can be implemented in Python code inside fitting inside just a few lines right here okay so fine I replaced my point with samples from the distribution now I need to regularize that distribution and say that distribution needs to be really really similar to my prior for example without the variational method you would probably just penalize points just by for example penalizing the l2 norm of something I take that vector I square it I sum all the component this is basically saying that this needs to be as close to zero and it needs to compromise between zero and how much data I get but now we're playing not with points and not with parameters Belov distributions and the way that I compare two distributions among many different ways but the one that is usually a person variational methods is by compare them by using the KL divergence and what this is going to do is it's going to say hey look that mu and that effect and that log variance buttons be pushed towards my prior so if I'm off to the left of my prior it's going to push a little bit closer and if my days are and my mewling veteran if my mu and log variance are very far away to the right of that fire it's going to push it more towards the left okay let's bring this back into reality and let's try to understand what's actually going on here if I train to my point estimate word to vector model I might get this kind of relationship back right so I get cane cream man woman I also have this really weird extra vector this extra matrix hanging around that's describing the variances around each one of these points what does that actually mean well the variance I can start to visualize and it might look like this right instead of points I now have the lips around this stuff and this is now describing a probability distribution for that for the put the vector for the word man and it's interesting right so right here that ellipse is for the board man is much much larger than it is for a woman for a king or queen and why is that it's because the word man here has is used in a bunch of different contexts it can refer to being a human male its definition number one it can be referred to like human beings and German right so like mankind or it can be used like sticking it to the man so like being a general like authoritative figures so the word vector for man is actually used in a lot of different context and the variational vector around that is trying to capture that so it's not just the average it's not just a point estimate of it it's actually being used a lot of different context so in this case I think this is a very powerful idea because variance is no longer just a regularization technique it's no longer just measuring noise it's actually telling me what my models claims was consistent with my data and this variance is not just uncertainty it's actually giving me a new kind of representation with novel properties so this is you can also see that some like the word to Gauss paper you can take the vector for the per Bock and you can see that it's actually within the vector for composer so it entails composer boxes the composer totally makes sense it's within the vector distribution for classical and it's within the vector for math right so now we're not just capturing similarities in the more we're actually capturing linear regularities and all kinds of entailment and the property that words can actually be subsumed entirely by other tokens and so this is something that the original word effect didn't have entirely and this is an example of learning hierarchical representations by using by understanding that these word vectors are actually can exist in a lot of places all at once okay so a lot about word vectors here let's try it on something totally different right so this is an experiment that I was working on the last few days you all are the first to see list aside from like a few Twitter people so take everything I say with a huge insult on the left hand side I'm going to turn on the movie in just a second is a traditional point I submitted teased me on raw and this pixel Santino deep learning just looking at the raw pixels and on the right hand side it's exactly the same data set but I'm going to use a variational format keeps me so I'm going to start in just a second but I want you guys to notice from the get-go that the ellipses on the right hand side are huge right so my model is very uncertain about where these points should be and as we start to as we start to evolve the whole model like you'll see those ellipses get a little bit smaller and smaller and so that's the models way of saying of expressing its confidence and of estimating its own confidence and expressing it back to you you can see that those variational the in the variational model those points are migrating when they're migrating that those ellipses are a little bit larger and that's its way of saying that I'm really not sure about where this point should be that's really useful because things like tease me have all kinds of like you know reading the tea leaves behind it so having a so knowing where those points actually are is super useful so you can always end up like in this kind of case right you can end up like looking at that top that if you're looking at the left-hand side here if you look at that appointment all the way up at the top left you know and not me or anything I certainly look at that I say you know that a real point I reran tease me with that thing not be next to anything else I don't know anyway do you run this thing and both tease me and variational teasing you will converge and at the end of the day only the variational model is going to have a discipline to measure for for its confidence right so if we can week we can see if I can actually zoom in on different parts here all right so over here this is a huge a list you know maybe I'm not actually very certain about that point for this point they're going to be farther up in different parts of this space and then gonna be extremely certain about I'm not going to turn about any good points over here so this is really hard to do when I can't actually have the same screen on my own laptop but having that notion of what to be confident about what not to be confident about is how I build trust in my actual model and it feel makes it feel like I'm reading the tea leaves a little bit laughs all right clearly so you guys been an awesome audience if there's a few things that I want you to take away its tried using SVD form instead of the neural network form of word de Becque try using PC although I think everyone already pretty familiar with tease me try using the K SVD to interpret your model I think this is very rarely that have I've ever seen a paper actually use the K SVD and try variational methods when it's important to know what your model doesn't know okay so that's my Twitter handle Christy moody I will actually be talking about a bunch of different stuff coming up we're going to try and play nuclear norms which is a way of building sparse representations for matrices and trying to thinking about them in the context of things like LSD ms which are sort of notoriously uninterpretable anyway sorry I've gone on far along enough you guys have been awesome audience thank you very much what will be up time for questions you know oh okay I would just be modified for next nearest neighbor associations rather than word pairs and how would that look in this sort of piece key analysis and in fact like when you look at so Missy panics I understand the question how would this be modified if you looked at just nearest neighbors Oh next near service yeah so if you if you look at a lot of the original teasing papers it's a little bit difficult to actually measure how good a job teasing is doing because it's an unsupervised problem but one of the metrics that they use is how faithfully is the nearest neighbor in this two-dimensional space reproduce the nearest neighbor in the high dimensional space and they ask that not just for the nearest neighbor but like the next nearest neighbor and so on so forth and you can see that teasing does a very good job of reproducing the nearest neighbor graph very very well oh thank you guys [Applause]