Text By the Bay 2015: Adam Gibson, NLP And Deep Learning: Working with Neural Word Embeddings
all right and welcome to deep learning with text so what i'd like to do before i get started is understand who's in the room so how many of you have seen some form of neural network before or at least have a baseline understanding awesome okay i'm speaking to the right people um how many of you guys have actually did deep learning at your jobs okay there we go all right that's that's more realistic okay so what i'm going to talk about today is some of the more broadly applicable things in nlp and deep learning so most people use deep learning for video sound images that's where you see a lot of the breakthroughs you don't see text covered very often um so what i'd like to do today is go through a baseline pipeline so in this case like sentence segmentation word usage and more covering the use cases of and like how you can mix and match the stuff from you know word to vac and some of these other algorithms with other neural networks so there's going to be a lot of coverage here at the conference today primarily focused on you know actually today and saturday on just like what is word to vac what's the theory behind it that kind of stuff um i'm actually gonna cover multiple kinds of neural ward embeddings so there's more than just word to vac so i'm going to make sure that you guys have a broader understanding of what the heck do i do with neural networks and nlp and what is what are some of the algorithms out there so there's a lot more than just word to vac and there's there's multiple ways to use these algorithms so what i'd like to make sure you understand is like how do i use the features learned from word of act and these these word embeddings and from there what do i do with them and then what are the possibilities okay so starting off uh there we go so first of all so basically like neural ward embeddings learn what's called a distribution they learn word usage so a word embedding in this case is a matrix it's a weight matrix and it's coefficients those coefficients represent word usage right so in this case like a word in is represented by its neighbors right so in this case if if i'm trying to learn let's just say i'm doing the named entity recognition task you know so i'm trying to recognize an organization and i have galvanized university right so galvanized university is is basically it's a proper noun there's university in the name that's probably an organization so in this case the the neighbor is university we're going to see that across multiple organizations the words are indicative of the actual concept so if you want to disambiguate words you want you use their you use their context it's very similar to what you see with um hold on so ah damn it ah forget it uh moving on um the word senses though um i'm trying to remember the the word neck sorry so word net you know you have the word senses right so it's a very similar idea what you can actually do is teach a neural network word senses so what the what the vectors represent if you have a if you have a vocab so very very standard in natural language processing you have a vocab still you have n wrote like you have n rows where n is the number of words in the vocab and then m is the number of features right so m is the number of features you want to use for whatever task you're trying to do so i'm going to make sure to have visualizations for each of these my goal right now is just kind of a baseline overview right so let's let's start by understanding what word usage is and that kind of stuff so in this case what can we do with these word embeddings right so basically what we can do is actually you'll you'll see that words that are very conceptually similar have a high cosine similarity so if you remember the embeddings i mentioned before if i have two words right you know if i have galvanized and university they're going to have a high cosine similarity so they're going to so they're going to be next to each other conceptually because again word vectors represent context or usage so what i want to do is using using these features i want to actually replace a lot of the more common tasks that we typically that we typically see so like you know i don't want to have to do part of speech tagging i don't want to have to figure out like suffixes right so one of the problems with natural language processing is it's brittle it's very domain specific there's a lot of labels and unsupervised learning has been a huge not trend right so there's no it hasn't been done right so you can do you can you can use tf idf right but it's a lot harder to do like sequence learning and this other kind of fun stuff because you need to think about well what are all the possible usages or contexts like suffixes prefixes right you know there's a lot to think about when you're doing text and then what about language so neural word embeddings can learn context so one great example of this is machine translation so if i want to map if i want to map two conceptually similar words i can use the same pipeline because everything's a vector and then what you can actually do is you can actually map concepts that are similar in different languages to each acro like across languages because it's all one mathematical space so these vectors again is an n by m matrix right so it's a neural network so this is this is more or less word devec so it's moral it's just an auto encoder so what you do is you train on word usage right so if what you'll what you do you have a sliding window of text you have sentences right so you train on individual tokens and you train on pairwise occurrences of the words so you basically you calculate a gradient relative to word usage and you use that to update the basically like the the baseline weight matrix which is the the the the word lookup table right so again it's a it's a multi-layer neural network you have you have the input layer you have the first layer which is the baseline representation you're going to use so that's your beginning weights and then you have the error so what you do is you back propagate on word context so you're going to see a lot of that throughout throughout this presentation throughout this presentation as well as at this conference so rather than being redundant i'm going to give more of a cursory level overview and again like make sure that you guys understand conceptually what's going on and then i encourage you to attend other talks for a lot more in depth like the math behind it and some of these other things so this is a baseline representation you're actually going to see this visualization again so you know you'll see you know auto encoders are common they're they're everywhere but basically the only thing the only other thing you should understand about the word usage and how they're trained is that there's different representations one of those is continuous back of words so what you're doing is you you actually train you actually train on words just next to each other that's like the more continuous bag of words then there's also skip gram skip gram is where you actually say you know if i have i want to say okay if i have i am teaching at galvanize i have i i am i am teaching at i i at that's a skip gram so what you're doing is you're interleaving words and you're skipping other word other words and what you can do is you can actually set what's called a window size and then basically what what then happens is you train on these embeddings and these embeddings then represent the word usage in aggregate so what you're doing is you're doing back propagation with an algorithm called hierarchical softmax for word deveck hierarchical softmax is training on likelihoods of work pairwise likelihoods the work the likelihoods of words are co-occurring with each other you can also do something called negative sampling negative sampling is a way of saying okay now i want you to i want you to learn words that don't come don't come next to each other so you want you what you want to do is it's almost like a supervised learning task in that you're doing back propagation but you're only doing it on the data so how does that how does that map back to deep learning so deep learning is known for representation learning so how many of you guys have seen the filters before like the the renders yes no okay all right that i probably should um that i probably that i will show after this so you guys can actually get an idea a better idea of what what a representation is and then we'll go from there so in this case like what are the typical what are the typical techniques so you what you can actually do is you can actually have a one hot one hot vector which is in this case these words occur it's binary you know the the appeal with you know and there's a problem there though it's sparse right when you have sparsity it's hard to learn it's hard to learn patterns and you tend to overfit right the the appeal with the neural word embeddings is that they're dense so what do you so what i mean by dense so all continuous data so there's no there's no particular feature that's gonna that's gonna that's gonna cause your algorithm to over fit so you can actually learn patterns more generalizable and word of act actually has built-in regularization as well so if you l2 regularize the vectors you can use those for more or less neutral neutral concept embeddings all right so all right so let's go through the traditional pipeline and let's understand baseline natural language processing and then let's understand where where word of that kind of splits off so you have your baseline understanding right which is in this case i have sentences that i want to learn from so i have sequences i have sets of text i have a corpus and then i want to run a tokenizer so the tokenizer is where you're where you're splitting up the text into individual words and then from there you might want to do sentence segmentation if you're trying to if you're trying to do some sort of sequence learning because like you don't want to train on a whole corpus right because you can't you can't really learn long tail trends like that so you you split on sentences maybe and then from there depending on the task you're doing you do part of speech tagging so part of speech checking for those who aren't familiar is just saying like this word's an adverb that one's a proper noun and then what you typically do is you use those features as a further representation for like more signal it's a more generalizable signal so part of speech tagging is a discrete set of tags that you can use that's pretty useful um so that's the thing the interesting thing here though word of x stops at tokenization that's it you do tokenization and you're done in this case part of speech tagging is typically used for representations where i want to if i want to learn what an organization is or i want to like do noun phrase extraction i want to pull out names of things you know you have a series of not you have a series of nouns you know and then from there maybe i'll do constituency parsing so constituency parsing in this case is just you turn a sentence into a tree you start with the words at the bottom and then you have part of speech tags above the words and then those form phrases and then you have the stuff at the top so like again like that that gets kind of crazy like that's slow right so is actually dependency parsing so dependency parsing is used in relation extraction what you're in in this case like what you capture is a graph of this a graph of the text instead how the words are interrelated so rather than relying on part of speech tags you actually have noun subject so the labels the labels in this case are actually relations like how words are related rather than based on their part of speech tags so again that gets kind of crazy like why do i want to why do i want to run why do i want to run all these extra features right so you know and in this case like let's think about how brittle that is so if i think about non-subject and then i have to think of every possible word that could occur that's a subject and i'm trying to do like relation extraction while it is generalizable to a point like you start you start to run out of you start to run out of a space you can actually use for any sort of pattern recognition we have to remember that machine learning is all about pattern recognition right so you're trying to you're trying you as a data scientist are typically specifying three noun phrases in a row or three nouns in a row are a noun phrase and then from there i can use that and maybe the fact that this suffix occurred and maybe this prefix occurred and then that represents an organization or that represents maybe positive or negative so sequence learning so there's two kinds of representations in in nlp we typically see sequence learners conditional random fields and we see bag of words so back of words is a more traditional approach you do tokenization right and then you break you break it you have a vocab just like the rest and then you have n by m where n is the number of documents and m is the number of words so that breaks down for sequential tasks because you lose context right it's a bag you count you know maybe you might do tfidf so you get slightly better signal but at the end of the day it's still sparse and it's still it's still a big matrix the advantage with let's just say neural ward embeddings over a bag of words is i can do something like let's just say i have a hundred thousand features i can actually take that and and mash it down to 300. so what one trick you can actually do is you can actually average the word embeddings and use that as an approximation for a document so let's let's so let's walk through that i have my neural word embeddings an m by m matrix n is the number the number of words in the vocab by let's just say 300. so what i do is i walk through each word in the in the document and then i say okay give me the vector for that word give me the vector for that word give me the vector for that word and then you take the mean you can actually use that as an approximation for tfidf one of my students actually built a recommender engine based on this so he used a combination of tfidf so used he you know tf idf is still very common but he also used the the word embeddings he he multiplied the word that occurred by its tfidf way if if it had one and he used that as like a weighted average to actually build a recommender so the unsupervised features actually learn like again like born patterns and they're actually great at approximating a document and in this case the the query document he used was only 300 in length so rather than rather than have to store all this you can actually just do a weighted average and you know your your space is a lot smaller you can actually do a lot with the base like the baseline vocab so rather than rather than rather than the sparsity problem you run into in this case you do dents you take means and your your feature space is a lot smaller so i just want to just want to make sure that we get like baseline nlp so again it stops at tokenization you might do you the only other thing you might do is limitization or stemming so you still have the same problems with the vocab though so if you have your n by matrix right it doesn't matter if it's word of act or glove as well so glove stands for global vectors that came out of stanford for those of you who might not have seen that before so you still have you still have your word embedding right so you have n by m you want to still compress your vocab right so you want to you still want your corpus to generalize right all right so tfidf let's see unigrams by grams 20 grams okay so in this case one one other thing i want to mention with the word embeddings you can also do phrase embeddings so if we now not only understand a vocab what you can actually do is you can actually you can you can actually have each individual word in your vocab be a trigram or a bigram or trigram so in this case like those words can actually be phrase embeddings as well so you can actually teach it short you can actually teach the word embedding short form text kernel hashing you guys have seen this before we're divac um okay so what i'm going to do is i'm actually going to skip to use cases now so let's let's let's let's let's just get a baseline visualization first so these are word embeddings so let's let's let's remind ourselves what this vector space is this vector space is you have your matrix and you want to run you want to find like nearest neighbors so in this case words that are conceptually near each other have a high cosine similarity so words that are near each other in this embedding space this embedding space is called tsne t stochastic neighbor embedding so what it is is it's basically t-sne is an algorithm for finding the likelihoods of words co-occurring with each other or features so it uses what you do is you actually you use something called an sp tree or vp tree a vantage point tree to actually learn likelihoods of words being next to each other it's a spatial embedding and you can use that to build you can actually use that to build this visualization so what it does is it finds it finds it finds words that should be near each other so in this case the representations there's a few there's a few there's a few ways you can learn so one of the look for sequential learning sequential learning involves what's called a moving window so you can there's a paper there's a paper you should look up called nlp almost from scratch nlp almost from scratch uses a neural network to to replace every single manual natural image processing task and it uses these embeddings so it uses an embedding and it does back propagation on the labels and it uses it it just uses those embeddings as features so how do you do that so if you want to do named entity recognition let's just say you break up a sentence and you label in this case sets of words and or windows you you do a moving window over the text and you say okay now let's just say i want to record i have a window size of three and i say and i say nitro nitro pdf documents that's an organ maybe that's an organization or that's a name right so what you can do is you can say give me the words from the vocab so remember i've already trained my embeddings right so the embeddings just train on word usage so what i'll do now is i'll look up each word that occurred and if if it's unknown it's unknown so that's another typical trick you do either way you have a window and you say okay now let me concat these vectors together right so in this case you can take these embeddings and embed them row wise right so let's let's think about that so give me the word for nitro give me the word for pdf give me the word for document in in the norward embedding right so you did you trained on word usage already and now i want to do a moving window and that that moving window will that window will have a label right so that label is an organization so what i did was i just generated a feature vector for a sequence so you can actually use the neural word embeddings for sequence for sequence learning so what else can so you you typically train this with what's called a deep belief network or another like any other kind of neural network it's typically for feed forward architectures you do this moving window approach so you can actually teach a static feed forward neural network sequences so in this case like why does that work so the feature space represents usage so you inject context by concatenating the vectors together so that will give you and that will give you the features for a supervised task so rather than doing part of speech tagging for name denting recognition what you can actually do instead is just use the embeddings and those embeddings represent a concept space that you can use for supervised learning so in this case how else can so how else can you do this you can also you you can also you can also combine these words with a recursive neural net so a recursive neural net in this case you build a constituency parser and then you can actually use that constituency parser to build so remember a constituency parser generates a binary tree and instead what you can do is so instead what you can so i'm if by the way i'm going to give references for all this stuff after so i'm more going to like answer ques so if there's any of this if there's anything that you guys want to see i have visualizations and all sorts of slides that i'm going to be sending out uh so don't don't feel like what i'm covering here is like completely like off the cuff like i'm giving you guys i'm giving you guys examples because like i'm not gonna be i'm not gonna be able to go through if i'm not gonna go through all that um in depth like i'm just trying to give you guys some stuff to research i want you guys to actually take something away from this talk um continuing um so if you have so if you have a constituency parse right so you build a binary tree right and then you say okay now give me the word vectors so again the word vectors come from that matrix that word embedding and then what i want to do is i want to i want to use the constituency parse to combine the words so you have the word embeddings right and then you mix and match the word embeddings up the up the parse tree so let me show let me let me actually show you what that looks like real quick so this this is what i meant so you have so rather than doing this moving window you can actually build a recursive neural network instead so if you in this case if you remember if you remember the consistency parts i saw you saw earlier right you'll notice at the bottom there are those vectors so those vectors are essentially the words that you're learning so you have the words and then you go up a tree and then each of those nodes has a likelihood of being a label you can also do unsupervised learning with this this is called a recursive autoencoder so the recursive autoencoder uses these word embeddings which represent word usage you then mix and match them and you so what you do is you you learn approximations of the whole sentence based on the structure of the constituency bars so you train a neural network on the constituency parts itself right so you can use these embeddings in either a moving window scenario where you where you where you train a feedforward neural network on a static feature matrix so let's just say if you have let's just say if you have a window size of three right and then three words and then it's a 300 it's a 300 length embedding you would have 900 features that you use for a window so in this case what you can do instead is you can say okay now i want to use 300 features for each of these and then i will combine these into another 300 length vector right so in this case what you do is you train and you actually train a neural network and learn contexts so you and then what you can actually do is you can actually go down the tree and get approximations for every context so this is this is a way to do sequence learning you can also use this with long term long short term memory neural networks as well right so long short term memory neural networks do not use recurrent by the way do not use back propagation through time for those of you who actually like are doing deep learning use long short term memory anyways so long short term memory neural networks they learn a sequence so this is this is a this is what's called a recurrent net a recurrent net uses something called back propagation through time so back propagation through time is an extra feed is basically an extra feedback loop where in this case you learn context a long short term memory neural network solves what one problem with with these neural networks though of back propagation through time also has a problem called the vanishing gradient the vanishing gradient says okay i'm gonna i'm gonna forget what i learned a thousand years ago or a hundred you know 100 words ago it's a way to generalize sequences so you can actually train these neural word embeddings on a long short-term memory neural network or a recurrent net so what does that look like so in this case what you can actually do is you train an individual embedding and then you basically build a language model with the embedding so the language model is a vector right and then what you can actually do is you can say all right now i want to i want to i want to train it on the next word and the next word and the next word you actually train it on a sequences so recurs a recurrent net is actually a generative model so you can actually use these neural word embeddings as a generative model so in this case like what can you do with this sentiment analysis so one way to do it is a recursive neural tensor network right so recursive neural tensor network essentially solves the same problem that long short term memory does i'm going to use the recursive representation just to kind of show you like what a sequence looks like because it's roughly the same idea so you train on a sequence and you learn likelihoods of particular words at particular contexts in in rel in relation with each other so a recursive neural tensor network also uses a binary tree for the representation right so you can use this in vision and all sorts of fun things you can think of a recurrent net and a recursive net as basically the new the new conditional random field right so you guys have used max ant models if you guys have probably used like head markov models um so in this case like the recur the recursivenet is basically an automatic feature engineering tool um with with like with with a built-in weight matrix so it's just hidden markup models with a weight matrix basically so in this case you weren't so anyways you learn varying length sequences the sequences are based on these word embeddings and again word embeddings represent usage to map this back to deep learning so if you'll see the if you if you look at the bottom there the the the handwriting task there that's and that's the endless data set those are learned features from a neural network so those are actually the weights drawing those though that handwriting so that's that's the automatic feature engineering so this was drawn from a restricted boltzmann machine a restricted balsa machine does is an auto encoder that does automatic feature engineering word to vac is also an auto encoder one you use with text another one you can use with images or what have you so you can actually also like so what so what's the implication here so while an rbm is related you can you typically do pre-training in a deep belief network you don't need to do that if you're going to do everything with word defect though word of act does pre-training for you so if you do word of act you actually can just use back propagation instead right so let's let's keep this in the context of nlp though so remember we have these word embeddings right and then you learn varying link sequences so those words already represent usage though so the great thing here is that i've already ran some sort of i've already ran some sort of pre-processing so in this case now i can just use these representations without doing any sort of i can actually do without any sort of pre-processing so all you have to do is train a vocab and then you're done and then you just you just go up the tree so what does this actually look like though for for if i want to do supervised learning so in this case remember that each each context has a label so in this case like for for recursivenet you have a softmax at each node the softmax represents the likelihood of a particular label so you in this case what you can actually do is use each of those contexts to identify like this parts negative this part's positive so this allows you to do fine-grained classification it with just one sentence one thing though is that these were the recursive neural nets can be slower to train um i highly recommend using long short-term memory instead i'm mainly using the recursivenet as a way of kind of demonstrating what you do with deep learning so deep learning typically with uses recursive nets though for a lot of nlp most recently is when people are starting to use lstms now so again long short term memory so the embeddings are the automatic the automatic feature engineering is the key here so neural networks vectors and vectors that represent usage that's so what i'll do now what the the talk ends at 11 10 right i think so we only so i only have a few minutes so what i'm actually going to do now is now that i've ran the gamut of holy crap that that's there's a lot going on here what i want to do now is take take the last few minutes for questions and kind of expand on things that might be interesting to people again you guys will get word to back all throughout this conference i wanted to give you guys i wanted to give you guys a take away here so you guys you guys are going to see word devec again and again if you want to go in depth i'm more like how do i build a deep learning app today so i'll take questions now is sure i mean okay so have you looked into the paragraph embedding as much the paragraph embeddings by quackley so it's called so if you use gensim it's called doctovac so the paragraph embeddings represent like represent document embeddings for so what it does drvec in this case trains the embeddings on labels so what you do is you teach you teach the word vectors labels instead if you want to do document classification i mean that being said another thing you can do you can use the moving window approach i described so like you you do moving window over the document and then you do an ensemble and then what you can do is actually so this generalizes better actually so you can actually train you can actually do a moving window so let's just say you have a word embedding of three 300 in length and then three like a moving window of three you have 900 length feature vectors and then you can build a classifier that classifies individual contacts and then you can you can actually vote instead so so you can also so you can also do this with you can also do this with recursiveness as well um there's actually better there so there's actually starting to be approaches used using convolutional neural networks as well so convolutional neural networks are typically used in vision for in this case like position and variant features what you can actually do is you can actually teach it document classification as well by convolving all the you can actually convolve all the words that occur and then and then back propagate on the label as well so what does that look like so this is a comp nat so what you actually do in this case instead of mapping an image you map you map you map you map the word embeddings instead and then what you do is you actually run a convolution on the out like all the on all the on all the words that occur and then you can use and then what and then what you got there on the on the final fully connected layer is basically logistic regression so that's basically logistic regression on the output layer so you can train you can actually train a classifier with respect to the all the words that occur and then like so you can get a good feature representation there as well um anything you guys want clarified at all like because i know like i went i went quick uh go ahead so i would i would use i would use word now i would actually use wordnet senses uh in combination with the tokens so what you can actually do is have each toke each individual token each individual sense be a vocab word and then you can just label your corpus for your particular domain oh so oh okay so fast convolutions is that your question okay so fast convolution's like okay so um disclosure i'm i'm the author of deploying 4j um i do a lot of spark and scala so i'm more the jvm but i actually wrote my own scientific runtime uh into like that uses cuda so i actually use spark with cuda uh to scale so i use i actually use nvidia's uh fast fourier transforms and convolutions uh for the gpu so in this case like in this case like what i do though is i allow multiple host threads to use multiple gpus so that's what i do with most of my customers is they use fast forwarder transforms with multiple gpus and then they spread that out across they spread the computation out across but either way like use use gpus if you can so okay so let me let me just explain one thing there so convolutions are a very computationally intensive operation typically done on three and four dimensional data so if you if you remember images are like images are r g and b you have three you have you have multiple channels right so you'll notice here that the con the confident actually in this case will will actually if you have if you if you have n-dimensional data look you have the number of the number of examples by num by number of let's just say number of channels by number of rows by number of columns that's four-dimensional data so a confident actually has a multi-dimensional weight matrix so that's represented by those feature maps there so it can actually learn all those things it can actually learn from all those features at once and it learns position and variant features this is this is the most computationally intensive of all the neural networks um so they don't so confidence have a problem comnets can have problems scaling as well um so you know like distributed you know it's like if you're gonna do distributed deep learning uh like at google or something right you know and that's that's what i specialize in is a lot of distributed stuff um i typically use feed sorry feedforward neural networks for that because like you know it's really hard to it's really hard to scale out like that convolution operation which is why most people use gpus um other questions all right um one thing i'll do then is show like show a really quick cool demo so if i can find it i think it's all right so these were these were the word embeddings i mentioned before so this is this is actually this is actually tsa in action so in this case i'm using d3 to visualize the words here so you can like zoom in you can see that some words are conceptually closer so like day and ago on the left over there right so this is this is a sparsely trained vector so it might not make sense but like this is actually a way to debug like how well your word embedding is trained so one thing i encourage is if you're gonna do deep learning debug your neural nets visually so use something open like i'm biased but i believe i believe you should use something open source hence why i made my own framework right um you know like use something open source and like actually understand what you're doing like you know like you can use a black box but it's only going to get you so far right debug your stuff visually so you can do you can do things visually with words the word embeddings you can also do things you can also do things like histogram the weights so typically like when you're training you have you at each iteration you render the histograms for the gradients as well as the parameters so and what you can do is if you see those are normally distributed every iteration you know that's going to converge you know it's going to converge faster and easier so debug your neural nets visually so what's really cool about deep learning is since basically everything's generative for the most part and everything's unsupervised you can you it's really hard to evaluate the features but when you do like you get to use really cool stuff like this this anecdote um so remember like the intuition here is you're learning word usage so you want conceptually similar words or phrases to be similar to to be close to each other oh okay thanks you