Text By the Bay 2015: Richard Socher, Deep Learning for Natural Language Processing
Recording: Text By the Bay 2015: Richard Socher, Deep Learning for Natural Language Processing
Hello everybody. Thanks for coming. Happy to be here. I want to talk to you today about deep learning for natural language processing. And in order to really appreciate how this is novel and why it's actually a lot better than a lot of traditional natural language processing, I'll start a little bit with a handful of slides on the past and the present of technology and and what's out there. and then go into and give a bit of an overview of of what's possible now with deep learning in the natural language sort of understanding and processing space. So I think the main way uh past approaches uh have been sort of lacking is that they made various very strong simplifying assumptions. So what do I mean by this? Uh, one very common way to classify text is using so-called bag of words representations where we basically have a count vector that is roughly the size of our vocabulary and we ignore word order
So if I have for instance a sentence here such as white blood cells destroying an infection, then I might just have you know a very long 01 vector that is the size of our vocabulary and I have a one for all the words that actually appear in that sentence or document. And that can work reasonably well for some simple information retrieval task where I just want to find all the documents that mention some words or topic modeling or some simple types of document level classification task where I just want to know is this about sports or politics. Right there you can get away with just saying well you know I see Obama and president and and the word politics so it's probably in the politics class and I see football and baseball and it's probably sports. So however, once you want to get at some harder problems or maybe also classify shorter text for finer gradations inside a class, you run into trouble with this kind of representation. So here we have uh another sentence namely an infection destroying white blood cells. And we want to for instance classify this as whether being it's positive or negative if it's if you see that kind of phrase in in your medical report. And now white blood cells destroying an infection and an infection destroying white blood cells have basically the exact same bag of words representation. But in one case if it's written in your report you might live and the other one you might die
So it's a big difference yet the bag of words representation can't capture that. And similar things happen in other kinds of sentiment classification where it actually matters to understand the structure of the sentence. So if you have a movie review sentence for instance and it says you know this film doesn't care about cleverness wit or any other kind of intelligent humor a lot of downstream models that you would build on top of this kind of representation that is just bag of words would basically count the positive words subtract the negative words in some way and say overall this is you know positive or negative and so here you have you know humor intelligent kind wit cleverness care lots of positive words not really a negative word yet the whole thing is really negative because of that one little dozen uh in front. And so ideally we'll have models that will actually learn this kind of stuff just from seeing a lot of examples. And at the end of this we'll actually can look at a live demo and see a model that will really solve that. Another way you could try to simplify this is instead of looking at the entire document you only look at the specific fixed window size around a specific word that you like to classify. So for instance, if you want to classify the part of speech tag of a single word, for instance, here this center word, then you might want to, you know, it might be enough. It might be okay to just say, oh, there's a determiner in front of this is the, right, or a
And there are most cases what follows after the is either an adjective or a noun, right? So you can get away with that. There's also other simple tasks like named entity recognition where you just want to find this a location or a person or an organization where you may also be able to get away with just looking at a fixed context. However, again, if you want to push beyond sort of these simple coccurrence patterns and really get you know beyond sort of 80% accuracy and really push towards 90 and above, then this kind of representation will also get you into trouble. So here we have another example from sentiment classification and the sentence is if you enjoy being rewarded and you might say right now this is a pretty positive phrase you know a fixed window size phrase that is positive by a script that assumes often talking about scripts is uh overall also associated with positivity in this corpus but then the next window is you aren't very bright. Now you want that window to really counteract the other previous two windows of the same size and really understand that overall this is a negative sentence and again here if you have you know you see the movie title blood work probably realize it's not a high quality movie. So basically these kinds of things uh you know and the kinds of tasks where we can get away with this have been reasonably solved. And so I think right now we have a lot of comatization where there are certain tools for simple task that just work incredibly well right out of the box. And so to give you an idea of of such a tool, here's uh one that is from my company Metammind uh where we basically allow you to classify tweets and and any other kind of text
And so here I just uh before the talk I searched for Uber pool and it will basically give you you know how many of the last in the last couple of days how many tweets uh were that mentioned Uber poolool were neutral positive or negative. This is a you know the simple sort of sentiment analysis case and you can basically see that it's in many cases quite accurate and yeah so here we have you know talk about greenwash I guess they said you know earthday Uber pool of course like buses and trains might still be better so people were unhappy about that um at least for the environment maybe not for your time um and then uh you know here are some some positive things where people had a positive experience using Uber So this is one specific specific kind of example. Uh and it turns out that if you wanted to train your own classifier, you can also do this very very easily right now. It's basically gone to just drag drop and learn uh with with Metamine's uh technology here. So all you need to do is find a very like define a very simple text file where you have your labels, a tab character, and some text. And then you can basically train a classifier that predicts those kinds of labels. And that could be, you know, buy sell. It could be happy customer, unhappy customer
This could be this is a pool um versus appliance versus painting kind of request. Any kind of label you could associate with a simple text. Uh you can basically train with this. So all you need to do is you create a text file here. It's positive and negative. and the the type of text is a movie review and you basically yeah you have two options. One is you just write seven lines or so of Python code where you say here I have my um my specific data set. I create a classification model and you know this actually works for images and text and then you basically type model.fit and that's it
So, in seven lines of Python code, you could create your own. But you don't even need to have any code. You can just take this file I just showed you and drag and drop it into the browser and then click on upload and train. And now it will basically depending on how fast uh the Wi-Fi is. All right. um upload it, extract features and then save a model into a production ready setting such that you can now have you know in just three lines of testing code a production level classifier. It will tell you a little bit about uh some of the statistics. If you go in here, this is a little bit of a noisy data set
So you won't and it's not that big but it basically gets you to roughly 70% 77% and it'll compare different kinds of algorithms give you some statistics on your data set and then allow you to kind of do some error analysis and see where it works and where it fails right so basically in the last you know one minute we trained a simple text classifier on an arbitrary problem so I feel like this is now relatively commodity there's still some you know work if you want to uh for for a lot of other kinds of settings. If you want to have an production level, you know, system that might take you extra work and this is something that we do at Metammind right away also, but there are lots of tools that can get you a little bit there uh most of the way there. So now what what's the promise of deep learning? Why why should we use deep learning when you know well there's some deep learning in this also but in general you know there are lots of tools that can allow you to do a simple text classification and I think one of the nice examples and example applications to really help you understand where it shines and how it allows you why it allows you to push beyond sort of this 80% accuracy number and and go higher and higher is sentiment analysis. So basically if you want to have detailed fine grain sentiment it's still not quite solvable. You saw this you know very simple straightforward classifier that works for any language and uh any length of document only got you to 77. And so what we now also have is basically deep learning based sentiment analysis and other kinds of text classifiers. And when is this important? Well, if you just want to get a rough idea, you know, are people on general in general sort of more positive or more negatively inclined about a new product, then maybe you can get away with just 80% accuracy. But once there are decisions that are really important, you know, in medicine or in financial trading and things like that, then you really want to get those extra 5 to 10% of accuracy
And this is what deep learning allows you to do. So, um, I mentioned finance. This is an interesting example where we got a glimpse in the world of that you know this is actually happening uh in finance. So here Anne Hathaway in 2011 starred in a movie. The movie reviews came out and then all of a sudden the stocks for the company Birkshshire Hathaway went up significantly. And so we see here that this is clearly already being done. And in this case, you know, you had the wrong entity dismbiguation and uh the wrong kind of sentiment about or the right sentiment but about the wrong entity. And so this is a nice example of where this also fails
I think for very simple and very long documents sort of general reviews right we can get up to 90% uh even if it's a very simple kind of problem but once you want to really understand fine grain sentiment about specific entities uh from just a single sentence or a single tweet then it gets more and more important to really understand the details. So if your document is very large, you know, and somebody really tries to convey that they liked um a certain product or a certain movie, then they'll have lots of obvious words in there and you can kind of get away with it. um for single sentences for instance there was this interesting uh data set from Pan Lee from 2005 and until deep learning came around the accuracy had basically never reached above 80% over seven years of research on this data set in the academic community and what we basically found when we analyzed this is that once you want to push beyond 80 you really have to get the harder cases right and there are interesting problems with neo negation and the scope of negation and other semantic effects. So, let me give you two examples here. We already covered this one. And what I mean by negation scope is here that when you say doesn't care, that doesn't has a scope over the entire rest of the sentence. But it might not, right? It might have a comma and then say something else again. And here we have an interesting uh so-called contrastive conjunction
That's the word but. That's the linguistic term for it. And the sense is it's not life affirming. It's vulgar and mean, but I liked it. And so you might say again from just a simple you know counting up words it is negative. But with this contrastive conjunction when you have these and you say X but Y then usually a sentiment of Y matters more than that of X. And so one thing that you very quickly learn when you work with deep learning is that it needs more data. Uh so you're giving it basically less information
You're not telling it these are all my positive words and let me write this regular expression here and if I see not followed by these 10 words then it's really negative and then if I see doesn't and maybe doesn't plus some verb then you know you don't give it any of this information you just give it a bunch of examples and you're asking it now to learn as much as you have as a person from just those examples and so basically what you often want to do is either find clever ways to collect deep learning data from various sources or just use Amazon Mechanical Turk or CrowdFlower to just basically give more annotation to your data set. And so this is what we've done and we essentially collected, you barely see here the the lines. We've collected these tree structures where we have all the uh syntactically plausible phrases and we labeled them separately. So here we have life affirming, it's positive, not life affirming becomes negative. Bulgar is negative and this is basically a very negative overall phrase and then you say but and then it kind of relativizes the sentiment. So we basically collected some more data. Nowadays you can basically collect uh hundreds of thousands of labels for English in in a matter of a day or two thanks to Amazon Mechanical Turk and and other kinds of platforms like that. And then we can use deep learning
And I can't really go into too many details here on the algorithm. If you're actually interested in this, I'm also teaching a class right now at Stanford and all the material is available. So if you go to cs224d.stanford.edu, I'll put the URL up uh at the end. Again, you can really understand all the details of this of these kinds of models. But on a high level, what's going on when you try to use deep learning to understand this kind of sentence? you can still use uh things like word vectors which we just heard about um and classify single words as a first step. So here you would say all right not and bad you know bad is pretty negative not bad not by itself often appears in negative context so might be a little negative too and pretty could be you know the pretty as in beautiful or the pretty as in somewhat of an attenuator. So maybe if the the model might think in most cases it's seen as beautiful. So it might say this word by itself in isolation is very positive
Smart is very positive and actually it's just a neutral word. Now what there are different kinds of deep learning models. Uh here we look at a so-called recursive neural tensor network. Um I'm mildly biased uh in that choice because I invented the model a couple years ago uh during my PhD at Stanford. But basically what this model would now do is it would find syntactically plausible phrases and actually combine them and not just average you know the sentiment that you have but use these word vectors these you know highdimensional uh lists of numbers essentially and really understand how they combined form the meaning of now that biogram. So here you have not bad and it actually learns that despite both of those being negative when you say not bad in that combination it actually becomes a neutral phrase and likely similarly here it learns that pretty by itself might be very positive but in the case of pretty smart it actually is somewhat of an attenuator like quite smart or somewhat smart and it learned that that only you know it's not very positive but just somewhat positive and then it also learns that as you combine things with you know more neutral words depending on what those neutral words are depending on where they are in this highdimensional vector space, it stays positive and basically it can build up this entire structure up until it knows the final sentiment of the full sentence. Are there any questions so far? Right. Um that's a great question
So the question is sort of do we have intuition for that vector space? Yes. Uh if you if you train um if you train word vectors based purely on sentiment uh they actually look a little different. Um you can actually initialize them with glove or wordtovec and that can help in in some cases. If your data set is large enough the and it's just about sentiment then the larger your data set is the less necessary it is to initialize um the model with these pre-trained word vectors but you can still do it it rarely ever hurts. So if you have them around already, you may as well. Once you propagate all a lot of sentiment into the word vectors, they start to get better and better at predicting that and you know understanding how they should interact in the composition and then they move around in the vector space differently and we can gain some intuition uh by visualizing this properly. And so here you see a two-dimensional projection of that usually you know 25 to 300 dimensional vector space. And on the first principal component here, you basically see that uh I'll zoom in in a second, but up here we have very positive words and down here we have very negative words and in the center we have lots of content words
So if we now zoom in here you can see you know down here you have you know jokes, tries, problem TV. So you know this is a movie review. So if you the movie comes out and somebody mentions you know should go straight to TV instead of wasting your time in a movie theater it's not very positive uh pretentious lacking awful and so on and then as you move through the space you see more and more sort of neutral terms animation and novel belief you know and so on along occasionally beyond and then as you move higher up into the space you see more and more positive words such as beautiful journey, terrific, manages, intelligent and so on. And here the color actually comes from just a pure classification. So unfortunately for instance while intelligent is somewhere uh similar in a similar part of the vector space and close to terrific by itself it's actually unfortunately not that positive. Yes. So that that's exactly right. So everything in this model and this is again sort of a part of of something that you really want to in internalize when you work with deep learning
Everything is learnable. Like you really want to give it as raw of an input as possible and the output that you care about and you let it figure out everything in between. And that includes in this case the word vectors. Yeah, great question. Uhhuh. Yeah. Yeah. That's a great question
I'll actually I'll show you an example. Um actually maybe we can pull up an example right away. Uh so the question is what is what about the scope of negation? Can we actually capture this um in a principled way? And so here we have um we have exam an butt example. So let's uh I think I typed that in already. Um so this is the example sentence that I gave you. And here you see um how this works and how the scope of the negation can really be captured because of the tree very easily. So here you have this movie doesn't care about cleverness, wit or any other kind of intelligent humor. And basically this model was able to learn that yes indeed these are all positive things here and you can care about all these and it's positive
Once you don't care about them or this movie doesn't care then it becomes negative and that is exactly the kind of negation scope that this model can can learn by itself. All right. So where does that leave us in terms of the accuracy? Basically uh you can use more data helps always and it helps every single model. So if you really don't feel like developing any kind of new algorithm, you just want to get slightly like 1% higher performance, the simplest thing you can always do is go to Amazon Mechanical Turk and just collect more data for your problem. So all the models benefit from this additional data. However, the recursive neural tensor network benefits the most and there are now sort of nice extensions. uh don't want to go into too many details there but there so-called long short-term memories that uh is a different way to compose the vectors and you can use those in both uh sort of temporal chain structures and tree structures and that works even better but we're not going to go into too many details there recursive or recurrent recursive all of these are recursive yeah this is a constituency parse but they're they're deterministic mapping between the two. So are you also at the same time whatever that's a great question
So when we have these kinds of tree structures the question is should we learn the tree structure at the same time that we learn the sentiment and it turns out that you could do that uh but it doesn't gain you much and they're now very efficient also based on deep learning parsers that are uh you know linear in the sequence length and their their complexity and so you don't you can just assume the parser is fixed. You don't need to learn it. Right. Yeah, it's true. And I was worried too. We we're running a bunch of experiments on on Twitter data as well. Somewhat surprisingly, it it still works really really well. Uh, I think the parse structure is essentially, you know, you can get some things wrong sometimes and it doesn't hurt you very much
The only time it really hurts you is if you have a parser mistake that's so bad that, for instance, in this case of um of this sentence right here, it it said, you know, sort of doesn't care. And it put that very deep down in the tree, maybe down here. And now you see sort of how you know if this was actually if it mis mispared the sentence and put the dozen somewhere deep down inside a sub sub tree it will eventually not be able to overpower the rest of it. Um but in general especially for Twitter the sentences aren't that long so the tweets so you can get away with that. Yeah great question. Uh so we can we basically here uh pushed what you know for over seven years on this data set was below 80 to 85 and now with some more advanced deep learning techniques actually to 86 uh and and even up to like 88. Um we can also look at some more specific subsets where we know you really have to capture the right structure and understand a little bit more about um the the semantics of the English language in order to get them right. And here we have a subset of the data where we have these xy constructions and there are 131 cases and basically the simpler models here like bagram naive base are really really bad and uh you know then you can use various deep learning techniques and get higher and higher
So these kind of accuracies here are do you get the left side correct you get the right side correct and you get at the conjunction is also correct. So once you have you know even you know 90% or so and basically multiply you know 0.9 * 0.9 * 9 um and so this is actually now also even uh this is you know a number from the original paper and this has has improved now also significantly to uh around 60. So here's some more interesting negation results. Roger Dodger um some silly movie. Um and basically in this particular data set uh humans were asked to change the sentence very minimally and change the entire sentiment of it. So the simpler your model is, the more it just kind of averages all the words, the harder uh time it will have to really solve this this kind of task well. And so here uh Roger Dodger is one of the most compelling variations on this theme is positive and Roger Dodge is one of the least compelling variations on this theme. It immediately learns that you know at least one word difference here makes all the difference for the entire sentence
So least compelling versus most compelling changes things a lot. And here we can look a little bit into the internals. So here we have uh negation of negatives. And what's interesting is that uh first we thought oh maybe all the fancy deep learning stuff all it's done is it learned that whenever you see any kind of negation word and anything positive it'll just make everything more negative right so not great not good not cool just all everything becomes more negative once you see not and so uh fortunately we we then you know discovered that really it it does more than just that so not by itself here was negative bad by itself is very negative And this is sort of the distribution over the different classes that is internal to the model. And it really says this is mostly neutral with a slight hint of positive once you combine the two words to not and bad. And that's really the only model that we found that actually captured that effect to put more probability mass onto the neutral and positive classes when you combine a negative statement with a negation. All right. So so far we basically were able to see that there these deep learning models can capture some compositional meaning and jointly learn these tree structures the feature representations
We didn't have to you know give it you know regular expressions or anything like that and learn any kind of language prediction task. Sentiment analysis in the grand scheme of things while you know it gets harder and harder you have to understand more and more about your data if you want to you know push towards 90% it's still in the grand scheme of things a reasonably simple task especially compared to things like machine translation or question answering. So here the second task that we'll look at is question answering. It's actually a fun competition sort of like Jeopardy but mostly for for high schoolers and and college students. Uh it's called the quiz bowl challenge. It's basically a trivia competition. the more knowledge you have about trivial things. I mean, some of them are less trivia uh than others
You know, this is important literature and geography and things like that, but then there's also things like sports and uh other stuff. Uh so here, does anybody have an idea of who who this is? All right, very good. Um so basically the structure of this is that in the beginning, the first sentence you need to know maybe you need to have actually read one of his books, right? and really know that, you know, there's a, you know, some unfinished novel title character forges his father's signature, get out of school. You know, it's like you really have to remember all the books of all the many authors that are out there and and all their plot lines to really get this right. And then it gets simpler and simpler and at the very end of the sentence, it's just like, all right, it's a German author, you know, before that it could have been, you know, like lots of different could have been a Chinese uh short story or something um German author and you basically give it, you know, the actual books he has written. in that point you just need to have a higher level knowledge and like have heard of him and heard of like the most important book um that he's written. So one thing that's really amazing is you know I described this recursive model for sentiment and I'm glad everybody found their way. Um the amazing thing here is that the exact same model I described for sentiment will actually also solve question answering
Uh so we use that exame exact same type of recursive neural network and now instead of training it to predict at the top of the sentence here a sentiment we're just training it to predict the actual answer of these trivia competitions. So in this case you know uh we would like to have the root vector of that model be trained to predict the vector of Thomas man. And you know we can basically see here a couple of different uh distances of the various phrases that we have inside uh these parse trees. And at the very root here you can kind of see all right Thomas man is the highest scoring character for instance of the first sentence and then various other sort of more similar also German authors come next and then you know Henry James who's a British author comes after after them and so we can combine um the original training data here so this is you know assume you only have the training data in this kind of format uh and of course there you have to have seen all the various aspects of Thomas M in order to be able to answer it. But what if you you know in the test set you have facts that weren't mentioned ever in the training set about Thomas man. So here the trick is to also train this kind of model on Wikipedia and the actual Wikipedia articles of those authors or those historic events or those presidents or you know sports uh players and so on. So whoever has a Wikipedia article you can also train the model and try to predict the facts um uh in the Wikipedia articles. And so again now we can visualize the vector space
And what's interesting here is that you now capture very different things in the vector space. You know, you propagate sentiment into it. You push, you know, on movie reviews, you get TV and pretentious and horrible somewhere in the space. You propagate historical facts into the vectors and try to predict, you know, presidents and then you will see that you know Woodro Wilson and Ronald Reagan and Jimmy Carter are in a similar area of the vector space. And this is again sort of a big takeaway message here. Whatever task you have, you can propagate the information of that task into these word vectors and into these compositional models like recursive neural networks and then they will try to ingest as much information as possible in order to solve your prediction task. What's also kind of amazing is that uh this recursive neural network model can actually defeat human players in some cases. So here you have each column is basically one game that a person played against the computer and whenever it's blue the model wins and and maybe nowadays this is not so amazing because we've seen Jeopardy and and IBM's Watson system but the difference here is IBM's Watson system was built over many years by a gigantic team with you know hundreds of millions of dollars uh you know to to work with and this was built by one grad student using deep learning Right? This that gives you a little bit of like an idea of the power of you know learning the features automatically based on your task with deep learning versus going in and writing regular expressions coming up with lists of words that are important for your task and so on
Once you you just the amount of speed you can get um to a really good solution is just incredible. And so so far I've talked about two tasks that are basically only inside language. But what's cool is we can also learn grounded meaning representations. And what I mean by that is that we can connect these word vectors to visual things that we see in the world. And so again uh the amazing thing here is that that same model I described uh for sentiment analysis, the same one that was used for question answering is the same one that we can now use to try to map sentences into a part of the vector space in which we've also mapped images into and we try to learn that they're both they should be very close to each other. They're both very similar in the vector space. So what we have here is a training corpus where we say for instance a man wearing a helmet jumps on his bike near a beach. We use a recursive neural network model to project this into a vector space
And then we take and this is a different model. I can't go over it in too many details. It's a convolutional neural network. It's also a different type of deep learning algorithm. And we take the raw pixels of an image. We map it through this convolutional neural network and we train the model to say both of these vectors should be very close to one another and then that's that's the only thing we tell this model. Everything else it again learns and has to back propagate through the network update all these various uh parameters of the models such that that will really be the case. And so here are some examples of one going one direction which is I give you an image and now you have a very large pool of sentences and you try to find the sentences that best describe that image
And so here you have you know a jockey rides a brown and white horse and a ded curl and then you know the the next two are actually not correct. So in this particular data set there are usually five sentences describing each image and you know you can just select the best one. Uh and then you know the the third one is also wrong and you know quite so and then the fourth one is again correct and you can kind of see here also this is kind of funny. very first sentence that uh was picked by the model to visualize or describe this this image was correct. But then here you have a couple of things that are wrong. And you can kind of see that the way you would want to analyze your accuracy is a little tricky because while this sentence for instance seems kind of funny to a human reader, an elderly woman catches a ride on the back of a motorcycle and it's kind of ridiculous because it's motocross and be quite dangerous for her spine. Um however in general if you took out elderly it is a person that you know is you know or taking a ride on the back of something that looks like a bicycle from afar maybe right so it's like in terms of engram overlap it's not so bad um so we can actually look at uh a demo of this that we sometimes bring up um we basically here uh can type in any kind of sentence and or phrase so this one is a couple sitting on a couch and it will now pull in images and the closer the image is to the center text box, the closer these vectors are in the joint vector space. So I can type here uh for instance bird and it will show me pictures of a single bird
That is kind of the standard image classification just one label for each image. But now I can say uh birds for instance in plural and now it actually shows me images of multiple birds. And again, the only thing this model has seen were lots of training examples of sentences and images. It doesn't actually tell you in for each sentence, you know, where that particular object is or anything. So, uh this is what what the data set looks like for training. Here you have lots of different kinds of things like a cat lying laying on a red cushion looking at the camera and so on. Lots of lots of training samples. Um, and now the model has to pick up these things by itself
So here we started with bird and now we say birds and the model actually has some sense of plurality and actually now shows you pictures of multiple birds. Then you can go to birds on water and now it shows you mostly pictures of birds that are actually on water and you know you can change this to different things like birds in trees. Now, I'll try to find images that are actually of birds and trees. And this is I would love to put this demo up, but you kind of have to know what kinds of things the model has seen. So, if you type in like a zebra on a red bus, then you know it'll just show you something random. Uh, and it'll have a low score, but it will look pretty terrible, right? Uh, so you kind of have to know what kinds of things this model has seen because again, there there's there's very there's basically no human engineering here. It just sees a bunch of pixels and a bunch of sentences together with those pixel and has to really capture all the rest. Uh so we can have one of your products
Yeah. So this is uh if you actually have um a related problem like that then this is something you can work with us to to work into your products and it'll be like easy APIs and so on to to do. So here we have horse and you can go all the way to like horse next to bald man and it actually finds that one image in the data set where a bald man is next to a horse. All right. Uh with that I want to conclude. Basically, I hope I could, you know, get you excited to explore deep learning uh largely because deep learning can learn grounded representations in the terms of uh in terms of images as well as compositional representations to really understand how words are put together. Not just classifying them in isolation, not ignoring the word order that is clearly important to natural language, but really understanding how words compose the meaning of longer utterances. And that combination can be, you know, employed in a variety of different tasks that you require either world or visual knowledge
And sorry for not being able to go into too many details for those of you who are more familiar with these models. But uh if you're very interested in actually, you know, understanding all the equations and how these models work, I'm teaching right now a class uh at Stanford called deep learning for natural language processing. And all the material is up online and you can read things there. Thank you. [Applause] I think we have two minutes for questions or so. Sure. Right. Uh so so the question is you know can you actually make these models scalable compared to very simple baselines and uh the the answer is when I was a PhD student in Stanford and the the model that you can download with the code I didn't care that much about speed because I didn't run it on millions of documents
Now we're a company and we have customers who want to classify millions or even billions of uh items each month and we worked on on the speed and now we have models that are a lot faster. Basically almost as fast. You can't really beat you know just counting each word and and summing it up but um you know not many orders of magnitude slower anymore uh compared to the simple models and they still have a much much better accuracy. So yes uh a lot has improved. Yes. uh in some cases it's based on GPUs but in other cases even without GPUs we can have very very fast deep learning models but those are proprietary so training oh uh I mean train sorry for the the time or the accuracy oh sorry um so yeah so that's a that's a great question so oftent times the training uh is often slower than than testing um and So that model is just it's just a better model that is faster at both training and testing. In many cases, we don't optimize that much for training speed. Um you can kind you kind of saw what you know when I when I drag and drop this file into the browser and already is really fast, right? Um that was also a reasonably small data set only had like I think 10,000 or so lines and uh that's a few seconds
uh but we still want to train on like many millions of lines and there um there's certain things that we have where if you go beyond beyond a certain level we'll we'll talk to you uh in person and maybe put it on our own uh very beefy machines to to speed it up even further. So right now is there a single model classification model we often we often compare different models and actually see and and often times there's there's sort of a couple of simpler models that we try for free. Uh and then once you say all right but I really want to have you know these 5% and I I need you to guarantee me it's above 90 on like any random sample of like 10% of my data. We'll talk to you and then we'll we'll we'll do some extra magic. Yeah. You talked about negation, double negation. What about sarcasm? Sarcasm is a very interesting question. It comes up uh it comes up a lot and and when I'm teaching this class all the students have have class projects and and there every year there are some students who say like I want to work on sarcasm detection and the the first question I have is do you have a data set? Uh and that usually kills that kills 95% of the the projects
The other problem is that in many cases the sarcasm you can get some things very simply like you say comma yeah right right and then you know whatever happened before it's probably you meant it as a joke and you're being sarcastic but in order to really do well on sarcasm you have to know a lot more about the person you like saying what you want to classify right I can have two friends and they both say the exact same word which is I had a great weekend coding the entire time and one hates his job and wants to like, you know, he's obviously being sarcastic because he would have much rather go hang gliding. And the other one is like a super hardcore geek. He loves coding and any waking moment he has he he wants to do it. And so, you know, is can you know that from just looking at that sentence? No, you really have to understand sort of a long history of people. How often are they sarcastic? What are their preferences and so on. So, it's it's a pretty complex problem which is mostly lacking interesting data sets to really make a headway on. Yeah. So for things like Yeah, that's a great question
So yes, uh you can actually very easily incorporate almost any kind of external thing. So here we had for images, right? We just basically in deep learning everything is a vector and it turns out everything in the world can be made into a vector. um which is you know it's kind of crazy to say that um but it's it's very true and so anything that you can put into a vector you can now add at you know any given layer of any deep learning model and then it will start incorporating that too and you can learn different vectors also for movies and we've done that actually also we have a paper um from I think a year or two ago where we train on learning whether two entities are in specific relationships like cat has part tail so has part is a relation relationship. Tail is a vector, cat is a vector and you learn and push uh these kinds of facts into into these vectors. But I guess in this case I'm talking. So, so these kinds of models give you vector representations for the text and then the table you can also put into vector space by saying every entity in that table, every row, every you know cell in the table is can be represented as a vector and then whether two entities in a table are actually you know in the table or not can be a classification problem and then you can put everything together. Um I have a paper I can point you to afterwards. It's called learning um knowledge based representations
Um, all right. I think we'll we'll have to end here, but I'll be happy to chat uh after. One last one. It turns out surprising little even on Twitter it works. Thank you. Thank you.