Text By the Bay 2015: Samiur Rahman, Practical NLP Applications of Deep Learning
Recording: Text By the Bay 2015: Samiur Rahman, Practical NLP Applications of Deep Learning
you thanks thanks Reggie thanks everyone for coming to this talk i guess i'm going to focus more on the kind of practical NLP applications of deep learning i know a lot of people have already spoken about word vectors but uh hopefully you'll find my my spit of it a little bit different and focused on like a problem that we're actually solving at madame arc so hopefully that will be a little bit different than all the other talks that we've seen so what is deep learning uh actually if you've been to either Adams talk from yesterday or mike's talk like tamir's talk today all of you should be aware that it's just Turtles all the way down right it's just a network of neural networks that is just bigger we've had neural networks for a long time as a tool chain we've just recently in the last five or ten years been able to train with better algorithms larger neural networks with different network topologies that you know we weren't aware would solve certain problems in the past so it's notnot it's nothing new in in one sense but it's completely revolutionary in another sense because it's solving all these problems that we previously were not able to solve and that is probably due to a combination of you know being able to do GPU computation all the new algorithms that helped train neural networks in in ways that are far more efficient than that proc of just back propagation used to be so yeah we it's kind of incredible how how much neural networks have you know kind of solved certain problems so like before we get into just like natural language processing improvements that neural networks have brought us computer vision with convolutional neural networks is something that everyone I'm sure has heard of that deep learning has broken the state of the art multiple times in the last like five years so I have any of you seen the demo from google that lets you put in an image like the paper that they put out that you can put an image and it generates a text description for you that's incredible isn't that like I mean five years ago we didn't we weren't even able to like Google reverse image search didn't exist all the kind of amazing improvements in computer vision that's happened for through deep learning it's pretty pretty cool so other than I guess speech recognition kind of falls in the realm of natural language processing but since we are focusing more on text here this is kind of cool outside of the realm of text that baidu along with Andrew hang from Stanford has built like a state-of-the-art speech recognition system with deep learning like they took seven thousand hours of conversation and coupled it with like hot synthesized noise and then they processed it through the massive GPU cluster that they've built up with the the money that bite is financed in deep learning with to andrew wang and they were they've built a system that you know they hurt that it beats google's voice api so that's it's pretty impressive that like they were able to just kind of start up within the last what I think Andrew moved over there like a year or two ago like so it built something within the year that beats Google's system that's been developed over like last ten years and I think the coolest thing is really what I was just talking about this cross domain representation that the neural neural networks and deep learning allows you to do this representation learning that seems to be like you can distribute models of things across different tasks so you can like have a computer vision model deep neural network for an image that then feeds into a other neural network for generating text so like you're and then you're kind of like pairing these two things together and they can be trained completely separately from each other and then the output of let's say in this case this is really cool that we can take an image and turn it into text but the coolest thing is if we want to do something else with it like let's say we want to take that image and then we want to I don't know run similarity searches on that image or some sort of other classification other than generating text or maybe I want to generate like similar images to this image any any sort of other task I want to do this first part of that neural network and it's it seems to be that we can just like apply these kinds of networks and these representations across like different tasks you don't have to replicate the tasks like you normally have to do with like you can't really in especially in text you can't really take your bag of words representations for one corpus and then like transfer it over and do it again use that same representation you have to redo all of that for every task so the greatest I think the most powerful thing that we've gotten from like these last five years or five to ten years of like advances in representation learning is the distributed modeling that we are getting from this so what about NLP so what deep learning NLP has been like making strides and breaking the state of the art in is things like obviously we're to check everyone's probably heard about that like a hundred times by now so this distributed board of representations with Ward embeddings with work to bag and glove I know I think Richard so sure is talking read after me and he was part of the team that created glove and then things like dependency parsing which I'll get into a little bit which is kind of taking text and turning it into taking a sentence and turning it into a grammatical tree that you can then do other things with sentiment analysis is something that's pretty common I think someone else gave it talk about how they do sentiment analysis and so many others so I'm going to get into a few of these first before we get into the application so what I know everyone's heard how we're tyvek is awesome and ward embeddings are awesome but go over it quickly again for a little for anyone that's missed that so the difference the standard way in kind of NLP or like machine learning algorithms doing any kind of classification or modeling on text is to do bag of words and then do some sort of transform on that or not at all maybe some people just do backwards and then don't even do the term frequency inverse document frequency transform when you can do that but generally what happens is on a large enough corpus you will easily get to like 50,000 dimensions and if you if you have like fifty thousand dimensions and you only have a thousand labeled documents you're probably going to get over fitted or in general it's not going to scale very well so like the problem that these board representations of this word embeddings are like helping us solve is that now we're squeezing the vector space down a lot like you can get it down to 200 400 m at 200 to 400 dimensions and still represent the word really well and use that as an input to whatever model behave and that way you only really you clearly won't need any where the curse of dimensionality that you would with 20,000 dimensions right so the other cool thing is you'll see that like you know in bag-of-words motel and hotel most of us would agree there's a pretty similar words they there's no in that encoding there's no way to know that there's similar work there's no like these are all each dimension is orthogonal and it means one thing and that's it and you can't do you don't know there's not know like kind of similarity metric in the spectrum however you can you can use metrics like cosine similarity and other things too that are pretty standard when you have when you're talking about like vector space math that actually isn't just like orthogonal vectors in each direction they're very similar to each other so we'll get into so it makes it that you can you can actually talk about semantic similarity which means like motel and hotel are pretty similar or but it also encodes information about verbs or like syntactic similar similarity like verbs or like what part of speech and things like that so like the word play or played and fulfilled could would be similar in a certain dimension because they're both verbs of the past tense so this kind of stuff normally you'd have to actually run through like an NLP pipeline of our speech tagging and all that stuff that you're kind of just getting free encoded with this word of any so I know you've probably seen this man woman king queen thing a couple times already but it's really awesome that like these analogies are just being encoded right into the spectre space when you when you train it with large enough data like the relationship between France and Paris is the same relationship a zillion rum so you could literally do like france- Italy plus Paris and those vectors would actually represent like be close to Rome so like this gives you an incredible amount of understanding into things like keywords that are extracting from your text in the past you could just you would just get keywords and you don't actually have an understanding you probably don't even know that this keyword that says games and then the other one that says toys are you just have games and toys as keywords keywords for some website and you don't know that they're pretty similar in meaning and you can't really relate that back but with something like this you can use it so that you can build up like a meaning of what the keywords that you're extracting so another application and from deep learning is the dependency parsing so that as first thing is is a kind of method to take the sentence like a sentence and turn it into a grammatical tree so it's basically like taking a sentence and trying to say okay so what's the root word what's the subject of that word what are other like Howard power each of these words related syntactically in this sentence this is pretty helpful in if you really want to like build sentence level classifiers or you know do any kind of like named entity extraction any kind of identification based on the syntactic relationships in the text that you're looking at so something based on dependency parsing which is a little bit simpler this is like this is pretty cumbersome representation right you know you're looking through like oh this is the route this is the subject this is a time operator so like if you have to build like some sort of kind of tree search algorithm this would be pretty difficult to do if you need something simpler you can use the technique called semantic role labeling of labeling and this task has actually also been improved by deep neural networks and the way that it works is so let's take the sentence matter mark is today announcing that it has raised a round of 6.5 million dollars well that's not an actual announcement that's a sentence that I just put in so don't take that at a quote that somewhere but what you do is you turn a sentence and take phrases and can split it up by the verbs that are in the phrase and their agents so agent 0 is generally analogous to a subject and age one is generally analogous analogous to an object so the word map announced you can see that matter mark does the announcing of some other rings and then its kind of split up down this way I'll get into how we use this and how this is really powerful and in like practical applications but you can kind of see that like there's a lot of meaning that you can extract you can kind of start to see like maybe maybe I want to know about facts that talk about announcements and I want to know who's the agents that are doing the announcements and maybe I can build a classifier that works on word level at this kind of stuff like what if i want to find every sentence that mentions an ounce and or something similar and you build a word tyvek based classifier so I along the same lines there Richard Souter also worked on one of the I think there's a demo on line of this you can go to Stanford's NLP sent to an analysis website and they broke the at the time i think is two years ago they broke the state of the art record for the center of the pendant i think it was a thing it's a penn treebank sentiment analysis data set by like five percent so huge improvement in sentiment analysis by doing recursive neural networks taking a dependency parts the parts that we were just talking about up here so taking this tree kind of structure of sentences and then encoding it like this and then learning based on this turk taking that into a turning that into a recursive neural network and then you know doing an oral learning task on that and it turns out it worked really well for sentiment analysis so let's get to the applications part already what does Madame Ark do with natural language processing so before we talk about what matter mark does natural language processing it might be a good idea to talk about what matter mark does in general so matter mark is a company that minds and tracks data over time of businesses in the world and there's a lot of great data that's just kind of lying around in the web and news articles and websites about businesses and we want to mine all that but we also need to mine it into a structured form that's usable for our company so you know machine learning and natural language processing is a pretty core part of our business to extract data so what are the tools that we use these are things that I can recommend and I've used in experimentation and production and they're really great tools and i'll be putting this PDF up where ever like somewhere I guess if anyone needs it I will send it out to you guys but inside this there's links to all these tools and where you can get it so probably anyone who's done any work in deep learning or no networks probably knows theano and Pilar and to already it's a python-based library that their guys from University of Montreal yoshua bengio slab Lisa lab-created pie brain is a great kind of prototyping tool it's not very fast it doesn't have gpu-based computation built into it but if you you know if you want to just get started it's far easier to get started on then like any of any of these other libraries just don't expect to get anything in production most likely with vibrate Jensen has a great implementation Jensen is a another Python library there's a great implementation of war tyvek that you can like use to train your own board vector models know learn is a very scikit-learn esque python learning framework it works great because you can kind of pipe it straight into all the great scikit-learn cross-validation kind of metric stuff that you you're the pipelining interface that you have with scikit-learn so you know if you if you spend a lot of time in scikit-learn no learners a good thing to look at in the java closure scala world uh Adam Gibson I think he talked about yesterday deep learning for jay is a great great framework it's it's being actively developed and it's grown a lot it's pretty very performant and you can like scale it out on Hadoop or on GPU so it's pretty cool neural networks very aptly named library is another great library that supports GPU so the other one the other the only API that I can recommend is actually meta mind Richard soldier I think is the CTO of this company and they're doing some really cool work where you can basically just upload your data set create a private model on their site and then query the model using a REST API with test data so it's the only one that I found you know compared to like Google prediction and all that stuff that does a really good job on text so let's let's talk about our problem that matter work so let's something that we've been working on quite a bit is let's build a binary classifier that can tell that any sentence from knew that if any sentence from a news article is about funding or not right for example the one that we just talked about madam arc today is today announcing that it has raised a round of 6.5 million dollars so pretty self-explanatory problem this way we we can extort extracting data so how do we do that so first of all let's talk about what we do with our word vectors how do we get word vectors so we don't actually use the either glove or work to vex you know pre-trained word vectors we actually trained it ourselves with a huge data corpus so we took a giant online available corpus the UMBC web-based corpus which has like forty eight gigabytes of text and kind of trained a baseline or tyvek model and then we took we've got a lot of our own data in news articles since we're we have like a base level we're tyvek implementation and the effect model and then since we want to encode you know the usage of language that's kind of common in tech journalism things like Internet of Things or other phrases that you don't never normally you probably wouldn't see and the rest of the web we still want to like we want to know about that because our domain is that those kinds of news articles right so we iterated 20 times on Tex from news articles that we've collected and the techniques domain so you know that's the reason why we iterated 20 times on it because it's a far smaller corpus than that corpus but we still want to move it push it towards that towards this domain so how do we take a word vectors and turn them into kind of like a sentence vector because in this case we we don't like we want to represent the whole sentence and figure out based on the sentence is this a funding sentence around so there's a couple different methods that we and initially thought about trying the first one is something that came out after we started working on this it's called the the paragraph vector model there's a link to the paper if you want to look it up there's an implementation of power of paragraph vectors and jensen's so if you want to try it out pretty pretty easy to try out we could feed it into a recursive neural network constructed by a dependency tree the same way the sentiment analysis tool that we just showed you works we could try putting it into a convolutional neural network which is a little bit it's a little bit like a recursive normal network but it's generally used for vision computer vision more than text but this recent paper I think it came out like a couple months ago showed that it did yeah it does really well with sentence level data without needing a dependency tree so you can literally just have a window of text that you're putting into this convolutional neural network without encoding it in the tree structure so you don't have to incur the the dependency tree performance hit and the other thing that we wanted to try is just other heuristic functions I'm a I'm a I come from a digital signal processing and EE background so I had ideas of how to combine vectors of things into a another single vector too so I want to try those so what did we end up trying so we did not try most of the stuff that I mentioned before but we tried a bunch of different composition methods so love to try all the other things that I mentioned before going forward but here's the things that we did try so as a baseline obviously let's put in TF IDF and try you know give do like a good naive Bayes make sure that like hey why do we why do this all this extra work if a simple model can actually solve the problem the hunch was it probably couldn't because at a sentence level with the amount of labeled data that we had there would be a high level of variance the dimensionality would be too large like we talked about and like the classifier would almost guarantee to overfit and that's what we thought but we needed to test that out then try these different like things like we'll talk about each one worked avec and then composing those words in different ways like do we just add the word vectors all together do we do some stop board removal and then adding them together do we like wait some words more than others those kinds of things like what can we do heuristic alee to create a sentence vector we could also add tf-idf into work to Veck right to do the waiting so if we use the inverse document frequency of each of those words and then wait the vectors maybe that'll give us some improvements and the other thing we could do is we could bring in the semantic role labeling about like knowing that an ounce is a verb that matter mark is an agent agent 0 and then the other the rest of the phrase was agent 1 maybe we can do that use that and do something so here's what here's the different composition methods we tried so given that let's say W sub I is the i thwarted vector and then W sub V is the word vector for the verb a 0 and a 1 or the agents here are some of the composition methods we tried so this was kind of based on my the ve signal processing background I want to try like hey it's very possible that something like cross correlation or convolution could give encode relationships between words in a in a way that gives us sentence like level information so we tried out obviously just add the vectors then ok how about multiplying the vectors which try that too how about we take the sum of the convolution of each of the vectors with each other we haven't used any of the semantic role labeling yet by the way on the fourth one we try ok what if we take the the convolution of the verb with the vert of the words from the subject so in this case an ounce is a verb and a zero matter mark is 80 so take the verb involve it with Madame arc and then add it to the ancient one involved with the verb and then add the remaining vectors so like what we've the map of the operation would be basically like now it's involved with Madame re sup + this whole phrase come by involved with an ounce then add isn't today right so that's one method and then similarly this is pretty similar this would be a tensor product instead of a convolution and be like inner product of the vectors and this is another combination method and the dot product of those oh sorry maybe I should just how about that that's better so yeah so these are goofy ideas that I wanted to try out it looks like could this really work so it turns out that the fourth method this guy was the one that worked really well for our task at hand so the first method with tf-idf too naive Bayes as expected performed pretty poorly because of its large mention allottee combining tf-idf with were tyvek provided a small but noticeable improvement like definitely over adding the plane word vectors there seemed to be that we were adding some context to the domain of the task at hand then adding the semantic role labeling and the sophisticated well not sophisticated I don't why it's not I guess the composition method of circular convolution with the semantic role labeling so like huge that's that's actually a pretty large improvement so turns out that you can kind of combine different different neural network outputs and then use it in your own way to make it work in your task so what else is possible can we take these word vectors and create document vectors so Mike tamir from galvanized just talked about one method of taking word vectors and composing to document vectors but there's a lot of different ways to do it right I just talked about what like a bunch of different ways that I tried out how to make sentence vectors you could totally apply the same method to document vectors and just see like how well it does for your task at hand we currently are using lda a topic analysis method for some of our tasks but we're planning on trying out document vectors for you know improving that task the word working on right now or or future tasks that we work on in our case like let's say we want to we've got information about companies we've got text from about companies what if we want to let users search for companies through some search box like give me give me lollipop companies and normally that would what we do is right now we use elastic search to do keyword searching but what if we wanted to take a more machine learning natural language processing approach to it we could create document vectors and then do a similarity search with the phrase vector of in this case lollipop companies and find documents of companies that most are most similar to that phrase maker so how would that how I I want to answer the question I haven't done it yet but how will this compared to like the other methods that we just talked about like paragraph vectors recursive neural networks convolutional neural networks there's so much so much stuff to try out what this is this is why it like the kind of current research in using deep learning for NLP has made it has given us a lot more a lot more options to solve the problems that at hand so what else did we do so I just talked about this document search thing right yeah we could we could take artificial intelligence and search for companies and then get better results because some companies will talk about machine learning and maybe it doesn't even appear at all in their description or their website that the artificial intelligence company but really like if someone's searching for artificial intelligence then most likely a machine learning company is also what they're searching for I'm using word vectors you could probably get a lot of that out right you can easily search against those documentaries so that's about it if you guys have any questions I'd love to answer them but Madame arc is hiring machine learning engineers and software engineers so if you want to talk to me about that and solve cool natural language processing problems talk to me after the after this or reach out to me on Twitter or email that's about it any questions yeah right so we've trained two different models actually one of them is you know context-free in the sense like there's no power of speech tax so like if you search for apple actually there it gets out a little bit of context if you use capital case a then you might get more information about the Apple the company and if you use lowercase a it's almost certain that it's the Apple like the fruit however we do have another model that is part of speech tag so we would do like Apple underscore n N or Apple I guess I forget the actual right so when we're not doing any like named entity tagging where we are doing more like part-of-speech tagging so i guess the difference between apple the fruit an apple the company will be a little bit different difficult to capture other than maybe if the sentence captured that it's a proper noun so if if you if the part of speech tagger is pretty sophisticated then you could potentially code that yeah any other questions thank you