Doing NLP with Transformers

SF Scala: Alex O'Connor, Doing NLP with Transformers

Recording: SF Scala: Alex O'Connor, Doing NLP with Transformers

[Music] start with the star are there any NLP people here already okay a few so I apologize to people in advance this is a little bit of a quick shot through this topic but I hopefully practical for people um I also have to begin with an apology this is the NLP side of stuff so I don't get to talk about generative design for JPL Landers or anything like that that's another department it's always great to be reminded of the COO of your cooler cousins when by someone else in the company but hopefully hopefully people will in nonetheless be interested how many people have heard of Autodesk excellent something seems to be two kinds of crys one group of crowds who look at me and stony silence and the other half who know what they know a little bit about us so very briefly Autodesk is the company that builds whose customers build almost everything we have almost any project that involves a large-scale construction or 3d modeling or a lot of M&E these days so a lot of animation 3d animation those sorts of things VRA are 3d printing all that sort of stuff our software is a big component of what people do to make that work and increasingly we're involved in what we call the future of making so we're fascinated by this concept of how machine assisted design is changing the way people work changing the way people create everything these days so one of the conversations that my colleagues and I are having a lot of the time with design professionals is what happens when you're not just faced with a blank page anymore what happens if you the sort of experience that you have when you use Gmail and it starts to auto compose to the point where you wonder whether or not you really need to be sitting at the keyboard anymore happens to our architects and our engineers and our friends like that so that we can do things like optimize layouts we can do things like grow in an evolutionary fashion all of those sorts of things so that's the cool part of Autodesk and that's the exciting stuff and we have a global and we have a global customer base in extraordinarily complicated tasks engaged in real serious projects and important work and that's where my team comes in so what we do is that we deal with when things go wrong with our product which doesn't happen very often I'm required to say but don't when it does happen the agents themselves need to be able to help people and the key thing for us and the thing that makes this such a fast environment for me as someone involved in NLP is that we have here an expert user base we have a group of people who have immense ability in what they do they're engineers they're designers they're animators they're architects they're physicists there are anything you can possibly think of and sometimes the problems that they have looked like the problem on the right there which is that they're doing a finite element analysis or they're doing a continuous flow analysis of a car body and answering a question about how our model works or answering a question about how that system works requires an immensely capable agent in the backend as well so we have employment we employ people who are very very smart about how their products work that means that their time is extremely valuable that means that for example with one of the projects a bit like this there was someone doing some work on em it was aerospace dragon dragon Alice's for an arrant aerospace structure it took two days for our an HPC cluster to just resolve the code before they could even begin to debug it so there's an investment of time there that's super important on the other side we also get all the queries that are like I can't find the download on the website my floss my password I can't remember what's going on and the difference between those things is how is everything that we are involved in improving as a data science team how do we make sure that our customer gets a response in 15 minutes if we can answer their question in 15 minutes and how can we make sure that a customer gets to talk to the one person in the world who can answer their question if that's who needs to be talked to and as an NLP practitioner as someone who's working in language this is a uniquely wonderful way to work on things because truly you don't get this opportunity to work in such a broad base of highly domain specific terminology very often so fundamentally what is natural language processing and I like to think of it in a very concrete way how do we take a set of words represent their concepts their context and their meaning in a fashion that is usable by a computer and this can be very simple this can literally be we're just going to match individual tokens separated by spaces for languages that have spaces and we're just going to say how often did the word angry get used and that's predictive to some extent but then we know a lot about how she means communicate we know that humans do not communicate in a very straightforward fashion if language were purely an information exchange approach our information exchange medium and I'm going to be much more pedestrian but language is an attention game right language is about maintaining the attention of the crowd or maintaining the attention of the person you're speaking to so you pun you refer you use acronyms you use metaphors you do all sorts of games to try and keep people's attention you wave your hands about you use people's names and all of those things Aleksey and what we're doing there is that we're playing an attention game which means that for computational purposes is extremely difficult for example in customer support sometimes we get a customer that says this is not working and I am angry about it but very often we just get a lot of capital letters and we have to infer that that's a problem and so the same kind of things happen how do we go from this very simple counting of individual words counting of individual tokens counting of individual section segments into something that actually represents somewhat the meaning of language so there's this very nice book that's been out recently called because internet understanding the new rules of language and it's a kind of an analysis of some of the ways in which we've changed the way we speak because of Twitter because of Facebook of course because of Amazon and everyone else and how they all shaped the way we changed the way we describe things the way we use words Prime has changed meaning very recently in a substantial way we used to use it a good example of tablet which used to mean if you're from the UK and Ireland meant the pills that you take as medicine or it meant the thing with the commandments aren't and now does not mean those things mostly most often when you talk about tablet you're talking about a what are they called fire tablet is that what I'm supposed to talk about here um but the critical thing is this even when we look at something incoherent to the outsider and let me tell you some of those complicated aerospace diagram things that's about 17 hours of Wikipedia browsing just to understand every word in the sentence even in those situations our customers use a coherent model they themselves show statistical relevance in what they say they say things in patterns and we can use power of computation to reflect those patterns and that is the entire thing that we're trying to do here and in practical terms that what we're interested in so the rest of this talk I'm going to be very good to earth I'm going to talk about how you actually go from this problem of having a blob of text that you're worried about into something that you can operationalize we're using relatively state-of-the-art technology and the answer is you use PI torch you use sage maker and you use a little bit of manual annotation and when I say a little bit I mean a lot but that's part of the process so the first test that you have is very rarely do you get your data in a nice clean format in NLP I think of all of our of all the domains that we have to deal with and natural language processing is sometimes the most difficult because in practical terms inside companies no one is thinking about the poor natural language technology people when they're thinking about storing data in a database they're just putting it in a grid somewhere and you have to try and remove it so this is a kind of a neutralized typical example of the sort of support ticket that we might have to deal with we get some metadata that we don't care about and then we get the actual text from our customer and we need to do a couple of things we need to recognize that core text and for example if you get a thread a dozen email so it's like really I hate your company and then at the bottom you have to extract that last piece of text from it that's the sort of thing that everyone thinks is a one hour job and then it turns into an entire PhD and what we do with it is we then have to think about what's inside there okay so in this situation Revit has been the tool that's mentioned it's one of our design tools the customer has put in a serial number that we don't want to spread around so we're going to replace that because that's a sensitive piece of information and often we get typos we get variance in language we often don't have native English speakers where people are typing stuff or have clearly Google translated we also have to deal with that sort of thing because that doesn't always produce the data properly so the first thing you have to do is think in terms of pipelines you think about the fact that you have raw data raw text stored somewhere and periodically you have to strip out everything down to this piece of text that you're interested in and then sometimes you have the advantage that you can look for specific tokens that you know are unambiguous if you work in Microsoft Office this example will anger you because while it's relatively easy to recognize Revit it's very difficult to recognize word and even harder to recognize Excel as tokens that are not something otherwise used or other with other context so my lesson is if you work with product teams tell them to make easily recognizable tokens for the names of your products and NLP pipelines torch text does some very nice ones Spacey does some extremely good ones there's a lot of tools out there that will help you to do that but ultimately for our team the key challenge is this how do we help our customers ask the questions they need answers in their own words because the hardest challenge for a lot of companies is they have a lot of documentation they have a lot of organization they have a lot of websites they have all of this information but they think about it in terms of how it reflects their organizational structure and their people's way of thinking about things and their people's way of organizing things and there is there hasn't been that shrink-wrap test there hasn't been that thing where you get someone who is technically able but not aware to try and actually find something on the website and anyone who's had to use a restaurant website ever knows exactly how difficult this is and so we as a team are interested in three things we're interested in using the state-of-the-art technology because that's the best way to get things done and it's the most fun we want to do it in rapidly because we don't have the luxury of spending a lot of time on the research of this we want to get stuff into production fast so that we find out what's wrong with it because very often your model is broken your assumptions about the variation of language are gone and every three months we change our a lot of our language changes anyway because of the way our products evolve and the last thing is we want to be able to share stuff both inside our team and around with every other team as well because it's critical for us to be able to get this results of our analysis out to not have to redo the work every doesn't a dozen times every time we have to extract this data and we do it consistently so for us there are a couple technologies that I'd like to talk about in detail does anyone recognize the leftmost symbol the hugging face I still don't know why it's called that but it's a fabulous company with a fabulous piece of technology hugging face has done a great job in taking the state of the art in natural language processing technology and making available to the world it is a net good of enormous size fast AI is a another set of libraries that have text analytics as part of their work it's also a course on how to learn deep learning I think it's actually by someone who's in the bay area Jeremy hard so I will talk a little bit about fast AI and finally you know drumroll please pi torch which we think is a particularly good way of doing these things so why do we like pi torch access to research code is the number one thing it is very easy to get quick access to what's going on it's nice to be able to read a paper at ACL and see the code pop up on github very quickly later in a platform that we understand and that we're ready to use it has the intuitive interfaces there's new normally pretty much only one way to do something I did some work as an undergraduate in pearl pearls whole thing was about doing it more than one way and honestly that was a terrible idea let's just tell everyone how we should do things in one particular way and I we appreciate your strong opinions on how to implement things because it makes life much easier and not everyone wants to spend the day working on Rubik's Cube code and new features there's an immense amount of code being produced which I mean Brad is I love that detector on video I have no use for it but I keep thinking I'm gonna find a way some way to have maybe I'll have customers wave the box in front of the webcam and they'll identify their product that way but that sort of stuff is very cool and you know captain we found to be very exciting all the stuff around pi techs is very exciting and everything to do with large models is greater so my torch is great support as a as a topic is under loved I feel as a domain I think that it is one of the most interesting ones to do and NLP is awesome so what is a transformer the reason that natural language processing has got so interesting in the last little while is the fact that we have an way of thinking about how to solve these problems this all kind of started about seven years ago when Thomas mikhalev at Google um started a project where he was able to finally think about this idea of how do we get from words to something that is a differentiable continuous space from a machine learning and word proximity was always a difficult thing before that point because concepts are associated with each other we've known that since the distribution hypothesis in the 50s but how do we actually start to get into a into a mathematical shape into a mathematical space that we can actually use for something smooth and learn about and differentiable like machine learning so period over the last couple of years we've been moving forward until the point where we get to now where there was a paper in 2017 called attention is all you need where they demonstrated that all of the work we had been doing up until now and recurrent neural networks on convolutional neural networks and all those sorts of things this feature called attention turned out to be the most interesting part of it all very simply what is attention attention says that I'm not just going to look at one word at a time I'm going to look at the word and I'm gonna look at everything around it and every meaning about every sentence comes from how these words are together within the segment that I'm interested in and I like to use the phrase the words words of a feather flock together so similar words appear together similar words appear in similar sentences in a similar way and by that process of learning we can understand all of the words and we can understand what they mean in terms of their context in terms of their meaning and in terms of their conceptual representation and transformers in that sense represent this kind of unprecedented ly large scale in doing that where we just have more more more and more data but it's also about the fact that we have this very off hands-off way of dealing with the training task which is that we are not supervising the task in a detailed way we are letting the model learn the language letting the model learn the grammar maybe even letting the model well let's say maybe for the grammar but certainly letting the model learns strong probabilities about sequences of words in a meaningful fashion so that gives you something that can be used or text classification for language generation for machine translation for any application that you can think of multimodal is an increasingly interesting ones of caption generation or story generation all those sorts of things can be done using a transformer and you don't have to train it from scratch anymore so when we talk about machine learning this is what we're talking about importantly this stuff was mostly started in English and with all NLP you have to discuss where is it in other languages and we are very fortunate that the work that's been done now is increasingly being done reproduced in other languages in a way that also represents progress there for example my favorite my favorite thing out there right now is one of the Transformers which I'll talk about a little bit later is called Burtt and the French does anyone know what the French model is called it's called camembert or camembert and the camembert model was I think Facebook in Paris so they've more time on their hands but and yes so we are getting to the point where we're also be able to deploy these in other languages lower we are rapidly progressing on the level of resource of languages as well so the high resource languages like English Chinese French German Spanish Italian are all being done very quickly Indic languages are rapidly following along it's very very promising to see that yep I actually don't know why they called it Transformers but the key thing about this transformer is that and what they did was instead of having a recurrent neural network where you have these this recurrent structure that keeps feeding backwards and forwards all you do is stack layers of attention on top of each other so you have multiple sets of layers which are all looking at the multiple cells below them and you're composing them each time and then passing them through a fee for network yeah I I guess so I don't know that that distinguishes it from any other model is you know perhaps Wow true and I mean it's certainly that that might be one way to think about it I am their level of commitment to this to the semantics of the naming I don't actually I wasn't convinced of again absolutely they you know I will whether or not oh not that's all that different from other multi-layered methodologies but you know I think convolution like word convolution is a more substantive often a more substantial crushing of that or transforming but yes in short there's a philosophical discussion there one interesting point that I do want to discuss about this though is that one of the things that transformers do that is intriguing is that transformers do something that humans apparently do as well which is that they read the whole sentence then they decide on the meaning so attention projects both ways in Beart right you look at the whole sentence at once and there's an interesting conclusion to that which is that when we talk about writing or when we talk about language it's a serialization of a simultaneous thought and we know this to be true because if you have puns or if you have any sentence that ends on the dangling word it's only when you hear the last word that it makes sense that's because we have a simultaneous thought about a language that we deserialize and you know it's an intriguing operational proof or an evidence of a way in which our minds work in terms of language sorry digression key other point about this and I just want to highlight this part here and the other key point here is multi-headed self attention and that means that in transformers we don't just look at the sequence of words once we look at the multiple times in multiple separate ways so it's almost like reading at different levels in doing that we get more compositionality and we get more ability to reflect differences in subtle differences in combinations of sequences if you want to train a model from scratch you can do so and our friends at PI torch have provided the Transformers module which is rather good for that and why would you do this you probably wouldn't do it if you were doing language but if you can think of any other large number a large set of sequences of repeated patterns of symbols that you can recognize that you might want to look for the context of then this is the place to be so code genetics those are the two that come to mind immediately you can encode them with a transformer so long as you have enough base data you need gigabytes hundreds of gigabytes of data but if you have that you can attempt to do something with a transformer here and those sorts of sequence models work as well so what does it look like in context I'm gonna talk about Burt mostly because while Burt is not the most up-to-date of the models so anymore it is probably the one that most people will end up using because it has the best support it has the best recognition and it provides most of the features that you could ever want so a Burt sentence looks a little bit like this I can install Revit 2019 and I would highlight for you two things one is that this is not the conventional way you would tokenize something so we don't just split words into individual words anymore now what they actually do is sub word tokenization so the model has learned a set of byte sequence per byte sequence encodings which are high frequency combinations of letters which it then divides words it hasn't seen before into this allows you to avoid an out of vocabulary problem that was a huge challenge with previous approaches to NLP I will I will get to that momentarily those are to do with how Bert looks at the model itself so Bert is able to look at this token these tokens in this yes oh yes okay yes so in the in the olden days of three or four years ago and what we would do was we would split pre tokenize the words based on something like if it's English we would just use spaces and punctuation if it's other languages we have to be more careful for example Chinese writing we have to actually build a tokenizer that's statistical of its own its own in its own regard English is a nice language it's fairly morphologically simple we don't do a lot to our words we just kind of leave them alone and rely on order to deal with them so always caveat other languages can be harder your mileage may vary not-not-not available where not applicable um but we have this problem if we hadn't seen a word before if that complete word did not exist in our database we didn't have a reference for it so we either had to infer it by trying to him like magic up some estimate of where it might be or we had to like initiate it completely neutrally so we didn't know what the word meant we just had to treated as an unknown token or we had to figure something else at what we've done here is instead of doing that what Bert does is it has a vocabulary of about 30,000 symbols including all the letters in the numbers so at worst it can break a word dent into very small individual pieces and then because the transformer is such a powerful model it can reconstruct those things so now you see that you are it recognized I can't install and 2019 all of those things came in as as single tokens it's seen those tokens before but revit it's split into two it's split into rev which is it seen before and a second sub word token for I T with the double hash meaning that it's a suffix because it's not an English word and it's one it hasn't seen before so there are some not what I would call non English words in the corpus because in the database because of the way the corpus is trained so here's the magic part what we do is we take this giant transformer model and we train it on hundreds of gigabytes of data over two specific objectives we do what's called multi objective training and the first thing that we do is we mask words so we say all attention is all you and then we give Bert a masked token and it has to guess what word should be there 80% of the time we just mask it with that masked token 10% of the time we mask it with a random word and 10% of the time we mask it with the original word the details of why you do that are in the paper and they're very clearly explained but it basically comes down to stopping the model from overfitting on the mask symbol second thing we do is we take that same model and then we train it again and this time you can see I have these CLS and SEP things what I'm interested in here is okay Bert you've learned about words now I want to teach you about sentences and about structures beyond individual words and the way I'm going to do that is I'm going to show you two sentences and you're gonna tell me if they're related to each other if this is the sentence that comes after the one before and I hopefully most of you have identified that this is probably not a coherent sequence so Bert should reject this 50% of the time I'm going to show you real pairs 50% of the time I'm going to show you fake pairs and when we come out of this we get a model we get a language model that can run from half a billion to as of yesterday 17 billion parameters so Microsoft just trained a 17 billion parameter transformer which doesn't fit on anything as far as I can see it's like very heavily and what we're interested in at this point is and the perplexity of the model how clear is the model on every word it sees how much does it it can it say I see Revit I know what that means I see angry I have a pretty good idea of what that means and I know with reasonable certainty what's going on so this multi-objective training the self supervised training as we call it I think one of the teams in university in Canada trained a 1.5 billion dollar model on an Amazon account and it cost them about $25,000 you do not have to do that there is our cost sensitive friend in the audience they are you and you get this for free so 90 percent of the performance of the NLP model that you're going to be using comes from the transformer and comes from this pre-built model that you get off the internet I will explain that momentarily um so you do not have to train this large model you download this from the Internet you can pick using the plugging phase transformers you can pick one of I think 11 architectures right now and you can even create Franken models where you take one piece of an architecture and connect it to another piece of an architecture so you have full range to do anything you like to abuse the data to as much as you wish once you have that downloaded then what you do is you can fine tune it so if you're like us if your Autodesk you have a large amount of terminology that's you specific you have a large amount of situations where for example our meaning of the word line is not what is used most cases because we're normally talking about drawings not like cues in a shop and in that situation you can one time find you in your model this is a kind of an overnight process most of the time it's very straightforward you're just doing the same thing as we did before multi-objective training we're just continuing the training with a different corpus once that's done to actually use the transformer what we do is we tell the transformer we want to do text classification or we want to do language generation or we want to do sentence similarity measurement or any of the major NLP objectives that we want and we feed it some data to train it this is very cheap to do this is a couple of hours of work to do and you get extremely high performance from this straight off the box I wouldn't do it on my computer cuz I have a Mac but we do it on stage maker and I will talk about how we do it on stage maker right now I think it's efficient most efficient to do it on something like sage maker because then I don't have to invest in in any sort of a Nvidia cards instead I can just spend that money on Amazon Prime [Laughter] so how does this broken process we decide on our base model am i speaking English do I want to use Burt do I care about case and then I take this code from the hugging face transformer is a glue example and I do a small amount of monkeying with it specifically we will talk about what you need to do in a moment and then we add our data to s3 we put our pre trained model on a cloud storage area we put our training data on there and we put any other information that we need and we change the data loader to those reference points and because of the way sage maker works this is all nice and transparent it's as though it were already a local file it's very easy to do and it means that you can put that there and our team has one place we can put it and everyone who wants to train their own Bert model can pull that data pull that pre train model very easily tens of millions of documents but you can that's only for the fine-tuning yeah depending on your tokens so there are some situations where you do want to do that for example actually this is a good tip that I forgot to put in a slide bert reserves a certain number of tokens in its in its tokenizer as unused tokens and they are given an index but they don't have any training associated with them if you have funky reason if you have reasons to add symbols to your text one example of what was done with this was to do style control and you can use those reserved keywords to do that in pre-trip and then you fine-tune on doesn't need to be a huge corpus but that will allow you to add a a token to control for example style generation so pro tip and so finding a transformer this is a Python code and mostly this is sage maker code so sage maker provides an interface what they call a PI torch estimator I'm not going to explain any of that because that would be stealing the limelight but the important point here is that we give it a type of an instance so this is a p38 x-large which is a GPU based instance we tell it that we just want to use a certain number of hyper parameters that I'll talk about in a second and we critical things you is because of the way pi torch is structured and because of the way a hugging face have built their transformers library there is a two line and requirements dot txt that installs all your your prerequisites and for me that is the best thing about this whole environment I no longer have to mess with kuala kuda i no longer have to do any sort of x configuration or figure out why my laptop is melted through the desk or any of those sorts of things i just go import transformers and i say and i use t GD m because i want to see a pretty-looking progress bar and then i just pass it a set of piper parameters so in this case we specify the baseline model that we are going to use in this case it's bird based on cased excuse me which is an english language bird model that is a relatively small model it's only got I think I forgot the numbers but it's relatively small model and it is uncased which means we've lowercased all the text that works in english it doesn't work in German but that doesn't matter because you're not using this model for German we tell it what labels we want these are just our internal way of labeling different types of support cases and then we tell it that we're using lower case because otherwise it will get very confused and then what I want to talk about next is these things here so the max sequence length and the Train batch size and remember that I said that we have this attention layer where we're looking at forwards and backwards across a sentence a set of these sub word tokens there is a problem here can anyone tell me what that problem might be hmm yes exactly I don't people don't talk in a very convenient 512 segment sequence and what happens is you either have to pad out the short ones or you you run over the sentence at the end now there is a paper called reform er which has extended that size they claim up to 50,000 tokens which allows the model to take an entire novella or a short book into it into it once still very experimental but very cool if working Bert allows you up to 512 but you can optimize if you don't think your sequences are going to be especially long if you don't think you're going to get a lot of essays then you can just use you can shorten that that speeds up the training it's in sub word tokens so it's a little bit unpredictable for that reason because you don't know how it's gonna split the words so you may have to do a little bit of engineering if that comes up but generally if you think you're getting towards that limit that 512 limit you will have to do a little bit of statistical analysis batch sizes are as you expect keep it small though because Bert is very memory hungry and the other Transformers are even worse learning rates as normal relatively small numbers of epochs between two and eight epochs tends to work pretty well you can go on forever if you were wish I'm sure someone here will put their hand up and say I did seven hundred epochs on Bert and got amazing results great we didn't need to and then it uses this learning rates as before so the nice thing is most of the training is the stuff you're familiar with it uses atom it uses the standard ways of organizing and my colleague who did a lot of the work on this code insisted on using seed 55 I always want to use 42 but I can't do everything and then all we have to do is load the model so we build that we load Bert's tokenizer because as I said it doesn't use the standard tokenization approach we attach this head once like we attach this head which is that we say we want to do sequence classification we want to do text classification in that way we want to return one of the labels that we showed you before as our as our output for the the transformer and you had a question Oh local rank - one is for doing distributed training I think yes don't worry about it now I mean I mean telling it not to worry about it don't do local ranks and that you can do fully distributed training on sage maker we're not doing it in this context but yes sweet so the better waved my whole talk is over you can just no I'm joking okay then we just load the pre-existing model we give it the labels and we pre train the model then we just customize it we download the vocabulary and go from there in training and this is one of the interesting things about transformers is there's a couple of different ways to start the training because transformers one of the tips that I will give you about them is that they could just fail they just sometimes get lost early and kind of crap out and what you can do is you can try a couple of different ways to use a higher or lower learning learning rate and to alter the learning rate psyche cyclically at the start so you use this linear scheduler in this case we're reproducing the configuration from the paper so we have fixed length but you can do all sorts of stuff around proportional or nonlinear optimization as you see fit and you can decide whether or not you want to have Adam correct your bias that nice thing about this is this later generation of code and uses the PI torch atom and doesn't use a bunch of other stuff that was very custom so do all of that you get yourself a state-of-the-art and text classification tool you can do all sorts of stuff with that you can do anything you want around text classification margin attach a different head and you can do sentence similarity and everything else so coming to the end apologies for the slightly long talk how do we do this with sage maker well there are a couple of things that we think make sage maker a good way to do this first we can share all our data on a big shared drive second we use notebooks a lot of the time the sage maker notebook is essentially a jupiter notebook which means that it's a lot more familiar for a lot of our people you can use the existing libraries you can pull out sort of stuff in it's easy to to manage that sort of stuff with NLP you did a lot of pre-processing on your data set so we want to keep those intermediary datasets somewhere that's easily available we do frequent backups because of this shared drive nature and because of the way that training works you just pay for the GPU instance as long as it's actually training so you don't have to worry about leaving it on overnight and bankrupting your company what sort of stuff do we share well we share all sorts of things and as I mentioned before very second last slide I mentioned all of these embedding things one of the things that we also do is we use a different architecture called and you LM fit to generate stable embeddings forwards this allows us to just say here's a piece of text give me back a representation that I can manipulate for similarity I can do cosine similarity I can put this into a tree I can do anything I like with it that allows me to have semantic similarity for Mike and I we use a model service for this where we just have an endpoint that you can fire any amount of text that can get back I'm embedding super super powerful because it allows you to rapidly accelerate the kind of analysis you might want to do in NLP because you're immediately into something differentiable when you use it so final slide we like notebooks sharing is much more about techniques it's about understanding it's about how you process the text as well as the code and the data and we absolutely love the PI torch community we actually also love the AWS people they've been extremely helpful to us in talking us off cliff edges when we try and do stupid things with transformers and that's it thank you very much just make some small links that are worth looking at one that first one is a way of visualizing attention if you want to learn how attention works second one is all the models that are available for Bert's start there if you want to try out transformers Burton more depth by a much more knowledgeable person is the next link and then the last one is called Trey and NLP has its imagenet moment and it explains why transfer learning is so important to the NLP community and how it changed everything my linkedin is here feel free to add me and hopefully not spam me but I'm happy to continue the conversation offline thank you very much great I know okay yeah so let me explain in a little bit more of a coherent way um and also show my slides so yes so transformers look at all the symbols simultaneously they do not they don't look in sequence they just look at all 512 up to 512 sequences at once they're just it's as though you were doing that speed reading thing where you're supposed to look back and look at the whole thing at once okay but what they understand is what we do is we project those attention relationships backwards and forwards so all the words we look at the interaction of all the tokens with all the other tokens in the sentence at once and bi-directional encoding is what Bert's sort of stands for and that's because it does that in both directions so that's the part of it like conceptually you can think of it as reading forwards and backwards but what it's actually just doing is projecting the influence of the of all the words simultaneously onto all the other ones yes so we encode position as well in Bert yeah it's another layer and I should have so I didn't want to get into too much detail about that but yes you encode - you encode position as well yes so actually this is a different architecture I kind of ran too long on the first half of the talk but same idea I give it a piece of text I get back a fixed length embedding that is kind of a as a math man Americal vector that represents the meaning of that sentence based on all of this stuff that we've learned yeah and it's a slightly different architecture and like the second system made sense but as I start to make more than one think this what story did that make sense okay so for the first part of the question we serve Bert models with ec2 instances with a flask over us you can also use sage maker endpoint hosting if you desire they work out to be kind of similar we use the ec2 for infrastructural reasons but essentially you just have a you can just have flasks wrapping it or you can write it in any fashion that you wish like that and then just write the the backend as you see fit so it's very straightforward to layout and there's tons of optimization you can do there quantization in natural language models is a big thing people have taken birth day into 8-bit and work it's worked pretty well so you can do a lot to kind of speed things up you can also do distillation where you train a smaller model from a bigger one where is machine learning going and we've had a big part we've had like in the last since 2013 we have had four major like generations of machine learning in natural language processing that have made huge strides forward and so the question is are we in a stable phase now where we're just going to keep building bigger models or are we going to learn something new along the way and we need to start thinking about how we connect code things like dependency things like more to do with the grammar all of those sorts of things into how the language works and there is interest in doing those sorts of things there was a paper released last week on doing sequence only parsing of sentences which is very intriguing because if neural supremacy appears in parsing we have an interesting situation where it can now do grammatical understanding and a little bit more of a meaningful way than before linguists in the room don't shoot me that's a kind of a very broad way of saying that but it's interesting nonetheless so we need to get beyond that second thing is things like the reformer like what do you get when you can put an entire novel rather than just a sentence or rather than just an arbitrary 512 character no am i came over who drew it's deep mine deep mine did the reformer yeah it's that stuff all super interesting and no no no Reformers very different like reformers structure differently the Microsoft one is making a bigger and bigger but even then it's an interesting achievement because you have to slice the model up in very specific ways and but we have probably reached the limits like 17 billion parameters is more than parameters than anyone really needs I think and there are initiatives in trying to do control generation so one of the open questions is humans aren't you can imagine that GPT 2 is a bit like someone who's been struck on the head and is babbling right it just kind of talks about the most probable sentence it seemed conditioned on what it seen before but it doesn't have any coherent mental model that it's reflecting so the question is how do we go from say a script or say a description of the environment to a coherence description of that environment so caption generation storyline generation all those tasks are leading in that direction about how do we do that sort of thing if we can get that done then we're in a great place you just give the recipe you know you just you know you can take this problem do you need one interrupts do you also look at the problems are still require range if there's music and grammars Chris Manning is much too near for me to say that he's obsolete and I know the answer to which tasks need linguists is all of them the question is at what point do you need to in to bring them in so I think that what's important about this sort of technology is that we're benefitting from an enormous amount of detailed work by the whole computational linguistics community on developing something that is a way of getting over the sort of stuff that we're not really problems that were there were problems of amount of data available and problems of how to train models stabili and problems of understanding that sort of stuff so you get further for free but you always need experts in the end and the moment you can afford to hire them you should be hiring them because they will fix things better and very often the other option is that you don't need this massive model and like these sorts of things there are very often situations where you will find that someone can tell you something about your your data and your language that you don't need to worry about this so this is a way of doing things if you've already decided you need to operationalize text classification is your answer but it is not the answer to the problem I have language and I need to fix it so computational linguistics is a and linguistics in general is an immensely important field and I should also mention other super important aspect this is underrepresented languages dialects very regional variations variations in script cultural variations bias all that sort of stuff is wholly uncovered in this talk which makes this talk radioactive but I didn't have time to talk about it I'm already way over time suffice to say like there's all sorts of language encoding that we're embedding here that's biased that is a bad thing and you need to think about all that [Applause] you