Scale By The Bay 2020: David Talby, State of the art natural language understanding at scale
[Music] hello everyone my goal today is to help you with you next nlp project uh by describing really what's new it's kind of right now where we are at the end of 2020 with the spark nlp everything i'm going to show you today is completely open source and free apache license so you're most welcome to use it for both a personal and and commercial projects so we we're going to uh cover three areas here we're going to introduce bulk in lp but really it's not just a basic introduction we will talk about where we are in terms of adoption and use and functionality of the library at this point in time and then we're going to talk about two interesting things one is a recent advances in accuracy uh so when we we say we provide and want to provide state-of-the-art accuracy it needs to mean something concrete and it needs to mean that state-of-the-art actually means that we reach the best accuracy it was ever published kind of academic benchmarks accepted peer-reviewed journals so we talked about some of the new innovations in that space and then we'll also talk about speed and scalability yeah because also now the library is three years old and what it means to be you know fast and support the newest platforms and use innovations also changed and we'll talk about some of the new things that are have been added to the library and available to you right now if you have any questions please ask him on discord and nikolas will do the filtering and i'll do my best to answer them as we go if not i'll answer questions at the end so a sparkling piece is an open source library that has apis in python java and scala that have the exact same functionality across those languages and the goal of the library is to provide the open source community with state of doubt accuracy speed and scalability for natural language processing tasks um the library is three years old when it started two interesting things happened that that kind of you know will enable this adoption one is kind of deep learning happened to nlp which means that a lot of the all the libraries that we all you know know and love and use it became significantly outdated in terms of accuracy and sparking people was among the first to it really go all the way in terms of making the the newest advances in deep learning and transfer learning available in a production grade scalable and playable package uh the second thing that works well for sparking the pst is based on apache spark as the name suggests which means that i mean it is still the only open source nlp library this is natively scalable and can effectively use a cluster and also in some important use cases and aspects it's also by far the fastest library to get those tasks done in terms of adoption in 2020 so 2020 has been a fairly big year for the growth in the adoption of spark nlp as you can see in at the end of january we will just at just under 10 000 downloads a week right now we with about twenty two twenty uh sorry ninety to ninety three thousand downloads a week yeah so you know about a million downloads every three months or so yeah so you can see it has been significant uptake in use of the library and in general what we are seeing is that um practitioners enterprises people are actually putting energy in production which is happening a lot this year they are generally choosing sparking would be to do it so about almost two years ago the beginning of 2019 is parkinson's here was going to you know but we had the first big survey the o'reilly a adoption in the enterprise survey that suggested sparking b is the most widely used nlp library in the enterprise by practitioners and with a number of other other surveys and results in 2019 that supported it in 2020 the latest survey we have is kind of the big nlp industry survey by a gradient flow that came out two months ago now in september and it's a great report by the way and if you're interested in natural language processing consider just going to gradient flow and dot com and downloading it one of the questions was really which library do you use in inspiring peace the most widely used once again by practitioners in specifically within the healthcare vertical sparking appears but it was used by 54 percent of the all the respondents in that area uh and and in general what we are seeing is that you know when people are doing you know doing kind of academic exercise or research they often go with you know a hugging phase or spacey or allen nlp when people want to take things to production usually sparking lps is the library of choice another thing that happened is there was a lot that was added to the library in terms of functionality and one of the things we keep doing we we just keep releasing software and spamming is a very active project so we've had 26 releases in 2018 with 30 releases in 2019 and we've already had 26 releases in 2020 it's really software at least every two weeks in quite a few of those releases it's significant functionality yeah so so this year for example we've reimplemented name entity recognition it reached to to reach the new state-of-the-art accuracy we've re-implemented document classification sentiment analysis it's not the only library that has a multi-label and multi-class document classification we've added quite a few transformers there are about 100 transformers that come built in with the library right now we've gone from 1 to 46 languages that are supported to some extent we've implemented deep learning based sentence detection and spell checking and you'll see some of those examples um so the library is is the code base so for example if you use it in python then you know you you do pip install and you get the code of the algorithms themselves it's also a an ecosystem of models and right now really the you know between pre-trained models and pre-10 pipelines are over 300 of them that come out of the box uh four different languages a nice but nice variety of nlp tasks so we will see some examples coming coming forward another thing that has grown is a which libraries which i'm so which environments are supported with spark nlp uh so java python and scala are fully supported in terms of api a dc was also a fairly robust r library that was contributed by the community that enables you both to use all the existing models but also train your own models in r in terms of what we officially support we officially test on databricks cloudera aws azure kubernetes uh there's quite a few production use cases on on cloudera but also you know on google cloud and what's on on installing spark from scratch and of course just on local machines so at this point you can very safely assume that whatever your development architecture production architecture is there are quite a few people who already use it in production so if you need the dockerized if you need it on windows whatever your situation is you know it's been tested it's been kind of the bugs have been cleared and it probably works we also have optimized bills both for nvidia for the latest gpus as well as for the latest intel chips uh so that we we actually make use of those new compute platforms right in the deep learning optimized how to achieve so that's that's over librarian that's what's available right now yeah see i'm sorry computer froze see what happens next here i apologize for that so nicolas are you understand on the call yeah i'm still here as your presentation software uh frozen completely like them they may completely froze now okay um so i need to see if yeah what i can do if not i may need to to you know disconnect and come back you know it looks like oh now it's coming back good so i'm not sure what happened here but let's continue where we were good and that's uh yeah so apologies everyone that's new let's see what happens next okay accuracy wise let's see some of the things that have happened recently so in state of doubt accuracy means uh do we achieve the same or better accuracy as the the best academically published peer-reviewed results on summer benchmarks and that's really our goal and in general what we do is usually we don't do new research but we are really good at productizing research so whenever someone claims to have improved the state of the art right so there's a new paper saying oh you know when i try this type of parameter this neural network architecture these embeddings i get better results we try to reproduce it some of the things they produce and generalize some of them don't if they do we take it on ourselves to you know to give the community the the production great trainable scalable version and here's one example if you need to do name entity recognition uh then sparkle p basically will give you the the best accuracy in in any production system today and it's an open source model uh so the underlying architecture is the bi-directional stm efo tokens cnn for characters plus a crf using a you know very large embeddings of course you mean healthcare you want to use biobelt or other domain specific embeddings together so it's it's trainable but for example if you just you know use it on the coin ll 2003 set here the results you get you get you know 94 on uh on the test set and you get like almost 96 on the on the training set uh here's how you use it the other thing we've done a lot of work with just making the api easier to use so first of all here's what name entity recognition is it means that you you know given a text like chandra monica met in central perk you want to identify certain types of entities so for example people in location right but also if you're in healthcare maybe what you care about these drugs and diagnosis i mean illegal maybe you care about you know corporation names but let's look at this example people and locations so you want the algorithm that will say oh the string chandler is a person and by the way it's not the city of chandler arizona right so you know you should not you know basically making a building is off a dictionary is not enough um a central park also it's not a real place right so it will not appear in any dictionary uh but but what you want the model to learn is that when you say someone and someone met in after met in english the only thing that could come at either a place or a time in central park is not a time right it's much more likely to be a place because it's um you know um it's a it's it's it's a noun right it's capitalized so it's obviously a location and here's how you do this in spark nlp we already have pre-trained pipelines they do state-of-the-art name entity recognition with birth for the pre-10 model in english and we actually have this in i think 13 other languages so do pipeline equals pre-trained pipeline recognize entities birth in english this will load that pre-retrained pipeline this would actually go and load load the pipeline itself either from s3 or if you have it cached on the local disk right or if you you know you're just offline for compliance reasons you load it locally it would load any deep learning models that are required it would load any embeddings that are required like you know built embeddings to the initialize you know tensorflow under the hood it would load everything and have the model the pre-plane model ready the next line is result equals pipeline dot annotate harry potter is a great movie so you have the pipeline now we say okay apply it on this text and what you get the result object is a regular python dictionary and if you look at the ner column of the dictionary you'll see that for each token it tells you um you know whether that's that token is part of any entity so in this case it will tell you that the first two tokens so harry potter are part of a part of a person okay as you would expect okay so this is really all you need to do you do the imports and these three lines of code another very very common use case is document classification a document specification basically any case where you take a piece of text like you know an email a legal document or maybe just like just a tweet or a sentence and you want to classify it in in some way for one example is sentiment analysis so is this a you know tweet positive or negative you maybe sometimes you need multi-class classification for example emotion detection so it's not just this is positive and negative i want to classify it between you know joy surprise fear you know happiness sadness all right there are more than two classes and you can do this and there are pre-trained models in spark nlp for a toxic comment cyber bullying fake news detection sarcasm detection spam detection and a few others uh in this example the example you see in the table we want not only multi-class we want also multi-label text classification okay a multi-label means that more than one label maybe may be correct for text okay so for example if you look at the and this is for toxic comments so so it can tell you whether a text that you provide is an insult obscenity a toxic comment a threat sexually explicit or an identity attack and of course because of you know human creativity one sentence could actually apply you know have multiple labels that apply you can be you know both threatening and insulting you know why not and and you know we try to in this slide here we have some clearly tame examples but when we trade that are examples that are you know much more vulnerable than that yeah so here uh yeah what spagnelp does first of all it implements a state-of-the-art deep learning model based on uh sentencing buildings uh that you can train it to get accurate without at least recent papers of trendy to get the most accurate results on certain on several benchmarks and then you also have a pre-trained embeddings and then some pre-trained models that come with the library all right so if you just want to come say okay i just want this i want the you know a toxic comments detection that just comes completely pre-trained it's it's a few lines of code if you want to train your own uh you know it's still a fairly simple thing to do and you'll see some examples soon but then you can still reuse for example embeddings right pretending meetings or transformers that come with the library uh here's how you use sentiment analysis uh so it's very similar to the previous example we've seen here we also have the import so we input spark nlp we call spark nlp.start so that would actually start a spark session in the background but one of the nice things that we we worked a lot on is the fact that really if you don't want spark you don't need to know anything about it right so so you know most of the use of the library is people working on the laptops on single machines right no it does not have to be clusters yeah so we we hide spark as much as we can if you will if you just if you just don't need it a phone sparking that be pre-trained import the prevent pipeline we load the pre-trained pipeline analyst sentiment ml in english and then we just annotate the text harry potter is a great movie and the result is a regular python dictionary and you can see that the sentiment is positive in this case okay so that's really you know if you need to actually unless sentiment that's really all you need to know um another thing that's really big in the past couple of years is embeddings and transformers for nlp because they really enable you to train new models with much less data than you used to need before because basically if you look at something like vert it it's a it's a it's a very effective form of transfer learning and things like birth or you know the other types of popular embeddings like you know albert roberta excellent elmo they basically embody a lot of knowledge of the english language uh so for example you know if you want to do you know if you want to train the imdb a movie review database and know whether you view as positive or negative you know what without those kind of embeddings you know you you you needed maybe 20 25 000 reviews to reach say you know 0.9 accuracy now you can do the same with 5000 to even 2500 reviews okay so you need far less data you know to reach the same level of accuracy which is very very effective when you're dealing with domain specific documents for example clinical documents we're getting more training data labeling data is is very hard and expensive so there are quite a few as you can see the quite a few word embeddings and sentence menus that come with spoken lp you do not need any external embedding on transformer library okay this is just part of the spoken lp ecosystem it comes with the library one of the nice things that this means means that when you load models like when you load pre-trained models embeddings are part of that spark nlp includes a built-in in-memory database so we take care of how you cache those embeddings how you load them into memory how you share them between different cores or threads on the same machine how we distribute them across the cluster and so all of that kind of magically happens in the back end in the in the background for you it's the same you know five lines of code and we take care of doing that and that's very significant because some of those transformers like your build large can easily be you know two three gigabytes in memory yeah so if you have eight calls you cannot just load it eight times right you just get an out of memory exception and there's a lot of engineering work we've done to make sure we can efficiently and quickly load them and use them another another model uh that we have now that comes out of the box and it's a deep learning model is language detection yeah you can see here some of the the resulting accuracy on the right yeah but it's a really nice model it has it's between 97 and 99 percent accuracy a full text at least you know 140 characters long for shorter effects sometimes i mean i mean just really really hard right so you have the word like neo chao which just exists in both you know italian and romanian if that's the only text you have there's no way to tell which language was intended um but you have enough characters you can almost certainly tell what the language of the text is another nice thing about this model is that it's it's very small so it's under five megabytes and that's important right because the difference between a five megabyte model and 500 megabyte model not just how much memory it takes it's also how long it takes to actually load it right so can you can you start yourself start you know start your modeling you know three seconds in terms of instead of 30 seconds which is also impactful uh another a model that we've rebuilt this year and a lot of what we do by the way we we go and we rebuild models so the nice thing is the fun thing about saying that you do state-of-the-art is that you're always behind right so by the time you you implement release the model someone published a new paper and say hey i'm you know the state of just improved right because now we can reach better accuracy so a lot of what we do is kind of we re-implement new versions of basically the same tasks so one one popular really popular example and an nlp task is a spell tracking and spell correction and what we've done this year is we've released and this is also this completely open source a new uh spell checking spell collection model that is deep learning based and implies most importantly it applies context so if you look at the examples here you have three sentences i will call my sister due to bed whether we have to move to a different site or we travel to three sites in the summer so in this case if you have the spelling mistakes and in all three sentences that the word that's actually there is s-i-t-e-r the algorithm would based on context would correct them to system site or sites in the three sentences based on the context even though the exact same token and as you can see it also deals with things like singular versus plural you can deal with some conjugations some people's personal names as well for popular names and the end result is if you can look at the first bullet here on the the main benchmark we found is the whole book and benchmark uh this this algorithm this the britain model for english delivers the world over rate of just over eight percent and we compared it with you know jump spell which also it's another open source a python library which also you know claims to do deep learning context based spell checking uh but here still we were able to reach a much higher level of accuracy with the way we've trained it okay so that's another example but also the set of other deep learning models even things that you think were easy things like you know sentence splitting so how you split uh sentence text into sentences is something that we have a deep learning model for so we can deal with things like you know lack of punctuation right ocr issues uh you know um the unexpected new lines right and algorithm will be as you expect because it understands much more of the language so that's around accuracy and you can see really what we do is it's not a kind of specific algorithm or a specific approach it's really just following following the research and doing the best we can there other thing to consider our speed uh so in terms of speed uh one thing that's really important over the past few years is how do optimization so can how effectively effectively we use gpus or the interoptime cpus and we have specialized builds for both here's one benchmark that it's gonna be in public and certified on aws we trained the name entity recognition in the french language and basically we ran it on both intel and nvidia optimized builds uh for the same same number of echos basically this was to reach the same level of accuracy in this specific case what we saw that the cascade laid optimized version was 19 faster 46 percent cheaper than the tesla p100 and there are quite a few cases where actually the new intel deep learning chip so you do better than the nvidia ones yeah but of course we keep optimizing for both in in different use cases we give you different results so that's one aspect of performance another important aspect of performance and speed is optimizing embeddings especially the large embeddings like the belt ones which are slow but there are some things you can do at the software level to actually optimize the speed and just couple of months ago in sparkling p 2.6 we improved a memory consumption of all build based models yeah so it's between 1500 of them by 30 we input performance by more than 70 basically dynamically deciding what we load and what we call when we apply birth so there's been a number of papers a number of benchmarks on that all you need to know is that if you use spartan lp you just get the benefits of that it's completely it's completely transparent for you and happens behind the scene another way to increase speed when using buildings is simply to use smaller embeddings right and distilling large embeddings has been a very popular topic in the past year yes as you can see within spark nlp the reason you have so many even just build-based models because you have tiny built mini build small built medium belt and then if you want you also have large build and and basically what what was recently found is that you know if you're steinberg you have a model that instead of being two gigabytes it's just 50 megabytes so it's far far faster to load right far faster to use in this case you see bear time is 24 times smaller than 28 times faster than build base and very often in some use cases for example things like sentiment analysis and motion analysis the difference is accuracy in accuracy maybe just one or two points right so so you may prefer to do this because really the the reduction in accuracy is going to be so small your user may not even feel it and so so we give you these options right and you can decide on your own trade-off between a performance and accuracy terms of scalability suspending nlp has been and still is the only open source library that's natively distributed it can run on any spell cluster so it doesn't matter if you're doing you know if you're building spark you know from source or yourself refuse data break so if you use it on cloudera if you use it on aws emr or wherever it is your cluster is it will work it will scale it does not need any admin permissions and such and it's a bit natively on top of spark so the spark nlp pipeline the pipeline class that we use is the spark ml pipeline class okay so it's really um we designed it together with databricks to make sure we make the most of what spark has to offer um this is the the one point will really depend on spark because you know um distributed and parallel computing is is really hard and what spark does really well is is dealing with those those aspects of how you minimize a shuffling and you know bandwidth bandwidth how we optimize caching how we really do execution planning of the whole pipeline in advance how we optimal serialization and we've done really a lot of work with data breaks and with the spark open source ecosystem to make sure we and this is really just plain just a lot of effort in engineering and profiling to make sure this works as well as it can of course you see the list of caveats uh distributed computing is not magic i mean if you have a cluster with 5000 machines you need to know a lot about how you configure spark and spark cluster and make the most of them and and of course this matter but the nice thing is really within the spark ecosystem there are now tons of tools and tons of knowledge on how to do that well probably more than any other a parallel computing network uh here's one recent example we've done that shows you kind of the differences and what you can expect in terms of the speed up you'll see on the cluster so here we can a training on the full set of amazon reviews so 15 million sentences 255 million tokens a single machine versus 10 nodes yeah not a lot of memory you see you know 32 gigabytes 32 calls per node and what you can see is that in things that are trivially parallelizable tokenization uh you can see you're actually getting super linear speed up right so instead of you know 34 am 340 000 words per second which is already impressive you get 7 million watts per second however we do entity recognition that requires embeddings transformers you know multi-deep learning networks uh the speed up there is a significant speed up from 1 to 10 nodes but it's less than 10x okay and this is really just nature of the actual work you're doing right so some things are very trivial to parallelize some things require by nature of the algorithms of mortality so uh this is really what i wanted to cover and the really the the the main points i want to say in the the kind of in the one or two minutes we have left is uh what you can do next so spagnelp has lots of live demos so streamlit apps and call up notebooks there are more than 80 of them so if you google spark np and just go to nlp.johnsonlabs.com click the demo tab there are more than 80 of those the streamless applications so you can see you know how you run name recognition how you detect emotions in tweets how you detect cyber bullying how you identify fake news and you can see text and you can see how the examples actually look the nice thing is for each live demo for each nice ui where you can try your things there's also a call up notebook so if you come and say okay this is nice but now i want to you know i want to try it with my own text in my own you know my own python you can open the co-op notebook just run it in your own gmail account and then you can run the notebook you can try things you can try your own text you can hack it you can do whatever you want with it right and it's you know i mean just runs yeah so it's a super easy way to start with the library and the secondary live examples there's another set of examples here so you can see detecting similar sentences right to semantic similarities toxic comments uh we have for sentiment analysis we also have aspect based sentiment analysis so the pre-trained model and examples that come with the library set and splitting date normalization and really the main thing i want to say be that 250 pre-trained models we really did not cover the majority of them and there are quite a few interesting language tasks that the library can do one more set of examples that are there live on the site is support for different languages and so you can see first of all there's the language detection piece but then we do have a full pipelines so a tokenization limitization out of speech namely recognition in quite a few languages and if you're interested in that really you know just just take a look see if it fits you if it seems nice look at the call-up notebook and you have the code you can run so to get started you can just google spark nlp or if you really want to and nlp does not don't johnson labs.com is there is the main website uh it has get started pages it says the full reference documentation and really important you can join the select channel so there are more than 2 000 people on the slack channel now the dev team is there as well daily so if you have questions you can say oh no i'm using this code i'm getting this exception and you just want to copy paste it somewhere and get it get a response it's a really good place to do that yeah so that's a really good way to get community support it's been fairly active with people helping each other daily on the demo page of that same website you have the live demos in the column notebooks which are a great place to start learning there's also a whole set of articles and videos and if you want to contribute we're always looking for more contributors here we need more languages more models uh as you said we do a version every two weeks so there's always a whole lot of more things we want to add yeah but also if you just want to you know update documentation add in your example at a new blog post uh please here's the gita page let's get up issues and pull requests like everyone else hey or you're really almost welcome to just you know contact me or others from the team but i'm always interested to know what people are doing with the library and if you need help if you want to contribute if you just want to ask a question you know we will always we always get interested so that's what i wanted to share please use it it's built for you it's built for the community it's say it's open source and completely free and the goal is just to enable people you know easier and more democratic access to state of doubt natural language processing yeah so with that um thank you for listening and i wish you the best of success with your [Music] projects you