Scale By The Bay 2021 : David Talby, Building Complete State of the art Natural Language Processing
so uh johnson fda is actually not named after the king of the north it's named after a real person dr john snow uh who's known as the father of the public health of epidemiology he was a clinician a physician in victoria london and the thing he's most famous for is helping stop a call it outbreak in london in 1854 and the interesting thing was how he did it he actually together with others he collected data and he built the map of all the people who died it's called the ghost map because everyone on it was a ghost and with it found out that uh the the most probable cause of the epidemic was one water well that was probably poisoned and the well was blocked and the epidemic kind of a calm down and that was a really one of the first examples of actually using data to improve public good which is really what the study of public health is all about and we joined snow labs our goal is to enable many many more john snow's to exist and succeed in the 21st century and help them with their own data issues so hence the company having said that you know we you know it it came across nicely that you know that uh the king of the north used the same name and then you know it detailed the beginning and then i think people still recognize the name for it so um today what i'd like to talk to you about in the next half hour is uh my goal is to help you build your own natural language processing projects in in kind of real production systems and do it all with free software so i'm going to tell you about some of the new open source and free software and that we built and i'll give them to you for free and and let's begin so we're going to have a quick introduction and then talk about three things we're going to talk about the nau library for python the annotation lab which is a labeling and data annotation tool and then the nlp server if you have any questions uh please save them to the end so what i like to do is have this do this presentation afterwards we have the separate room and i'll be there for any questions you may have so natural language processing is a growing into really a fundamental enterprise technology and something that is very very widely deployed and we have uh other than kind of mechanical evidence we have several industry wide data points to show that one of them is for me the nlp industry server 2021 that was published just just a few weeks ago by greg and flo uh that showed that in 2021 compared to 2020 60 percent of technical leaders stated that the nlp budget was growing and a third of them said that it was going by 30 or more and really what we are seeing is in general ai and cyber security is basically the only areas where people are still investing in during the covet pandemic and so it's growth in 2020 more growth in 2021 and here you can see very very significant investments which really means that in industry in enterprises the new research advances we are seeing in deep learning transfer learning are actually now transforming into real projects within real companies that are aimed to go to production another thing that caught me by surprise was a another report from six months ago that focused on healthcare ai and us technically specifically within healthcare what are the the kind of the the key technologies that they use to kind of to manage data and there were four technologies that more than half of technically the name which was data integration nlp business intelligence and data warehouse and the interesting thing is that other than nlp the three other technologies here have been on this list for three decades and they'll probably be on it for another three decades nlp is a newcomer right so six seven years ago you know there was no way that many people you know it was a niche thing that would not have been mentioned but really this is just another a data point that shows just how prevalent the technology is being used right now in in production enterprise settings so uh one of the things that uh you know i and the team at johnsonville think about every day is how can we best help how can we at the open source community the industry take advantage of this these new technologies new opportunities really be able to succeed and succeed faster and there are many different types of answers but there's one answer that pretty much everyone agrees on that's going to help everyone er which is giving you so giving all of you the community the industry more free software because that's usually the easiest way to get people to you know just take something run with it build cool things uh and and you know help all of us make progress uh and for us our focus has always been starting with enterprise grade security scalability robustness and so what i'm going to show you here are tools that are definitely kind of proven in that respect all of the the tools and libraries that we'll talk about today i'll kind of expand the spark nlp ecosystem sparkly by itself it has been growing fantastically over the last year and a half it recently cost 10 million downloads of this just on pipeline alone not counting you know maybe in the entire javascale and spark ecosystems uh as of a month ago it has a 31 percent of uh nlp practitioners not just in healthcare in general use parking lp and uh suggest pipelines over 24 times it goes since january 2020 uh so those bentley being a very significant boom in 2020 and 2021 you know the library is adopted use and deploy which is something weird it's you know we are very thankful for and we definitely take the responsibility we've just celebrated four years of releasing software every two weeks and our promise to to you and the whole community is that we will keep you the state of the art so we keep looking at everything new that comes out of research the things that actually generalize actually um and actually reproduce our commitment is to give you the production grade scalable and trailable version of them so uh let's begin the first library that i want to show you is the nlp library for python uh which you get by just signing pip install nlp right so it really is just the nlu library of python our goal with it was a to to provide all of the power behind spark nlp uh but in a much simpler way right so people really who don't care about you know tensorflow or vital sleep learning don't care about you know smuggle skating or anything it just want like super simple ways to use what's already there so one design goal was to get to a point where everything almost everything you can do with one line of code and you'll see some examples of that another big design requirement i would say was to integrate very nicely with the rest of the python ecosystem so as you can see you can just send an entire you know pandas data frame you you get pandas data from out uh you can use you a regular python screens that will erase dictionaries in a very intuitive way it's very easy to integrate with you know the common python uh data processing libraries visualization libraries send data around is to really make this camera super easy to use buy some pythons good good python citizen and of course completely open source so the new library is apache 2.0 license so it's open source and it's free to use including for commercial purposes with no limitations so um the new library has starts with two very simple methods that you need to know which are load and predict and in this this example shows you how in one line of code you can do spell checking and spell correction so any you don't code a spell spell is the what you call actually the spell or the kind of the keyword uh to load the english spell tracker pipeline so that would fetch it from the web if if it's not all of the cache locally if it's cache locally it would load into memory it would load embeddings you know whatever it needs set up the pipelining memory if you have you know gpus multi-threading it would deal with all of that so that the model is ready to go and then a dot predict i like peanut butter and jelly means that it will return a panda's data frame and that looks like this so you'd have a vote for each token and you can see the words a like peanut butter and jelly all four words they will be spelled and the checked column would give you in this case the corrected version for both so that's all you need to do to do kind of autocorrecting python nowadays you've installed nlu and this one line and you're good to go all the magic happens behind the scene so that's one model uh there are more than four thousand more that are available today to you uh for a really different kind of nlp task from simple things like sentence splitting to more advanced things like this you'll see translation question answering so let's look at a few more examples this is sentiment analysis so i know you dot load sentiment dot predict i hate this guy sammy here you don't get the whole per token you get a open sentence it will advertise to guess what's the the most kind of most relevant uh pandas data frame to return if you don't like it you can ask specifically for a different level of granularity but here you can see that the sentiment is negative there's a confidence interval that comes with it and you also get other things that are computed like this is like the tokens and this library is multilingual all of the spark nlp models are available to you here is we have access to a models in more than 200 languages as well as some multilingual models and embeddings and transformers that are part of the the library and another thing that you see here is you don't have to send the stream to predict so any dot load length so length is the the default recommended language predictor detector dot predict in this case we send the python array with two strings in general what you can said you can send a string or you can set an array you can also send just a pandas data frame you can also send the spark data frame and then you'll get back spark data frame so if it is it does need the scale distributed it will it will use that you can also send the array data frame or moding data frame yeah sumi can kind of correctly optimize and handle things like you know multi-level environment or other kind of more kind of you know scalable pandas implementations in a very transparent and very native way right you just use the library you want sending you to predict you get the same result and under the hood he does the kind of he does the correct optimization does not do conversions and here you can see we do a language detection we have one sentence in english another in french you get the language and you get the confidence interval for each um sometimes you don't want a you know you don't do the library to do a test we just want you to calculate lots of embeddings here right just you know you just i just want either word embeddings or sentencing bendings or whatever may be the case so you can do that and you can do a whole bunch of them in general you can do a whole bunch of things in in one line so you do any little dot load and then you do birth space elmo space alley albert space excellence versus usb universal sentence eventing space glove and then don't predict tomorrow do is it would load all of these embeddings into memory it would calculate all of them so in this case just you know word embeddings anything except the sentence eventually you just get one one vector for the sentence and return if you return all of that to you in one in one pandas data frame so other than the fact that this may take a lot of memory simply because those models can be big uh this is really this kind of by far the simplest way to go and use and calculate those embeddings either on one one string on a large body of text another a very popular uh use case a kind of ndp task and set of models here in its name entity recognition is looking within text and recognizing specific types of entities and so the and any other club any other club the ner uh kind of the default in english um recognizable here which is based on the autonomous though i think 17 entities it can recognize out of the box but really i think there's several dozen pre-trained nell models that come in and that's in english only in this case you can see we the sentence and the extract entities like angela markel is a person donald trump is a person germany is a location and american is a miscellaneous because it's it's an adjective in this case uh if you don't have a pretend ner model another thing you can do is unsupervised keyword extraction right which basically in this case we use the egg and there's also now a newer egg-like algorithm that is more accurate yeah but to keep things simple basically what this does is you know take a sentence or of course can be a a whole set of sentences and what you're asking it is kind of you know in an unsupervised way give me kind of really that the main keywords are the main phrases from the text so in this case it would read this sentence on the screen and tell you that any leo nlp and python library are the three key concepts or phrases that it was able to extract together with the original sentence and the confidence score so that's another one liner you also have a other name and recognition keyword extraction and you also have a fairly loud set of text classifiers from you know sarcasm classifiers spam classifier and so language classifier and there's also several multi-class classifiers for example one very popular one is for identifying toxic content and it is multi-class because there's more than one way that i said this can be toxic if you look at the example here you're so stupid as you can see it returns two classes and this is a toxic sentence and an insult these specific classifiers can also specifically look at sexism look at identity attacks and the the two or three other kind of classes that you can identify together with confidence uh so so that's only one pre-trained model um another thing this example shows you is how you can use mods in different languages yeah so you can see you do new.load en dot classified.txt en means english if you put pt it would be portuguese if you put ru it would be russian and if you put xx you it means that you want the multi-lingual mode what we don't show here is there's also the ability also in my nanotech to train to train your own models yeah so there's also a fit method it does that i'm just not showing the examples here for a full time but you can use it online um another interesting thing you can do with an elio is summarized text and also just it uses the underlying kind of spark nlp models so nlu.load summarize you have the data which is called predict on the data so really it's the same thing we just split through the data into several lines because it's longer that's why we want to summarize it and you get the pandas data frame with the text and the predicted summary there is also an ability to configure how you want it so basically how aggressively you want to summarize uh so those properties are also the document of the website here we have uh i think some will be 50 and 100 different python notebooks show examples as well as the reference implementation one other interesting thing to show is a spark nlp and hence nlu also come with some some interesting prepaid language models and some of them include their own knowledge basis right which is how they do things like closed book question answering yeah so for example in your dot load the end of t5 predict who is the president of nigeria and you get how do you get the correct response what is the most spoken language in india you get the correct response right even though they actually the question is not grammatically correct you still get the correct response and same one with other answers yes we have quite a few hundreds of translation models embeddings and transformers name anti-recognizer text classifiers um different types of summarization q and a models basically spell checker gamma checking all of those tasks out there and other than just actually doing that any lieu does includes some some other interesting things one thing is visualizations so for example if instead of a nlu.predict you do a new.load.this for visual visualize uh for name entity recognition tasks and models you get this kind of visualization you see on the screen right you get donald trump from america and argentina from germany and you see it actually visualizing you see it in your notebook of course you can use it in a web application if that's what you need there are quite a few other types of visualizations this example shows a part of speech tagging and dependency parser so any loot load that dot type so this is typed dependency parsing and visualize billy went to the mall and if you get this visualization right on your notebooks on your screen that shows that you know billy is a is a non-phrase right-wing is a verb d is a determinant and so on and he also sees the types and kind of the links between basically between the past in addition to those visualizations which are really super great if you want to do a you know just different type of debugging or just see how well the model works nlu also comes with a nine different complete streamlit ups yes so if you come and say okay this this any other thing is nice but can i just have a small web app where i can actually type things and see what it does you know same thing for text classification for dependency pulser or as you can see in the example here it for in a clustering and doing kind of three 3d interactive exploration of embeddings and there are applications here to to do that for world debating for sentencing meetings which is great for kind of whether for debugging or just understanding how different texts work there is a streamlined app that does that it takes exactly one line once you've installed the library to actually run it run this library uh you know with some text but then you have this interactive web app and you can do it you can do it yourself uh here are uh you know here's the current cheat sheet and same thing it's a one-liner uh for example you know if you want one for a sentiment analysis any dot load sentiment but these underscore stream lead classes on the data that you have and that's it just it's slow it closes into active application with the data with the models it's run and you can you know play with it change it see how it goes yeah so overall what you have here is a completely free open source super easy way to use thousands of models out of the box that are production grades scalable and heavily used in industry if you want to get started with this does it's a one-line installation and this is vocolab and you have other kind of other examples in case if that's what you need there's full documentation there's the link here uh and there's also community support um so if you go to sparknlp.select.com it does you know my team is there every day those are hundreds of other people that are there on a regular basis and you can ask anything from general questions to you know copy and pasting you you know your exception stack trace and usually someone will be there and be able to help you so that's the nlu library er the next thing uh that we found useful and founded something that's that's kind of that's missing in the industry is is a it's a really strong enterprise gate good annotation tool and we're happy to say that johnson labs annotation lab is now free by free we mean we mean free so it's unlimited everything yeah you get the software you install it yourself uh unlimited users documents projects tasks models unlimited everything no upsell it's also a share nothing model yeah so as you know a lot of the work hey transform does is in the healthcare space most of our customers are in high compliance a regulated environment uh they cannot you know it does not work for them if you have a sas model they have to send anything to us and in general if someone says you it tells you that you know they give you such a tool for free but you have to share with them you know all of your annotations all through documents and that is not free at all so here this is a share nothing model you you get the software you install it it does not call home you can work in an isolated environment and so it's a completely free tool uh other than that it's it's a capable tool and we we've been using it for two years in production now in dozens of projects it's very stable and robust in that respect there are other annotation tools and really i mean i the reason we did this i talked to maybe 20 teams in the industry over the past you know last couple of years to build the orientation tools either because the other open source tools were not strong enough because the companies behind them want to upsell you to tools that have security have project management have internal data agreements or better analytics or active active learning and we wanted to give a free tool that that gives you all of that out of the box um or because um um you know the tool but they were really as a service and then either the you know pricing became as expensive as you scaled because it was your per user per document um or you actually had to share kind of documents and data which was problematic in some industries so the goal was to provide something that is really free that is really enterprise grade and and definitely something that's robust and kind of being used and you can rely so and here's the annotation tool i'm going to show some screenshots and kind of highlight some of the interesting features uh so the the very first thing you need from from a data from a data annotation tool is really just productivity you want the annotator who spends like hours and hours a day to be you know one-handed keyboard one-handed mouse in the zone keyboard shortcuts one click edits making sure whatever i'm doing uh you know marking entities relationships assertions normalized values sequence through sequence classification i can do it all very easy within one screen um you want versioning in audit so i do something as myself when i annotate i submit basically commit a version if someone else does something they have their own version there's a compare version tools it does a full audit trail here so which is very important if you're working for example in healthcare or pharma there's version control you can see who did what and they also shared guidelines yeah so if you have specific annotation we say okay oh when we look at someone's hate a height for example it says 186 you know cm is the the cm part of the should it be annotated or is it is it separate what happens if you have a different unit what happens we have two spaces and you want to be very very specific about those guidelines within a team so you can share and edit guidelines other than that there's a whole lot of features around a project management and teamwork so first of all you can define different projects people can belong to different projects then they can only see what they're supposed to see within the projects and within the roles within a project each task has a state diagram yeah so you know it starts it can be pre-annotated there's someone you can assign and change the people who are assigned to do it once someone submits a job you can either say okay it's done or you can say no it needs to go to a reviewer a reviewer can send back um work if there are arrows and to correct it with comments there's also in addition to versioning there's also a commenting feature and of course we manage the audit trail of all of that state diagram um you can do tagging right so for example you you can have tags on your what's in your training set what's in validation set you know who does what and there's also the research and features here to remove duplicates the tool supports different kinds of projects so not only textual ones also you can annotate images so bounding boxes image classification so on you can annotate audio you can annotate video you can audit html you can combine some of them together into a project we need to annotate multiple things at the same time uh so all of that is there the other thing a project everything is kind of user based and role based so when you define a project as you can see here you can assign people to the project and to specific roles so you can say you're an annotator you're a viewer you're only a viewer and someone else only views or who can kind of view dashboards uh so that really high compliance projects can can happen um the big thing and really the the huge huge kind of you know 10x productivity uh boost with the annotation tools comes with the fact that you don't actually upload documents and start annotating one by one but you start with some pre-trained models you pre-annotate data okay so that you start with a prediction of what it should be the human annotator annotates by correcting that prediction so only the things that are wrong or been er need to be fixed and that you know should happen quickly in terms of the ui and then in the background new models get trained and basically you have an active learning group and without any data scientist in the loop you you can you basically improve the model as you go so you have the ability to pre-annotate so you can load all the models and entities that already recognized and right now we support a document classification entity recognition relation extraction and assertion status detection to do that and you can pick which fields would be annotated by what you can filter which task should be annotated right only tasks in these stages would be annotated you can choose model choose embeddings and the whole thing runs in a separate container so this runs on a on kubernetes uh so when you want to pre-annotation it could be any a heavy job it runs in a separate container so it does not interrupt other people it will interactively annotating in the user interface once you have some annotations either pre-annotations or you're done with your cell phone combination uh the tool supports automated model training and active learning uh so um you can either click train now and say okay let's let's go and train or you can just you know move active learning so that it's completely automated as soon as there are enough enough new examples a new model gets trained in the back end you can watch real-time logs uh the training also happens in an asset bonus container right so just kind of it gets spawned it gets into a queue and the model returns in the background without interrupting anyone and if you push the little settings buttons you can see that you can also if you want to if you don't want auto nlp you can choose your own embeddings choose your own hyper parameters and filter really which completions it should be it should be useful for me david one minute warning sure thank you so we'll yeah we'll complete this section yeah yeah and then the pizza will have to come and ask and wait so the um next thing you want is you want all those models to go somewhere so there's a local models hub which connects all the models that you downloaded from the internet from the online model have all the models you've trained here and models you that you just uploaded manually uh so especially if you're in an offline you know hyper security environment you have kind of you can manage all the mods that are available for peer annotation for model training here and that's really part of the tool and the last thing is uh the more there's about those more than 20 different guy for analytics so you can track your project right so you know we need to get 5000 documents done how close are we to being done you can write productivity which people are doing better more productive more accurate than others and you could buy us in the dataset right so as you annotate you can say oh look we have way too many women where too many kids way too many people with diabetes here which you can only see when you annotate right because that's why you're annotating you can identify that bias really early on right and fix it by adding or remove removing certain documents as part of the process yeah so we have that as well as inter-annual agreement which is critical so either when two people disagree or when a person in a model disagrees and you can you can see this you can see this visually and when you really want the two people to talk and see okay where are we you know what what do we need to agree on you can drill down to individual examples and text get to agreement and update the shelf guidelines so that this happens so if you'd like to get started the annotation lab same thing a oneland installation there's online documentation including some videos on how to get started and community support uh so we'll stop at this point uh we'll not talk about the nlp server given the time uh what i would say though is look we build those tools for you for the community and we see really already great uptick of the annotation layer by the new tools that's there please put them to good use and please send any feedback send us feedback i mean we are looking to learn as well so really so we can do a better job for you and know what to build for you next thank you