data.bythebay.io: Trung Diep • Ronald Sujithan, Text Analytics Simplified
Recording: data.bythebay.io: Trung Diep • Ronald Sujithan, Text Analytics Simplified
great good morning uh thank you for attending this session um for this audience we don't have to convince you that uh informational text is can be found everywhere across the web uh on public domain you can find news article blog post discussion B of all sort uh also in private communication there are text messages email messages internal reports and such so the nice thing about working with text is that it can be readily found and used uh the challenge for text analytics is to do more than just NLP syntactic paring and to extract semantic content from it and that's what we're going to to talk about today okay so in this talk we will demonstrate uh in two ways the text analytics the first is that we want to show you how to take any free form text uh whether it's long short of any kind and extract uh meaningful semantic content to produce structured data so that you can then use for further analysis uh for data mining or machine application machine learning applications the second part of this the second thing about this text analytics is that we want to do it in a way where machines can call these services and the results uh formatted in Json can be interpreted by machines really there's no human interaction in the whole pipeline of this and Ronald will actually show you um a text analytics pipeline that we built from scratch and we'll show you the entire pipeline step bystep taking starting from URLs that you can extract for news for example and taking it all the way to produce cemented content and then visualize the results okay so the text analytics API that we are using is from the data ninja family of services there are actually four set of services the one I'm talking about today is focused mainly on one the smart content which basically extracts uh any form text to produce some sort of structured data and I'll go through what that structured data is uh these services are rest for API and they can be accessed on the cloud from anywhere um the API is relatively simple to use but underneath all this is a pretty impressive knowledge base that we keep it up to date on a daily basis so new Concepts and new categories are being added uh on the on a daily basis and we want to do it uh in a scalable way through the uh Power of uh cloud computing and big data so that we can do this with uh with ease so the structured data that we are returning includes the foll and when Ronald goes through this uh in the Json format you'll see that there's actually a lot more I'm just giving you sort of the simplified view of them uh the four main types that we returning are Concepts categories entities and keywords uh for the first two it basically basically characterize some of the ideas or generalizations uh from the text uh the last two are more specifically to certain entity that we can extract keyword and so forth and we also return sentiment as well and you'll see an example of this I think the best way to show uh text analytics is to give you an example so what I have done is taken an excerpt of a news article and the news article is quite long I just took like a paragraph from it and you'll see that it talks about the zika virus uh outbreak uh and you notice that none of the text mention the zika virus explicitly but what we can then do is to uh call the data nure services we can then extract that if the concepts include things like the zika virus out outbreak uh the microsof uh zika fever and we actually return many many more I'm just showing here the the top three we also return the categories from them uh and the categories can be at different level from something very specific to something very general they specific in flavy virus all the way to diseases and disorders and then health is much much more General so depending on your use case you you might be interesting in something more specific or something more General entities are different from Concept in that concept can be discovered even though it's not mentioned explicitly for entity it has to be mentioned explicitly in the text but it doesn't have to be spelled in the same way or Capital capitalize in the same way so here we can identify that Brazil and the World Health Organization are two entities and then sometime related to that are people find keywords also useful there's some overlap between entities and keywords I am not showing the overlap pieces but it can also return those entities in the keywords and sometime keywords might be more interesting because it can detect things like mosquito born virus as something that is of interest to you or birth defect so all of these things can be run through the smart content Service uh using the data ninja API and you can check it out at data ninja. net okay so data ninja API is more of a service what we want to do is to take that service and build something more like an app where you start from from beginning to end uh we use this uh use uh show this use case to kind of give illustrate uh how you can use these uh text analytic services so we start with news aggregator uh and we use four and produce text from them extract the text and then fee each of these article text through the data nro services to produce the structured data that I was mentioning and then from there we use uh machine learning to detect the CL topic clust doing and identification uh and then we have actually a visualization through an app we call this app the newsbot ninja and you can check that out there at newsbot uh datan ninja. net and I'll have all of the URL at the end as well okay so Ronald you can do the thank you TR so let me get my jup notebook up here okay so um what we want to do in this demo is to follow up uh in terms of what trunk described the data data services the different apis and uh the output the rich semantic content that we can extract from text and go ahead and build a text analytics pipeline from scratch uh and we going to do this live so the idea would be to build something similar to The newsbot Ninja service that trun mentioned uh whereby we're actually going to go to Google News um the the current uh Google news page and um fetch the page extract the URLs do some cleaning to to find actual uh uh news URLs uh URLs that are pointed in news articles then we're going to show how you would go ahead and integrate the data ninja Serv service into a text analytics pipeline um we will run that uh for some uh small number of URLs out of what we a could uh collect here and then we will do some interesting machine learning with it we we will try and run through some clustering with it the whole thing is going to be live with actual data that's currently available here so let's uh get started with that um so the the first thing we want to do is obviously do some pre-processing right um so in our case um most of you I'm assuming are python developers are you're familiar with python so we're going to use the the Python's beautiful soup Library which uh uh does an excellent job of fetching uh a URL and then uh allows you to declaratively find uh interesting uh markups in that uh HTML so in our case we just going to pull all the links um out of that uh HTML page once we have that uh we will go ahead and uh collect the URLs we do want to filter out some noise uh so we don't want for example uh links that are pointing back to YouTube videos or Wikipedia Pages or advertisements within URL so we want to clean up some of those things but once we do this looks like we've got uh links that are more or less pointing to um currently trending news articles uh so after some filtering we get usually around 200 or so links let's see what we get here yes so we got that many link um once we have this the next step in the pipeline is to go ahead and integrate uh the data Niner Service as trunk mentioned uh we are using masap which is a API hosting service for hosting our API we've recently started to use the Amazon API Gateway as well so you have a choice um the Amazon API Gateway might be easier because you might already have a developer account and you can simply use that uh to sign up for our services once you uh sign up you will get a key uh there is a free tier that you can start using uh and then you'll just simply input your key uh pass it as part of the header and all of our services follow fairly standard uh R protocol so uh the one that I'm going to demo here is our smart content service um the smart content service the payload is simply the URL there are additional parameters that you can pass and you can check that from the documentation but essentially you will pass in a link and the data ninja service will do all the heavy lifting at the back end as FR mentioned we built our own um Knowledge Graph and we update it on a daily basis so we use that to extract the categories and Concepts and keywords and uh and sentiments and so on so we can take a look at what it looks like for a single URL before we go ahead and uh collect the information for um uh the the hard URL that we've collected um I I'll come back to this I just want to go ahead and U start this process very quickly and we'll come back to that okay so um you saw on uh trunks PowerPoint Slide the different things that we return uh when you call the data ninja service especially the smart content service and here's the actual raw output in Json format right so uh so this particular article is about Warriors um Playoffs schedule um and uh what we have done is to to go ahead and uh first of all extract the text uh from that web page and let me show that first so here's the actual uh article text that we've extracted from that page so one of the internal services that we we call in the process of generating your output is to take the URL and apply our own machine learning techniques to eliminate all the the boilerplate content from the web page and just extract only the text that we think is really the representative content from that web page so you can see right so we've extracted including the actual schedule that's uh more or less in in for in the format that appears on the web page that information that's then gets fed into the actual uh uh engine that does all the the extraction of the the categories so you see there's some number of categories that uh identify or tag what's the content on that page uh we have um some number of um keywords I'm just going through so keywords have uh sentiments um then you see there's a lot of content here um then we have uh Concepts so these are the concepts that we've identified from that page along with the extracted text so we give you all this information as a Json response and so you can very easily use this information to do uh all kinds of interesting work with it right uh so one one quick example that um we wanted to demo here is that um remember earlier on we we went to the Google News web page and collected all the URLs and and uh We've cleaned it up and then we also did some further cleaning here we want to have pages that have at least 100 characters in them because you know we want to do some clustering with this data so so that reduces the number somewhat but you know we'll work with this this number of Articles so so what we want to do now is to sort of try out doing some clustering so LDA is a popular clustering algorithm that people use so we will use the the version that comes with the Gen Sim so the first thing we will need for that is to uh prepare our uh environment so the first thing we will do is to create our stop word uh list so that we can eliminate them uh from uh the clustering step and then uh here's a helper method for us to to be able to do that so first we will go ahead and tokenize the text remove stop words and then convert it into a vectorized form that LDA requires and then you'll actually call the the the routine to do the clustering and then we will do some post processing to produce output in a way that we can display on the screen so we'll go ahead and uh run that and once we've done that we can actually call the routine to to do the clustering so we'll run that too it takes a few seconds for us to finish that this is all live this is the current page as you would look at Google news right now nothing is canned about this particular demo so so let's see what we have here um so uh this is an very interesting cluster as you guys probably know uh this is a trending news about uh one of the girls who who was kidnapped uh from Nigeria being released so that looks like a really good cluster usually uh uh there's a cluster about Microsoft making announcement about their mobile operating system um obviously you get some trusters that related to the presidential elections um so you can see um I as you can notice I I've trusted for uh 20 we can maybe reduce that number increase that number so you can play with that but the idea is that um you know in a space of about 10 15 minutes we've actually built something started to end uh and letting data ninja do the heavy lifting along the way and um even though you know this demo is about using uh web pages and and news articles which are easy to get at but you might have legal documents or design documents or Healthcare documents any of those documents can be easily used with their and IND the services to produce the semantic content that you need and then do other kinds of uh machine learning and um uh post processing that you can do with that uh text information now we have uh obviously built a much more complex uh pipeline uh as trun mentioned uh which will in soon the newsbot ninja Service uh that uses some of these ideas but we leverage the rich content that there and ninja provides to uh build out uh our uh service so with that I'll pass it back to trun to um okay so basically we uh talked about text analytics and as I said mentioned before text can be found anywhere and and it's hard to use them directly so we want to provide a way to convert them into some kind of structured data that you can do further analysis on and data ninja Services allow you to do that but if you want to do much more than that uh with a you particular use case we illustrate one particular one called newsbot ninja you can check it out uh at newsbot ninja. API datan ninja. net uh you can get the key to these API relatively uh uh quickly so I recommend you urge you to try it out uh and yeah what is news newsbot is what we just show with that that that use case yes it's in beta so you can try it out and see uh the kind of trending news uh cluster and particular topics uh track different sentiment different entities different categories from from that application okay I think that's it thank you yes so you talk about Concepts right you do concept extraction but what if you were a bus certain set of Concepts that are quite unique to use is there a flexibility for us to for that business to create its own models so that you can run it on that idea so that you get better results yes so I mean this is true okay yes the uh question actually you get a free T-shirt with with that as well uh to encourage more question um the question is whether you can get some kind of customization let's say if you're for a particular domain and you're interested in certain vocabulary and you want to supply that and and the answer is yes we do it in two different ways one is that you can provide us with a custom dictionary of things that you're interested in or even things that are not even public domain uh or or particular kind of vocabulary that has uh another meaning or so forth so we actually also provide API that you can do this programmatically uh and update us on a daily basis if needed and we'll start extracting using those vocabulary uh for you so that's more the ENT extraction side there's also the concepts and categor side that we can also provide and return specific categories because the categories and concept that we have is you know millions of Concepts and category and for some people that's too many and they want only you know 500 categories that they are interested in we can also customize that for you as well yes what data set do you use for Concepts and uh categories what do you how do you decide there must be like a yes so I mean the knowledge base is uh is is the main thing and we try to scrape as many publicly as well as uh use some proprietary um data set to help us get as big a a knowledge basis yeah and this is done on a continual basis yeah I can't go into detail on which specific one but you we can talk afterward okay ml algorithms you are using here there are so many ml algorithms but concept extraction entity extraction what what algorithm you use what algorithm do we use um well I can't go into too much detail about that but I think the the main thing for us is to see what I would recommend you is to try out the we have a demo that you can try out I saw okay okay that is but but I'm interested more in the technical side actually how that what algorithm is best suited for this kind of right and so I think there's a a bigger conversation to be there we can maybe take it offline talk about that yeah maybe I can just quickly interject so you know when you build a complex pipeline like this really you don't use a single algorithm so pretty much anything that you find in a natural language process machine learning text we use them somewhere right so we neur models so we use many many different algorithms um graph algorithms um based on where that algorithm particularly does it its job for us so we have a fairly complex uh pipelines that do a lot of the the back end heavy lifting uh that runs on a regular basis to update our knowledge graph so most of the results that you see is driven by the knowledge graph and so the update process runs on a frequent basis so Knowledge Graph is a is a graph database or what is the what is the data store for Knowledge Graph uh so we we've largely developed our proprietary uh storage mechanism so um so that is a serving part so that serve from a in know regular database but in terms of what we do at the back end obviously we use our own Big Data Technologies by had and SP and whatnot where it makes sense to process through the volumes that we need to De with yes there was a question over here any other questions I just want to also quickly mention uh The Notebook is actually hosted on GitHub so feel free to go ahead and download that modified it's totally self-contained everything runs uh the only thing uh that you need to get is a key Either from Amazon or m shape if you can go and get a go ahead and get the free developer key then you can play with it you can modify it and you can try out how the system fits for your specific needs specific use cases right and we also have a booth right now yes a manual uh so so this example is just to show how standard L uh we actually don't use the standard we developed our own clustering algorithm and so we do something great and also I just want to mention that we have a booth upstair so check it out if you have time thank [Applause] you