Unsupervised NLP Classification with Clu...

data.bythebay.io: Matt Seal, Unsupervised NLP Classification with Clustering

Recording: data.bythebay.io: Matt Seal, Unsupervised NLP Classification with Clustering

hi uh I'm Matt from openo as you just saw um and we're going to talk kind of into a realm of data science that I did early at opengov and we're renewing kind of actively at at the time um so I want to open a little bit with kind of you what is open gub if you missed the talk right before us so we are a software as a service company that provides uh financial data management visualization and collaboration for governments in the 21st century this is kind of a a something where governments have been behind um the kind of tech industry they're often times stuck kind of in the 80s or 70s in terms of technology so they they haven't had the help that a lot of other um uh companies and uh private sector has had in get collaboration to build up better and better tools so we're trying to solve that problem and we're also trying to solve the problem of connecting like governments themselves connecting governments within them within their own organization to other governments and to their citizens um currently we're uh just over a thousand governments depending on how you count them um in the US and a couple in Canada and we've been around for about four years I joined in the first couple months of the company and been here ever since So today we're going to kind of outline a challenge that I hit when when I joined opengov and that you know is still persistent and I've seen uh colleagues of mine also hit where you want to use traditional techniques you've learned you want to you know apply data science to get some value out of your your data from your customers but that there's sometimes that's very challenging in New Market spaces um in in some places there's no repository online for you to look up there's no relationships already established that you can you can use in your data and training data especially in Niche markets um can be really difficult because you need domain expertise to even generate it in the first place and uh it it becomes a really large challenge for uh supervised learning So today we're going to walk through a few of those difficulties and some processes we use internally um to to kind of get our foot in the door in that space so I talked about you know problems with these uh you know very specialized spaces um you know government records in particular are are a great example of this there is tons of government records online they're not organized nor are they well classified except at a very high level and that they really leads to if you want to do you know valuable insights out of data it's hard to to mine that out of out of the public records that are available and uh on top of that like there's very few knowledge bases to to uh leverage off of the existing ones like wordnet and dbpedia these all these other sources they're rather sparse in like our domain they have the high Lev terms and where they came from and that's about it um so the the other problem we have is as a as a young company our data grows very fast it almost consistently since like day one it doubles every six months it just it just keeps doubling every six months which is is a challenge in itself um yeah I already covered the knowledge basis limited training Daya is also sparse uh you I tried building some for ourselves it got outdated very quickly um and so that is a hard thing to um establish well especially as you change the type of things you're asking about your data um so as I talked about our data growth here you can see this is the curve of um our some one of our label tables that just keeps growing um this is all of our labels as they've gone up and you can see you know the training data that I put in in um 2013 didn't really cover very much of the space now and um this is going to continue to happen so we really have to kind of Garner some insights without having you know lots of training data um that doesn't mean that we won't use training data like semi- supervis is a great step from on supervised but we had to start somewhere so um you know when you're exploring textual data or unstructured documents uh you you don't know many things especially if it's a new realm you don't know kind of how many topics there are how many subject how many like precise classes um and you you can run into this problem a lot when you're trying to discover an antology like we do uh social trends and markets you don't know how to kind of bucket things to even you know ask the question of is this A or B you don't know a b or c or if there's z um there's a CP kind of a couple families of unsupervised techniques you know one of the traditional AI approaches to unsupervised is to sort of Define a utility function and uh let a program explore a space where you reward it for doing well and you try to Define what well is and you try and Define the space and you hope it tells you something useful um and you can learn like Optimal paths through that space you can learn about what's uh some end result that you can reach um but it it is kind of you have to have a lot of uh engineering effort a lot of knowledge put into that um algorithm to set up the space and in an easy to explore way um the other alternative to the kind of utility optimization is clustering so clustering relies not instead on having um you know very enriched data is that it relies on associating data in state spaces relative to you know other data that you found so if you just have lots and lots of data and it's growing exponentially it's perfect for clustering other than the fact that clustering goes a little slow as you get too much data but it does tell you uh answers it answers many of the questions about what's in this data what's the big pictures um as you get more data those pictures get clear so there's like a huge swath of family of of clustering algorithms um uh there's you know centroid based if you implement K means I think in school back in the day I implemented K means in like every single class it was at least eight times um and but you know there's some more interesting ones you run into like there's em um where you're trying to do uh you know probability distribution maximization minimization uh and then there's hierarchical is a glomer of clustering and these are just three classes there's there's many more um and they kind of arrive in sort of two major flavors they're sort of bottom up and top down uh clustering approaches the um top down approaches look at I have all the data in one cluster how do I break it up into smaller things until I get to a number of clusters I like the bottom up approach alternatively is really looking at pretend everything its own cluster and then try and group things that seem to mean to have relationships that matter into larger clusters so of that uh former group is hierarchical gomera of clustering which is a mouthful um in in this method uh it basically does a bottom up it groups things together and it scores each possible grouping as you group different clusters together until you have all possible groupings from nothing group to everything in one cluster um and there's a particular U the original author of that uh that algor one of the two authors in that algorthm came up with a greedy version of this method which actually run has really nice performance characteristics and it solves a lot of the problems that that are in sort of unsupervised techniques um at least for data exploration purposes um you get uh it's resistant to data missing data so if you don't have all your edges it's okay because uh the friend of your friend becomes your friend in a clustering algorithm so if you have chains of relationships as long as you can get there somehow you'll often times group together even if you don't directly relate um and you don't need no starting State you don't need how many clusters you don't need a starting centroid you can just run it on data and not have to parameter tune very heavily um and so what does this clustering kind of look like so here we have some labels where um I've kind of colored in um ranging from red to Green the you know how much this thing is like that thing according to some algorithm and in this case this would be common something where you've gathered a bunch of data and you said I've got sentences A through Z and I need need to know which ones relate to what and these are just some arrows where we say these are the relationships we put them in a graph if we don't know K means like we can kind of see it visually that there's three groups here but if K means you'd have to tell it or do you know sort of K optimization algorithm on top of clustering which gets complicated uh hierarchical clustering just will give you a nice group of clusters and unfortunately you can't see it on the screen because of the resolution but uh we have three groups one around the law enforcement one around sewage and water and then one around the street cleaning and Street shaping um and it will it will solve and find those and then um you give you this which is a dendogram um and what it says is it'll do is pretend you know the data you just saw was in this dendogram it would go and slice at what point is the the quality of each of those clusters highest and in this case it kind of sliced right across the graph at something around the 6 range of this of the y- axis scale and so then each of the color groups is a different cluster of terms or uh objects that you've put in this graph um this also lets you say you know it can tell you what the best choice is but you can override it too if you're really looking for three things and you know you have three things ask for what the three things are you can just slice that dendogram at a point where you have three clusters and so it has a lot of a lot of capabilities to sort of tune and parameterize especially if a quality score doesn't quite match your expectation you can tune it up or down some all right so today we're going to be doing some live coding and we're going to go uh dig into this site called uh gink which United States Common Sense recently um just launched this year and it has a huge collaboration of budget and audit um PDF from governments across the US so it's a really nice centralized source to to kind of find some information in in our domain that we're working in um I had done this we've done this effort in the past on our own and they have a they have a better repository than we do um but you know what's in these files we don't really know so um how about we have a machine learn but first we're going to go pop over and look at at what some of these look like so uh the gov rank site here you can search and and I can walk through so if I look for um uh San Francisco I can pop in there and I can find you know all the all the data that San Francisco's published each year assuming that our internet works well I can show you instead what I actually downloaded which works just as well oh there we go yeah it just took a second so here's San Francisco here's all the Audits and budgets um you know all the way up to 2016 you can download these kind of look at what the content is so I have some of those downloaded already um here's the one you know we saw in the if you were here with a prior talk we talked about ason Texas's data structure this is what their budget PDF looks like um it's it they range from like pretty pictures and nice graphs to like a giant blob of text that is a picture of an image that put into a PDF um this one's a little bit nicer than than many of them and so it has a lot of like there's some topics in here you can see there's some chapter heads and and they talk about kind of what the bget document is about this is a really big document it's 156 Pages this is kind of average size they build this every year um and it it has a lot of information in it um and so but you know to read through this if you want to read through a thousand of these and find information that's that's a pretty hefty task so let's not do that and let's have a machine try and tell us what might be in here all right uh so we're going to start we're going to do a little I python um shell here to kind of explore some of that data so yes what's itth GitHub is I did not post on GitHub but I did attach it to the talk so the code is attached to the talk along with data I used um the uh here we basically have I put a few utility functions here we've like read the data and then we we're going to use this progress thing so we can see how slow or fast something's going later um and here's a few sources I download about I think about 10 budget PDFs you know a size enough that you can get some intuition out of but not so much that uh we are going to be you know taking forever during a talk to to process it all right so kind of what what's in these texts right so um if we want to see what uh you know what's in let's say Sunny Veil you know we can take this grab the text we loaded from Sunny Veil um let's look at maybe the first five sentences or actually from here we're going to look at the first uh let's say 100 characters so you know we can see great there's some like City Sunnyvale director of City officials so on so forth maybe what's you know our last 100 characters this file is really big so last 100 characters actually takes a little while to get to and when you pulled the text from the PDFs you didn't do any cleaning Step at all just relied on the tool correct I use a PDF to text uh the PDF to text is and actually it's okay if you have a question in the middle list is really confusing it's fine it's fine to pop up with a quick question um I may ignore it if I'm out short on time uh yes this is just a PDF to text it's just a standard text extractor it does however not you know keep the the nice structure you seen in a PDF sometimes the text just ends up on new columns even though it's part of the same logical group or part of a different group on different columns so we'll talk about this later it's going to cause a little bit of an issue um but for now you know we have this uh data that we've uh collected up all right so first let's get some nltk uh if you haven't used nltk it's kind of the B backbone based Library you want to start with when you do things but it's not perfect uh um so here we're going to have a a few uh I'm going to grab my notes for this because the loaders for all the different types of things can be a bit uh dense so we need first we need to find sentences in this data we're going to we're going to extract sentences we're going to try and find a sentence which uh tells us something about the data so um let's grab nltk dat. load and we're going to load a tokenizer that comes with um with nltk called punct it actually doesn't do great on PDFs but it does better than the other ones um but that's okay because you know we we just kind of want to do some exploration and then we'll come back and we can say where we could do this better if we uh have more time so we've got this um we also want to tokenize our sentences so we're going to make this uh tokenizer here which we're just going to split on kind of uh new lines or actually any white space uh to get our word that we want out of this um great and then we have uh we don't want to you know we definitely want to prevent things like stop words from messing up our results which they definitely do if you um if you don't take care so stop words in this case if you're not familiar with them are uh words that are very common in the language and generally don't carry a lot of semantic meaning they mostly are structural to the sentence and in this case we're just looking at sets of words bag of words so we don't really care about the stop words um and so these are also techniques you can just kind of pull in these kind of come for free with nltk um you can make better lists if especially if you have domain knowledge you can add domain stop wordss which we use a lot and then we're going to uh make a little stemmer here uh because if we just look at the words themselves you'll find that they often times have random characters in them or are capitalized in weird schemes or um have you know adjective or verb forms that we don't care as much about as the kind of subject matter uh oh yes thank you I swear it's my native language yes so this is the problem with trying to talk and type at the same time uh thank you guys you you guys are my my proof readers my uh code reviewers as I go here uh the later parts are a little easier to type because they don't have this deep semantics so uh I'm just going to make a little helper here um and also in a second this thing will tell me if I messed up so that's because I can't spell great all right so we now have a bunch of tools from nltk we've imported them now let's actually use them so here all right so first thing we kind of want to do is let let's generate some sentences um so we're going to we're going to use this name Tuple which is a a form in Python to sort of keep track of list like objects but with a name so we're going to make this sentence name Tuple um and we're going to add uh you know a few attributes here so we can keep track of it we're going to do uh original so we're going to get the raw text in there we're going to pass in uh tokens so we're going to make some tokens with our stemmer to that we can compare with that are uh you know more sensible and then we're also going to list the source like where where did this sentence come from because otherwise it's sometimes hard to tell if you have a bunch of data from the same point or from different points um and then uh we're going to make a little tokenizer make it sound so easy uh well I went through this talk several times on my own so uh that made it a little bit easier uh to talk about it so what we're going to do is here for first let's we're going to start with some for Loops I'm kind of manipulate this as we go so we're going to say uh forward in our extract uh words so we're going to extract some words from some sentence object that we get um now we need a sentence to actually uh you know process to work on this so we're going to say um this is going to be uh well okay sorry I forgot yes we already have a sentence coming because it's an argument so some things want to do here like we can we have the sentence uh we can say just W uh which would be just the word by itself but that's not very uh helpful because you know as we said before uh there's lots of ways to represent this we could do lower um and that helps a little bit now now like capitalization won't mess with us but the reality is is that we probably want something more sophisticated and this is a good first step so we're going to stem this and I'll show what these look like in a second so the stem is going to take that um that word is going to like kind of compact it down to just a few characters that kind of are only the consonant values not exactly but effectively well we're going to we're going to do a little bit of filtering here because otherwise this still won't help because you get a lot of like token size one token size really really big so first thing we're going to do is um we don't want stop wordss right we said we didn't want those great now we don't have stop wordss uh what else don't we want we probably don't want um we probably don't want to pull in anything that um that doesn't really so when I print out these uh text files you're going to see there's a lot of random characters in the PDFs so we really want to do is we kind of want to skip any words that really aren't words um so we're going to do uh we're going to match the the text that we're actually getting in here and we're going to say Let's uh only grab things that kind of have uh you know sort of a through z type uh operators on it this means that we're only going to get text if we get something that has other things besides text um let's skip it um yes so and and ideally we might be losing some information there but for now we want to just explore the text we if we want to do something else we do it and then I'm going to do this little step because if I don't do this um the data won't look all that great as um experimentally discovered so this will cut out the stems of length one and zero which some of the words get split by individual characters uh can I make the screen bigger does that look better great all right so this uh now we have a tokenizer so we can tokenize a sentence uh so we actually want to let let's actually make those sentences so uh we're going to do four source and texts in the text that we pulled up above so up here we're going to have text. items um and in here we're going to let's grab the you know the original sentences so um so we're going to grab that text we're going to grab some original sentences and then we're going to get some uh some tokens so tokens we're just going to take that tokenizer function we made we're going to map it and we're going to use these o sents so this will convert each sentence in the tokens and then we're going to um make an actual sentence object uh and actually because we have have to map this data in we're going to um we're going to do a little bit more to form this I tried to make this a little bit cleaner but um bear with me we'll we'll just kind of get through the uh how to build this nicely and we're just going to zip these things together I'm using a lot of different python built-ins to kind of go quickly through this um but if you're curious any of these that the code will be online so you can quickly look through it all right so and let's uh print sentences you know maybe the first three I probably have a typo oh we need to append so oh and we should also probably uh further filter these sentences a little bit more all right and I still have enough minutes I think to get through this uh so we have this we're going to take the so Lambda we want filter that my uh Ruby development is rubbing off on me sorry now Ruby has its uses name tupal is not defined because I can't type this is the problem with live coding the worst case I'll just copy over the working example and we can move forward um yeah and actually for the sake of just speeding us up because we are running out of time uh I'm going to go cheat oh look at this magic code um yeah but we went through it um and it's Let's uh get that of which apparently I typoed in copying all right uh it's cuz I called extract sentences the wrong thing and tokenize sentence stop words is not defined okay so that should work thank you uh PP let let's let's look at this now uhy while it's running well I'm ending over one thing and modifying the other so it should be okay some these senten are pretty beefy so here we go here's our sentences um you can see here we have city of SAU Leo strategic resource um and down here we have our tokens so here's our different tokens that we're going to use to compare and this Source comes from budget sauco so this is our souro data we've broken into tokens we've kind of grouped it all up together and now let's actually kind of um you know do something useful also I parsed all of those PDFs so now we have this for every single sentence PDF this is about I think uh 40 or 50,000 sentences before we filtered all right so next thing we're going to need is jakar we're need a little uh function called jard similarity uh and for the sake of time I am going to just copy this over I'll explain what this is this is a common function for uh doing comparisons uh between two sets of things so we have sets of tokens inside of our um our uh extracted data here jard is a great way to start to kind of compare what those sets look like um and it basically it takes the um uh the intersection over the union so that you can kind of see you know how much overlap is there between these two sets and we're going to set this threshold number um the threshold number is just going to be something where we say we're going to look for everything has ja card score at or above that threshold um and that that this is important because uh you don't want a lot of noise in your algorithm or it's going to take a really long time a lot of these uh clustering algorithms and things like that really rely on um the data being sparse like you want all the the kind of useful sparse data that would come out but you don't care as much about uh things like uh things that have very low similarity and then generally the as I said self-repairing so even if you miss some good relationships they should still appear all right so we can do a really naive thing here uh we could do uh for index we could do for sentence one in sentences for S two in sentences and so on so forth this is going to be horrible um the end result of this is that it's going to we're going to take a very very long time because with n squ of 70,000 what's that it it doesn't matter CU I'm not going to run it um but yes so the uh sentence one sentence two so if you if went through and did this and we tried to do the jard so if we did right here um if we did jard of sent one. tokens and uh sent two. tokens this will do literally hundreds of millions of comparisons and that's just not going to be realistic so um what we're going to do here is we're going to insert a little snippet um which is very useful and I didn't want to talk about going through here but it's an estimator so on your spare time outside of this talk I don't have time to go through it but there's a thing called minhing minhing estimates the jard similarity between two sets um it's used a lot in document comparison to find a similar document this will give us a much lower number of things we need to go over uh and I encourage you to look up I put I posted where the Wikipedia page on it and the um data source that the library that implements this so we're going to have this little Library um we have jard defined we have this uh Min hashing and we're going to insert uh some Min hashes for our 26,000 filtered sentences great so now we have Min hashes for this and now we need to actually do um some Edge generation and unfortunately for time's sake I'm going to have to jump a little bit um quickly through this so we're going to mostly be copying over code and get to the data exploration which is more interesting so here we're going to do we're going to kind of iterate through sentence one sentence two we're going to do query we're going to do the index oneals two and then we're going to run the jard score we're going to get above the threshold we're going to add it to edges and then we're just going to run this and uh populate a graph with a bunch of edges so this runs pretty quick we're going to run you know 26,000 of these um and apologies for the quickness but uh we want to get to the interesting part all right and this is the clustering part so I'm going to use a library that I implemented a few years ago um at the time there was no good greedy agglomerative clustering all them there wasn't even an gomera of clustering algorithm that was good um since then there's been a few other sources I think s kit or scit learn um has uh has a a glomera of clustering algorithm now and there's a few other libraries that have them when I when I went back and looked but this one is one I implemented um and we use at opengov um it'll work it's kind of upper bound is 100 million edges at 100 million edges you start taking weeks to run instead of seconds or minutes this should take I think 13 seconds to finish is it is it built on network it is built on yes it takes in a network X graph yeah I'll explain a little bit so you're going to take in a network X graph and it will run the cluster and give you back an object that you can ask about the Clusters and you can specify I want K clusters or let it tell me um and it it works operates entirely on a networkx graph it builds another graph internally that has the whole dagram and then gives it back to you um and it's implemented I posted the paper as well on the talk about how that's implemented and what it does so uh we're going to go let's go get some uh I'm going to pop in these uh functions here and we're going to talk about them the so here we have this map plot lib we're going to go and plot some things first we want let's let's we want to say for a big cluster of things we want to know what's what's the central object in that cluster um so what's a sentence that represents that cluster so we go we're going to iterate through and find the one that is closest to all the other things in the cluster similarly we're going to take a score and score how similar are the things overall to each other so we're going to run these two things and we're going to get all the sentences clusters uh and we're going to go ahead and sort them by the similarity score we have here so we get the most the Clusters that have the most things are similar to one another at the top we're going to plot that and then we're going to go grab only clusters who have um who have you know between greater than five elements and clusters who have um who also have multiple sources so we're going to plot this and we're going to see this graph which is nice so the top is this sort of there's 25 good clusters there between the size of 5 and 50 that have uh you know a pretty decent threshold for similarity this means that uh if I wanted to print out one of these so if I um so if I print out interesting cluster you know one or zero print so here you can see like here's an interesting cluster it has budgeting operating Revenue totals um from previous year and you'll see in here there's a bunch of other things referring to uh budget operating expenditures and other things so this is what a cluster kind of looks like it's got like K five to 50 sentences in it and um so now we we kind of we can kind of use this to judge do we have clusters that are meaningful if you have something between the kind of. 3 to8 range those are things that could have value in them if you're kind of outside that range you're often times going to end up in the the boat where um you don't really have something uh useful uh because either it's just the same term said over and over and over again or it's actually um it's actually a uh you know a bunch of noise that got clustered because there's no really good clusters um you can see we'll do a little histogram here so this is going to print out you can see the the size of our clusters so they go all the way up to about 100 and then there's 10 like good multisource clusters that are fit within our filtered parameters so we have 10 topics we can we've now pulled out of this data and here's the histogram of the sizes so you can see it um and so now let's let's say we have these interesting clusters so let's go ahead and uh you know print out a few um so let's print out maybe the fourth one great so here it's going to talk about you'll see uh if we grab out the centroid we can do that as well great so we have accounts for use of BAL of accounts for use of fund balance and balancing purposes if you actually read through these um you'll see a lot of them talk about balancing and funds they're all about like accounts used to keep funds in balance and ensure that the right amount of money is in a fund this is a really important topic in budget PDFs um it tells you you know something about how the governments think about their budgets and we just pulled this out by totally unsupervised Technique we populated out and popped out hey cluster 4 talks about this subject that we didn't even know existed before but a domain expert would agree that oh yes this is a very important thing that budgets are trying to tell you um and we have lots of other clusters in here that do the same thing um so we'll have like the sources of money so intergovernmental and revenue sources um how they F thing so what is the source of money that's funding thing another thing a budget trying to say how is public money being allocated and used um and there's a bunch more of these and if you download the example you can kind of explore the data results we have here and see some of these we do have some noise in here you'll see like right there's these like characters here like this sentence here look doesn't look very good and it looks like it's actually from multiple lines that kind of had the same terms grouped together this is because our sentence parsing and our tokenizer wasn't very effective and if we had gone back to the top and done a better sentence parser and kind of split those things up in a in a more informed manner we would get much better results and we would get more useful clusters another thing we also did we're just using the words in the sentences that's actually a pretty weak heris it's really neat we can pull out topics just by words the reality is if you want to really mind this and get something that you could get some real classes out of and get with more data is you want to go back up to the top and kind of extract out word senses you could do a better stemmer you could go and look for what's the like topics that you found in other domains with those words in them and pull them in all as features of your set that you're going to compare then you run the exact same algorithm you have to change nothing in your algorithm but how you define the inputs of the things that you're jarding and all of a sudden you're going to get much much better results and uh we use this internally to int a lot of information about our labeled data and to extract out and identify classes of our data um from like any input from anywhere um and great so I wanted to stop there and I got through the code sorry I had to paced a little bit more of it than I intended um but uh we looks like we have two minutes for questions thank you Matthew thank you very much um we do have questions but also people want to eat so are you available for question afterwards I think it was really really cool what you did there thank you oh I'll be around so I'll be here all day and I'll I'll try to stick around by this room for a little bit if people want to come up and ask questions sure thank you very much if somebody has a question right now a short question please no okay um no yes I I do text summarization how would you give a summary for your talk hashtag for not data scientists pleas um why would I summarize this talk it' be hash machine Discovery hasht machine Discovery I can work with that um I'm in your audience asking uncomfortable questions the system would break for non-english languages um it would but you could very easily either translate to uh stems that are in English or you could even and actually this system with stemmer you could load a stemmer for a different language and run it on that um Additionally you can also do there's a lot of great sources for translating online you can do rough translations which actually work really well in this instance so if you get a French document you can actually like Canadian documents you can take it and you can translate it and put it through a translator and you get a rough estimate but you're already taking a rough estimate of the sentences anyway so you only lose a little bit of quality what about Slavic languages what about what Slavic languages Slavic languages uh I haven't got to dug into that but I do know there are options for um like babet can has a bunch of great other multilingual parsing capabilities so you can pull out sinets that relate to English or other more established languages from uh languages that have less time focused on them rule based what's that rule based yes rule based languages that is something we are completely discounting in this process we can like if you have rule-based languages right we're doing bags of words so um the order of the words in here we threw out we also threw out like any semantic meaning of the stop words in the middle um so we lost some information there we could recover that and get more results if we wanted to with more Advanced Techniques noun identification for identification noun oh sorry noun identification noun yes that's what uh actually if you look at Wet is a great resource for that though it hasn't been updated in a while and I really wish somebody else would adopt that and kind of bring it up to uh the new update the Corpus but that's a great resource we use it internally for uh determining whether a word has a noun sense a verb sense an adjective sense all those type things and you can use other tools like the Stanford NLP parser which we've used as well to try and ract out which one's the most likely candidate for a particular word in a sentence do you want to expand it to other languages uh yes uh I think and actually this will be a concern for us is we get more customers with other language data we already have some French data but it's also paired with English language description so we're okay there I want to sell you my Corpus I want to sell you my parallelized Corpus 22 language 1 billion words that' be wonderful buying so yes as we get in in the more countries right now we're mostly focused in the US but I mean the rate we're going in a couple years that won't we anymore so yeah thank you thank you very [Music] much