sfspark.org: Sandy Ryza, Semantic Indexing of Four Million Documents with Spark
Recording: sfspark.org: Sandy Ryza, Semantic Indexing of Four Million Documents with Spark
[Music] how's it going I'm sandy thanks everybody for coming who here it's written the spark job before cool think about half who wrote that spark job and Scala versus Python but I don't know how you would raise your hand in the sponsor that Scala raise their hand Python raise your hand okay also but a quarter and a quarter sorry I know it's for a weird setup here might take me a second to get oriented so thanks everybody for coming I'm basically gonna give you a talk that is a use case walkthrough it's the goal here is to give you a sort of basic understanding what it looks like to build out a simple NLP application with spark before we start out quickly about me I'm a data scientist at Cloudera I lead clutter spock development a little while ago but I've gotten out of that game a little bit and I wrote oops go go I know it's gone sorry I have a two different little things going on over here all right I wrote some of this book which has some of the content that we're going to be talking about tonight so we're gonna be talking about a technique in natural language processing called natural latent semantic analysis and latent semantic analysis is basically a fancy name for applying a matrix decomposition to some text data we're going to be applying this technique on a data set that I imagine some of you are familiar with its Wikipedia it's the full Wikipedia set of English documents this is about five million documents and to analyze five million documents is about I think 46 gigs but it's not not huge but you know perhaps you want to look at all the languages you need some sort of system that allows you to process that amount of data do complex stuff leverage tons of CPU and cores and to a rescue here we have this system called Apache spark which basically allows you to do this and do it in a very very simple way so everyone here as soon as heard of spark who's heard of LS a litter latent semantic analysis okay SVD oh well it's like a lot maybe you can correct me when I say things that are false about it so what does it look like to carry out this analysis practically we have some sort of raw data we want to parse it figure out how to actually understand whatever a binary formulas in we want to clean it we want to assemble it into some sort of form that we can feed into a matrix decomposition we want to actually carry out that makers decomposition and then we want to interpret the results so you'll notice that a full maybe 50 maybe higher percent of the steps here have nothing to do with math or statistics or machine learning and so it goes with actually doing most analyses and building most bad applications most of the work lies in finding the data pre-processing the data understanding how to read it and then assembling it into some format the elektra allows you to do your analysis also please stop me for questions at any point a big fan of interruptions if at any point you are confused or curious about something in particular so let's first talk about the raw data that we're dealing with here we have the full Wikipedia data set which is all available online it's XML formatted it's about 46 gigs uncompressed and this is a little sample of what some of it looks like so we have this so XML structure that it's up fed into and then basically they launch into this full text there's all this crazy little formatting stuff that we have to deal with and then at the actual words that we care about so how do we how do we read this data how do we understand this ugly-ass format locally reading and parsing Wikipedia XML data if something that people have been doing for a little while and a lot of different settings apache mahute has a nice xml input format which is a MapReduce input format that allows you to split up this enormous XML file into individual documents that we can use this is some spark code that does it and so while this is a input format that was originally built for MapReduce spark is able to leverage any sort of MapReduce input format that was written for any strange data set and you can just feed it in in this nice beautiful little way we're referencing that XML input format and then the two data types that it returns whenever you use a MapReduce input format you return key value pairs it's a little bit more constricting than the Generalitat spark as we only care about the value we only care about the string the text so in this map function down there we we pull that out so this is what it takes to open up Wikipedia data in spark if you look at this once you've already done this so we'll have these nice little pieces given to each a spark partition well if we map over them we'll get a document per per record but there's still all this junk in here that we want to deal with so I'm gonna lie it a couple a couple steps that are the ugliest so let's remove punctuation and do splitting up strings and cut to maybe a G someone interesting part of the the cleaning step so what we'd like to do with latent semantic analysis is look at documents find out which terms show up in them and then compare them to other documents based on the terms that show up in those documents so doing this and looking at the the co-occurrences the similarities of documents for terms and terms for documents were able to come up with some sort of understanding of the relationships between terms and documents using this understanding we apply this matrix decomposition to uncover some lower dimensional lower dimensional representation of the full corpus of data so the idea is that the full corpus of data is some sort of lower dimensional representation plus a bunch of noise on top of it so by applying LSA and looking for the lowest dimensional represent or low dimensional representation that still fits the data pretty well we're able to get rid of all this noise and uncover what are some of the key concepts the key structure underlying the data set that we're looking at so to do this and it all revolves around this this concept of co-occurrences so this term shows up in this document it also shows up in this document therefore these documents are probably likely to be related in some sort of way and that a relationship increases as the number of terms that co-occur increase in actual English there are lots of ways to say the same thing both on a sentence and you know higher level but also on a particular particular word level so we all have all sorts of punctuation and grammar let's say cactus and cacti if we see cactus and one document and cacti and little document we'd probably like to strengthen the relationship between those documents even though we don't have the exact same string of characters so so it's important that we're able to reduce down into some canonical form for for each term for each entity that we that we care about the process of doing this is called limit ization and the little roots of the words we call them limbless so we'd like to be able to take strings like the boys cars are different colors and convert them into these these canonical forms like this nice English sentence the boy car be B different color doing this can be really simple Mamillius stemming stemming is sort of a low-grade way of doing a limited ation where you look at particular strings and apply a bunch of heuristics to them to try to get to their roots and you can go fancier and look at contexts look at the words around try to understand what the part of speech that a particular word is in a sentence and be a little more intelligent at this limit ization process this is fun stuff but it's also hard stuff so luckily we do not have to do it ourselves there is a really nice library at the Stanford core and Opie library which provides this this functionality so calling it is basically as simple as this little bit of code the the way it works is you build up these little NLP pipelines these little steps of operations that have to happen in sequence on a document in order to understand it in this particular case there are a few steps that are required before limit ization first you have to tokenize the document you need to split it up POS stands for a part of speech tagging which is used as input into the limitation process and then you actually do limitation 11 station itself any questions on this stuff before we you've gone I believe that Stanford Coronel people project does have support for other languages so it's trained on some reference data set which I think is maybe a Wall Street Journal data set which happens to be in English they might have other reference data sets for other languages but you can also take your own reference if you have some a huge corpus of documents and train it yourself and use those weights for the further process any other questions so yeah the question was is there any difference between what we did there and loading up a normal text file so when you load up a normal text file into spark you probably use the text file API and that's going to give you one record per line so it's gonna you know use the split points as the newline characters in this case we want to have one because we care about documents in the set of words that show up in a document we need to group the lines that are in a document together so the XML input format is actually going to do that and put multiple lines into a single record that makes sense but the API is are very similar if you look at the invocation back back over back over here this is if you look inside text file basically calling this but passing in text input format instead of XML input format yeah great question so i alighted that as well but there is a nice library I actually forget the name even though I used it we chat so let's look this script yen which which has an understanding of this wiki script and is able to strip out the relevant stuff and do the appropriate kind of things I think it's called cloud 9 like someone put it together any other questions for a cube gun um so another thing that we want to think about when we do this analysis is we have these words that show up in almost every document so that's gonna show up in every document B is gonna show up in every document including these in our analysis is only gonna add noise so it's possible that we can find an algorithm that's robust to this sort of evolution but but ultimately we know that these words are conveying no extra signal so it's pretty common in these kinds of applications to go through I'll look at every word and every document and strip out all of those and B's and some lists of 140 words if you look online it's easy to get these lists of common stop words and have them stripped out so this brings us to maybe the the heart of the of the process the heart of the pipeline which is building the term document matrix who's heard of the term term document matrix man okay great so for those who are unfamiliar so that's one refresher this is a term document matrix it's simply a matrix where we have as the columns particular terms and as the rows particular documents or the other way around but in this case because we have a record per document we're looking at documents as rows and then at each cell in our matrix we want to express the affinity for a particular term to a particular document if the term doesn't show up in that document at all we're gonna say that term has no affinity for that document and if it does we're but we want to have a higher number one very simple way that we could construct a term document matrix like this is basically having cell the number of occurrences a particular term in a particular document and this is a nice thing but for our purposes it's a little bit skewed it for example would give extra weight to documents that have lots and lots of terms in our analysis versus documents have small number of terms another problem with it is that terms that show up very often we probably want those have less wait on our analysis so if you know half the documents in our corpus have the word out rhythm in it because we're looking at some pretty skewed computer science corpus we want the the occurrence of algorithm in a document to matter less we care less this algorithm shows up in a document because algorithm shows up in almost and almost every document so in addition to being the relevance of particular term to a particular document it's also sort of a indicator of how much we care about that about that relationship to the rescue is this steam that this common scheme for filling out the eight entries of this term document matrix called tf-idf it stands for term frequency times in inverse document frequency so it starts out with this with term frequency which is basically the the number of times that a particular term shows up in a document and then IDF deals with the other problem that we were sort of discussing which is that a mini-documentary score for every word not for every document word pair which is called the the IDF the inverse document frequency of that particular word and that's simply one over the number of documents in the entire corpus that contain that word so the inverse document frequency of something that shows up very often in the data set would be very small in reality instead of just using the inverse document frequency by itself we usually take the log of it because that's the sort of distribution that term recurrent occurrences in documents I tend to take any questions on that for a coupon that's a great question you know what the monkey algorithm yeah look at that yeah it bridges sort of the tail aspect of the world in that let's call a speck something well so so what does it look like to actually represent this this term document matrix in spark and in Scala the best way to do it is probably to build an RDD of vectors these vectors are not Java dot util dot vectors they are or dry patchy that spark that Lynne alders what is it hungry a vector can be represented either by a dense vector which uses a array under the covers I think Delegates down to a library called breeze which allows you to do for ab numpy blasts have operations on these arrays or there's a sparse representation which which stores vectors as a sequence of index and value pairs in this case I think we probably want to be using the sparse representation because in any particular document the the words that show up in it are much small fraction of the total number of words that could show up in all documents so so using lots of magic that's totally alighted for our purposes we build up as our UD effectors from that our DD of strings that we had a little while ago we apply limited Asian do all sorts of counts calculated CFIDS and shove them in these and these vector structures if those assumption so so when you say the metadata is is there metadata that you care about other than let's say the title of the document mm-hmm I see right so yeah the way I've traditionally done that has been basically to maintain like a and so maybe store each picture as a tuple a vector comma ID which will maybe be along or a string and then elsewhere have an RDD or local structure that map's strings to strings to metadata so you can you can go and do that mapping you leave a confuse that it says something your Trident has not been great for your problem [Music] and all my clients are supposed to be able to handle this I just learned today the pipeline library they don't pass through IDs mm-hmm yeah I mean so it my my particular analysis we haven't done hashing TF so we have do you have I mean it's sort so this is with ml pipeline's API mm-hmm what it will you merge the what's it called out count vectorizer count vectorizer um thank you all audience members have answered this question for me I appreciate it any more questions before we get going so now we get to actually apply the SVD this is fairly uh I guess officially explained what it actually means so the singular value decomposition is a matrix decomposition you take some further matrix and you decompose it into three matrices that when you multiply them back together achieve the original matrix we look at our data we have M number of documents and n number of terms we're going to get these three matrices U is a M by n matrix S is a or so we're assuming that we have fewer documents than terms in this situation or rather fewer terms and documents in the Wikipedia is that we have five million documents and there's probably something like fifty thousand or a hundred thousand relevant terms that we care about for a particular analysis like this so the the minimum number there it determines the the minimum number of the term in documents determines the number of the shape of these matrices that we look at so assuming that n is the number of terms and that's smaller than the number of documents we have this matrix U which is that which is M by n will actually show these in a second we have this matrix S which is it which is n by n and we have a matrix V which is n by n as well so this actually look like so there's our you matrix there's our estimated tricks and there is our V matrix the u matrix roughly gives us information about all the documents in our corpus the V matrix gives us information about all the terms in our corpus and then the S matrix is a matrix that's provide unites the two and give us a particular rank on the different combinations here so let's be a little bit less less hand-wavy about this and also talk about this concept of a low-rank approximation we go back so the central idea of applying LSA of applying latent semantic analysis to our corpus as we talked about earlier is we'd like to uncover some sort of lower dimensional representation of our corpus that we think is somehow more true to it we hope that our corpus is some sort of more condensed representation plus a bunch of noise and if we look at if we throw away all that noise and look for what simpler we can get more information about about what is going on so so what like what particularly is this low-rank approximation going to do first of all it's going to help us account for a tsunami by condensing related terms so sanata me is the situation where the same word or rather two words mean the same thing and tend to show up in the same concept the same context so you may be method and function we all know this or not exactly the same thing but they probably show up in a lot of the same context probably a mean lot of the same things so the low-rank approximation ideally will be able to account for this account for this Anatomy and show that the connection between method and and function and when we say show the connection we might mean that when we look at two different documents I want to look at how similar they are to each other if we see method in one of the documents and function in another document we count that toward there's some similarity which without this lower Kirkham presentation we're not we're not at all able to do so LSA is going to we uncover these kind of relationships between between these different terms for us we want to account for a poly semi by placing less weight on terms that have multiple meanings so sorry if a particular word has multiple meanings we can sort of smooth that out and have that have those connections matter less so if we see that let's say I can't think of any example right now someone tell me example what is it mean great great example use that in all my future talks thank you the word mean means couple things that means not nice and also means mean so clear things up for you ideally we'd like to would like to if we see mean as not nice and one document and mean as to have meaning or a meaning in and another document we'd like to have have a not a huge connection between those two documents so LSA can't actually distinguish between these two meanings in any special way it doesn't look at the context around particular words but it can say let's wait mean a little bit less compared to other words because we know that it that it shows up in these two different broad context and allow other words to come up and and dictate which documents are similar to each other and then last of all we hope they can just sort of throw at the random noise that tends to accrue and in real-world data so so SPD singular value decomposition allows us to find the rank K approximation that has the lowest for BD and just Frobenius distance from the original matrix I can use LSA to identify words that have different meanings I haven't thought about that it's possible that you could look at words that decrease in importance in the low ranked representation but I haven't thought about exactly how you do that it's a good question so the terms that we're using in this situation are just words not part of the speech tagging that's actually a great point that if you wanted to distinguish between mean and meaning or those two different use of the word mean if you had unique terms based on part of speech tagging that you had derived from the Stanford part of speech tagger that would likely allow you to be more accurate and your analysis the diet words or like the title attending needed by emphasizing more we're entitled so we're treating the entire so we're not looking at the title at all and you know that's probably not the cheat because because we do we think those things might have some effect and then we're treating the rest of the bag of words we don't look at ordering we don't look at any sort of sentence structure we don't look at any sort of punctuation just backwards any other questions so SVD can find the rank K approximation that has the lowest for being assistance from the original matrix what does that mean loosely it means that we can find a if our original matrix is in let's say n dimensional space so so so we're expressing everything in terms of if you have 50,000 terms we're expressing our corpus in terms in 50,000 dimensional space we can pick some lower number some number that's lower than 50,000 K let's call it let's say a thousand perhaps and we can express our entire corpus in K dimensional space so so so given given these three matrices that we decompose after after finding this little rank approximation we can show them back together and we're again we're gonna get this 50,000 by five million matrix but it's gonna be a lower rank like there are there are fewer basis vectors that are required to express all the information inside that inside that matrix and in particular of all the matrices that have rank K that are expressible with a with a lower number of dimensions we're gonna find the one that's the closest to the original matrix by this metric called Frobenius distance which is basically summing all the squares of differences between corresponding elements in two matrices so apologies if this is sort of too quick or too hand wavy but please ask me questions about this before we go on the basic idea here is we're trying to find a low-rank approximation that still is somewhat faithful to the original matrix that we were looking at so we talked about before we have these two parameters and five million documents and 50,000 terms and we add this additional a parameter K which is the number of dimensions and the lower-ranked representation that we care about that's something that's not chosen by us we could test out different values that we wanted to and we use K to basically take this these three matrices us V and chop them down a little bit we can basically just sort them in a certain way and I simply cut off elements to get our lower ranked representation so so we end up with these truncated versions of us and B U is a matrix where every row corresponds to a document and then we have K columns s is a matrix where we have a diagonal matrix so the only nonzero elements are on the diagonal and it's K by K and then B for every for every column we have a particular term and then all the rows are our K so we have a yeah let's get like here so one way to think about this is in terms of contents so given our enormous data set with 50,000 terms and five million and 5 million documents we want to decompose it into K concepts that we think are useful enough to to represent it so if we look at our matrices we have we have these Docs these documents on the on the rows and all the columns each column is a particular concept so if we look at this concept which is one of these columns of our U of our document concept matrix will have some value for every document and that value is basically telling you how much that document participates in that concept so assuming there are LSA or SVD was able to uncover concepts that we think of as concepts like let's say math then we would find you know along the math concept that all the the documents like let's say algebraic topology and calculus have high weights in that column and documents like flowers and San Francisco might have low weights so so again SVD is able to take our data and decompose it into some set of concepts this the same thing applies for four terms so for each of these each of these columns in our U matrix we have a corresponding row in our V matrix so so the rows that are let's say the furthest left on the document matrix correspond to the rows that are the highest up on the other term matrix so so if math is our particular concept we might see that for the representation of that concept in the term matrix we see words like calculus vector addition triangle come up I have high weights where words like flour Apple Tarzan what would have would have low weights that makes some sense so so so you know what was really important understand about about this is that there's this there's this matching between columns of U the document matrix and rows of the term the V matrix and then what sort of unites these two with this with this s matrix in the middle is a there's gonna be a single value for every concept in this in this s matrix and you know we're gonna look at look down as we look at the diagonal of this of this matrix there'll be a day a value per concept and that value expresses the importance of that concept in the entire data set so so if the partition a participation of a document in that or a term and that concept it explains a lot or rather if the concept is able to explain a lot of the variability in the data set it's gonna have a high value and s it's going to be more important and if it's a low grade concept it's going to be a less important have a or value inside this diagonal of this s matrix any questions on that actually writing this analysis is super nice you take your RTD vectors you build the row makes matrix out of it which is simply wrapping it and it's real matrix object you decide on your K and then you tell the matrix to compute its own SPD and that gives you back this this this SPD object which basically contains these three matrices that we've talked about you have to tell it to compute U which is a little bit more expensive if you want to use this u matrix use a use a big matrix so it's stored as an RDD again it has one element per per document which is per record and our original RTD and V is a smaller matrix so it's only gonna have like say a 50,000 50,000 elements in this longest dimension if we if we only care about 50,000 terms so we've gotten this nice it's nice that decomposition these three matrices that if we were to multiply together we would get some sort of similar matrix to the original matrix that we had but in a lower rank representation and we'd like to be able to take all these numbers and ask questions of that one question that we can ask is what is what are the top concepts that we care about what like what is this what are the main axes along which our variation in our data is expressed what dimensions in terms of space and document space explain most of the variance in our data so so again as we talk to it before we can we can look at our matrices here and the first concept the most important concept will be represented by these little shaded shaded green parts it's going to be the leftmost column of U the left and top most value in s and the topmost value in B we can look at the second-most important concept which just moves everything a step over exactly yeah and so the exactly that's exactly what this code is doing right here so to go back if we're looking at the top words and the top concept we want to look at basically that that green bar which is that all of the which is the top row of the V matrix and that's going to have a set of numbers and each number is going to correspond to a particular term as you said we take the numbers which are the highest and we look at the terms for those particular numbers and those are the most important terms so that so those concepts so this is a bit of a bit of it's not really spark code it's a scholar code that's doing exactly this it's fetching the V matrix from the SVD looking for the top the top values in it and finding the terms that that correspond to it how do you interpret negative numbers just as less less enter this is there some sort of how would you I guess you would say that that term is very opposed to that particular concept so they like really don't Co occur very much at all or like they're like the incidence of that concept in a document is gonna make you unlikely to see that particular term I think that's right and then so this is doing the exactly rather yeah so this bit of code is doing the exact same thing that you were talking about but for documents so it's finding for the most important document or rather the most important concept what are the particular documents that have the highest affinity for that concept in this case where we use the distributed an RDD operation to do this because the you matrix the document space matrix is represented as an RDD then the code is pretty concise in scala for carrying this out here's concept one in our in our corpus so these are the terms that I have the highest affinity for it and these are the documents that have the highest affinity for it it seems like this is a little French townships or something like that which I guess is pretty is anybody heard of these these places departments like and there's no equivalent in the US okay well I'll just except for you remain confused so so so yeah sorry what do you say Department but so Department and like a government sense like a bureau or like or like a like a County like it like a region okay okay so that is French law regions apparently explains a lot of variability in in Wikipedia so so maybe we could you know call this French regions as a way to describe this concept maybe there's not a simple way of expressing it but it seems like these are at least thematically we wouldn't covered some set of things that are thematically related concept to so we have terms like that genus species moth family Lepidoptera and the documents are all these particular genesis of bugs so this seems to be a related to the biology of bugs and again it's kind of cool our LSA has totally grouped these together for us we haven't done any manual supervision to say that let's say the word genius is related to the word species or that Gigantor trike culture and Alice is related parapodia any questions on that so this is nice a sort of inspect and make sure that we've done something pretty sane what if we want to do some particular task that's maybe more useful to someone who's querying so so let's say given a particular set of terms we'd like to find the closest documents in in the latent space so we're saying we take the simplified representation about your data we we input a bunch of a bunch of words and we might want to find documents that are let's say conceptually related this is a search problem this is like what you're doing Google but additions are just searching based on occurrence we might find documents that don't have any of the terms that we've searched for so we might type in method as a search storm search term and come up with documents that have the word function in those terms because we know that or rather the low representation understands that method and function are pretty related to each other so so what we're interested when we do this is basically the the score for a term and a document remember if you if we go all the way back over here and we looked at the term document matrix we had these these numbers that Express the affinity for a particular term and a particular document and so if we look at our LSA representation we basically have the same thing so if we imagine we were to take these three lower rack matrices and multiply them together we'd again get this term document matrix but the particular values in this matrix would be a little bit different in the tf-idf values that we figured out earlier they didn't have this this merging the smooshing of related words as well as accounting for Polly sent me by giving a deeper prioritizing words and and it basically would carry through this low-rank approximation back into this reconstructed matrix so so again if we want to take a particular term and find the documents that are most relevant to it we look at that term and look at all the scores for all the documents I you know it in the in the column of that term and pick the ones that are that are at the top if we want to search for multiple terms we can just do that and smush together those scores to find some sort of waiting also if we were going to do that we'd probably want to take IDF weights for the terms for the query terms in order to prioritize rarer terms and and our waiting so the nice thing is that we don't actually have to reconstruct this enormous enormous matrix if we were to do so it would likely be fairly expensive one one fact about this matrix is that because we're doing this LSA and this little record proximate low-rank approximation it's likely to no longer be sparse like the original matrix was so we're likely to see nonzero values in a lot more of the lot more of the spaces so if we had to reconstruct this it would be very we it would use a lot of memory space and then computation luckily do it like doing this does not require us to do this full reconstruction we can basically cancel out some terms and in our matrix operations be a little more clever so one last thing going back to this is suppose we don't want to compare a tournament document suppose we want to compare two terms we basically look at the the vectors the columns of this matrix that are associated with those terms and take the cosine similarity between these between those two terms so we take term one term to do let's say a dot product between them and that's gonna tell us how similar term one is to two term two so this allows us to not just find the top documents for a term but also find a top terms that are most related to any particular term right that's what this that's what this bit of code does it finds the top terms that are related to our term so if we type in Radiohead as a term we get Radiohead directly catchy riffs lyric this last one is a little weird upbeat you know our algorithm is doing something kind of weird but maybe that should have a negative opinion II if we do a print relevant terms for for algorithm we find algorithm heuristic compute constraint optimization complexity these all seem relatively reasonable to us on the on the in that little column we're looking at the similarity which is basically the score that the cosine similarity between those two columns what is uh I think this is just the same thing as we had on that last slide so querying like we talked about before the top documents for a particular term if we type in type in for kind of tree we get all these trees that are related to fir trees or maybe they're not particular blade to fir trees or just other trees and so again this is based on these latent symmetric relationships and I guess that's pretty much the last example that we have so that is all I have for you if you want any more detail on this we have we recently wrote this book advanced analyze with spark there's a chapter that deals with a use case that's much like this all the code for this is available on github so you can go there and check it out and then the ML Lib documentation itself has a lot of information on how you can use SVD that has the hashing TF and the count vectorizer and all this nice stuff that allows you to pursue your NLP quest I think that's yeah thank you very much [Applause] yeah it's a good question I think I was using a six node cluster which had maybe like 24 corals on each node as and no 100 gigs of ram something like that it took about 30 minutes for a you know a full SVD run and then it was once the results were stored a memory is pretty cheap to actually compute those can be those result like scores yeah exactly don't quote me on that Minh some multiple of that like two hours I did not compare with all the a that would have been even for the actual topic stuff that would have an interesting I did not have the chance to do that another Google that could not familiar with that Oh Google that go right were they like trained it with like neural nets and stuff like that whatever I did not yeah I have a chance to compare it with us so the question is all the major seasons that they were looking at were those marked data structures yes oh so when we looked at the row matrix that's a spark matrix of represented by an RTD underneath there's also a matrix class which sits on top of a breeze matrix but has spark apos on top of it shall we one more question all right thank you [Music]