Text By the Bay 2015: Dmitri Gusev, Extended Swadesh List
Recording: Text By the Bay 2015: Dmitri Gusev, Extended Swadesh List
attention and uh we will begin the talk on extended swadish list version 1.3 uh I represent Purdue University I teach computer and information technology there and let me begin with the epigraphs uh words are useless especially sentences attributed probably best to Bor and another one that matters uh probably even more uh Randomness is a beast and you fight it with volume uh this is a quote by Ed Scher uh the author of two prominent uh chess engines Rebel and prodo uh who experienced firsthand uh how many games you have to test an chess Engine with to get a uh good stats on its relative strength strength to other chess engines and what is displayed here is one of the many uh Renditions of the legendary Tower of Babel uh and the uh introduction here is more from slides for the students so most of this can be skipped except uh for the idea that this belongs to the subfield within natural language processing that is known as comparative historical Linguistics and we are studying how the languages are related and their people within the field they try to reconstruct protol languages arrange existing languages in groups of related languages and naturally one uh question that comes to mind is can computers help comparative linguists in their work and um a few of them actually worked on that and found out that yes they can we just have to be careful what we're doing uh the inspir ations for this particular study are Nate silver I don't think there's a need to tell you what this guy has done and Graham Banks who you might not know but he is one of the founders of the prominent rating of Chess engines ccrl and there we have uh chess engines rated based on statistics of thousands of uh games that they play against each other to figure out how relatively strong those different juristic that play chess are how does that relate to Linguistics it does in the field of Lex statistics and glottochronology um there will be no glot of chronology here done per se uh but Lexica statistics attempts to classify languages based on Counting pairs of so-called cognates which are words that have a common origin and this approach was pioneered by Morris swades a famous American linguist of Jewish origin and that's whom swadish lists are named after they list many basic meanings of the language and the most popular currently used list contains of 207 meanings and it was obtained by combining the original 100w and 200w lists designed by Swedish himself there were some intersections there and that's why we have 207 words in the 207 me meaning list notice that the original Swedish list is English Centric in the sense that Swedish picked the meanings that could be expressed with one and hopefully only one English word and we will try to uh step away from being English Centric uh this is just a a sampling of the 207 meaning swadish list obtained from wiary as a matter of fact I separate thou from you in the subsequent uh processing which is really no big deal because that makes it 208 instead of 207 uh so there meanings there the corresponding ones from English French and German which is an interesting triangle because English is a Germanic language contaminated with uh some French and so there's a triangle there that determines okay is our English um conly closer uh to German than to French actually it is but there's some uh even on the Swedish list there is a word Mountain that's adopted uh borrowed from French the objective of the study ultimately is to find out okay do languages actually uh diverge or continue to diverge that is to say um most comparative linguists agreed like in this picture of fireworks that we are deal with clusters of related languages uh there is no way to deny that and at some point uh they diverged from one or more protol languages thus the reference to the legend of the Tower of Babel and the approach uh is to take the languages that are well documented such as Coptic uh sorry such as middle Egyptian ancient Hebrew ancient Greek Latin Sanskrit uh or oral Church slavonic Old English and such and look at their either direct uh direct derivatives such as uh for instance it looks like Bulgarian would be the best choice for old church slavonic it's based on prior studies it looks like either Spanish or Italian would be the best choice for the language that came originated from Latin French is further away from Latin than Spanish or Italian or you know Romanian also is further away uh standard Hindi has an interesting Quirk that there were some words that were from far that were uh taken into such languages as uru and then then uh since Sanskrit was documented the some Lang some words were returned into standard Hindi as Hindi was being standardized which makes it closer to Sanskrit than it would have otherwise been however as far as the word stability is concerned it's okay with me because that's one way for a word to survive be documented and then be returned into its predecessor language um we could take in some case a quote unquote nephew language such as Modern Standard Arabic as a counterpart to ancient Hebrew and uh do the following thing compare those languages pairwise and to their ancestors and uh compare their ancestors pairwise to see what the shape of the resulting uh Tower of Babel would be is it diverging like these fireworks or is it actually converging as borrowings appear across the modern languages and uh make them closer than their ancient uh ancestors maybe and it's unlikely that it's slanted in One Direction so I don't think that would happen so either the Tower of Babel grows that way or it grows this way or in a strange Quirk it would grow vertically and they would stay at the same uh distance from each other after they diverged initially and we don't know the answer to that uh the natural question being well hasn't it been done before uh not exactly there is a classical publication by Robert Lee from 1953 on the basis of glot chronology which reports cognate percentages for 13 such language pairs some of which I just listed in like Old English and Modern English middle Egyptian versus Coptic classic Latin versus modern Tusan which is I suppose close somewhat to Modern Italian uh ancient classical Chinese to Modern Mandarin and uh I spoke to colleagues here about the interesting varieties of Chinese uh and uh he used however Swedish lists that contain 200 to 24 meanings not even the same number of meanss for the same pairs uh but considering the modern methods where we had uh many thousands of people ped to enable Nate silver predict correctly the results of presidential elections for first for 49 out of 50 states and then 50 out of 50 next time or or how many thousands of games now uh chess engines play with each other to get uh good ratings of their strength uh why would we just stay at 200 to 214 meanings uh what would we do with the words that represent those meanings is another thing and that's approximate string matching with uh some interesting variations to that that we would get to next uh there would be some the tradition is from damro and Lenin and was used initially for uh correction of typos so all right spell checking and uh they would um check into basic operations such as insertion deletion and substitution that would allow to transform one word into another as in this example taken from the textbook uh transforming the German Appel into an English Apple interestingly not using transposition which damro did where if we use transposition we could transpose e and l at the end and get a better distance uh from Appel to Apple distance being a function that has certain properties this is for the students I'm sure everybody here has either a formal or good intuitive notion of what a distance should be uh lenain distances are well known they were introduced back in 1965 lenain distance one allows insertions deletions and substitutions no transpositions like I pointed out distance two uh doesn't even take substitution so for that to do a substitution you have to do a deletion then you do an insertion uh the Wagner Fisher algorithm is a classic that's again found in textbooks there's an example here there was in fact another talk earlier in the same conference that also gave a a different example of how the Wagner Fisher algorithm would compute 11 stain distances in this case it computes 11in distance one what we do with a set of words however okay for an individual word we know how to compute the Leen stain distance what we will do if we have a set of 200 or maybe more words uh corresponding to meanings in the swadish list well then let's notice that the word length impact uh is significant here if we in language one uh to language two have some short words such as a versus e ba versus C Co versus C and so on uh it looks like uh uh would not get a uh we good uh we would get uh short distance because the words are short when we take the longer words that are off and off Bob and Bop uh s and sick here and here uh we would get larger distances because the words are long even though actually the words are more alike than the previous ones uh that trouble can be removed though uh yes okay uh by normalizing this by the sum of maximum lengths of the two words involved in comparison for Lenin distance one and some of all lengths of the words involved in the comparison for Lenin distance 2 the reason why we normalize by different things by different distances is because in one case we have substitutions in the other case we don't uh the code for that is available on my GitHub that I created for that and it's called the project is called Wagner Fisher swsh you can get that code it's actually pretty easy uh there are some remaining problems even under this approach and we'll get to them soon one is with translators false friends if the meaning changed a little bit like for the English 2 versus Russian do uh the cognates are there but the meaning changed uh it's actually closer to do in do and some other Slavic languages than in Russian to the English meaning of two uh then how do we weigh the difference in meanings versus 11in distance that we can mathematically compute no problem between the spellings of the corresponding words let me leave that open show you a funny picture from a paper by pel and others from 2007 that shows a tree of Indo-European languages uh I inserted a smiley where instead of Slavic it says Islamic uh that's a typo there in a respective uh Journal I think in science uh the the trees can be built like that and so here we have a sample tree from that same paper uh showing a group of uh Slavic languages Baltic languages nearby uh Scandinavian languages Ro Roman languages uh Germanic languages and so on and so forth uh so yes languages can be changed in trees their mathematical methods that are well known for doing that such as minimum uh uh length spanning tree uh maximum likelihood tree and so on it's similar to the methods that are applied in genomics to create trees of how the Alor how the organisms evolved and that's not dissimilar to how languages evolved uh other studies were published where uh the trees built from language similar similarity uh were compared against the genetic studies and mostly the genetic trees of how the people evolved genetically uh match their language groups with some notable exceptions here where there are some mismatches uh like with dravidian language which is uh away from Indo European and is spoken mostly in Southern India it turned out when they anal Iz genetically the southeast Indian population uh they find more closeness to the rest of India then they find between the Indo-European languages such as uh Hindi and Udu and whatnot that the population speaks there as compared to dravidian languages so that's unusual another good example of that is Hungarian where they do a genetic analysis on hungarians turned out they're closest to the neighboring Slavic people however the language of hungarians is closest to the Finnish and utian languages spoken in Scandinavia and the urals uh there is a theory of nostratic languages that groups languages and several uh families this was developed by ilich vitage and uh the map here shows how these nostratic languages would be distributed according to this Theory across the old old world and let's observe though that the languages are written using different alphabets and different other systems of writing such as Chinese hieroglyphics and we should think okay if the language is written in a different alphabet that doesn't immediately make it all that different from a neighbors language we should switch to some other approach such as using phon names uh which are units of sound like okay if it's if it's written differently then does it sound differently uh the approach here is originated with bodu and decordon a prominent polish and Russian linguist uh and uh there are two popular phonetic alphabets for writing things using phms IA and APA and one is more popular in the United States another one is more popular in Europe and mil um from Ohio State now he is at North Carolina made an important step here to propose a phony similarity Matrix which would mean a set of Weights assigned to Transitions from one phon to another so that if we're doing a deletion or insertion of some phon or or substitution of one phon for another we would not uniformly assign one to the cost of doing that maybe it's less costly to replace with r or less costly to replace p with f than some other replacement uh it should be not very costly to replace M with B if uh we have stuffy nose um so uh this is a next step to extend the approach Lenin distances and people study that for instance Grant condra from the University of Alberta published papers along cognate identification using such phonetic similarities so again not new but they're also interesting problems with phonetics uh with phonetics of the ancient languages that we just mentioned such as ancient Egyptian or ancient Hebrew or even Sanskrit they may sometimes be difficult to determine or even Latin for that matter because the languages are quote unquote dead we just don't have uh any surviving native speakers of middle Egyptian or uh you know Latin or whatever ancient Greek uh then we really have no choice will'll end up using modern reconstructions of ancient phonetic systems and we recognize the danger but we really don't have a choice as far as I can see another question is how we should treat synonyms if we have different words with identical or similar meanings um and uh that I think I will uh postpone any discussion of and move on because uh I have a short talk but the uh slides are available and uh okay one possible solution instead of phones use something like the sa ajp code proposed by hman and others and they devis the code that would allow to ENT enter the data for swsh lists using uh the qw e r Ty keyboard only and restricted it to 34 consonant symbols and only seven symbols for vowels uh I wish they continued to distinguish between e and U for example they don't um but anyways uh that's the idea though it's a good idea because with phms they good for capturing differences between uh uh dialects um okay uh so let's uh uh quickly explain uh basically we said that for statistical purposes to get representative statistical samples we want to extend this swadish list and then let me skip a few part parts of how we would oh no I wanted to talk about cats no no time to talk about cats uh let me just show this graph how the Lenin distances stabilized in an extended swadish list extended to 620 plus meanings uh from uh the me from 100 or 200 meanings uh that's the main point behind extending this wsh list is that this Lenin distances with will stabilize and that may matter when the differences between distances are at stake uh thank you very much do we have any time for questions no okay then you are welcome to ask me questions after words uh thank you very much for this forum's hospitality and uh let's uh uh the slides are available like I said uh thank you very much and have a great day