Text By the Bay 2015: Sergei Winitzki, Traditional Russian Orthography for the 21st Century
so let me begin um I'm going to talk about the traditional Russian orthography um I will say what it is what it is the difference between the modern autography and the traditional one uh why we need to use that traditional autography today um I will explain what are the special challenges that we encounter when we try to do that and um how a project that I started about 15 years ago uh coped with that and enabled me to do some text processing in the old orography um so what is called old or traditional Russian orography is in fact the orthography used by the classic Russian literature so I listed some writers that you might have heard about even if you're not a Russian speaker um by the way may I ask who is a Russian speaker all right um so I will show you something that would make sense even if you're not Russian speaker um but basically all these writers are uh the ones who studied the Russian in the old orthography at school and continued to write in the same way in 198 the autography was reformed um four letters were eliminated from the alphabet and replaced by other letters these letters were actually physically removed from printing presses in most cases and in some cases by mistake another letter was also removed that was similar to the one uh as you can see um that was not supposed to be eliminated and uh people who saw Russian documents dating from the early 20s and early 30s often saw the lack of that letter being compensated by an apostrophe um the spelling of certain prefixes suffixes and certain word endings was also changed um I will talk about that in some more detail the motivation for this reform was to help the poor School pups who had so much trouble learning the spelling and to um simplify and bring the spelling closer to the pronunciation which it did to some extent however outside Russia the old orography continued to be in use well into the 1940s um for instance this is a first edition of nabokov's short story storm that was printed in Berlin in 1930 Nabokov started out as a uh writer for the Russian community in uh Berlin and Paris and his Works were printed in the traditional autography now for those of you who are Russian speakers you see there are several changes that are obvious by looking at the text such as the letters that were eliminated and U other changes now it is not unknown that language orthography changes that's natural process here's a an excerpt from 17th century English text and if you look at it you see small changes such as a double l in natural or the spelling of um the words wisdom with an e at the end or the word judge without an e at the end um these changes are relatively small so suppose in some situation we would like to produce a text in this orthography or would like to uh have a critical edition of a classic text so how would we prepare let's say a spell checker for this orography um there are some very small changes they're quite systematic so basically a dictionary would suffice um you don't think so wasn't standardized at that time you get the same different way um certainly so you could you could try to yes so you could try to analyze that and find some kind of prevalent spelling that you could declare to be the spelling at that time in order to prepare a critical Edition um I would definitely solve most of your problems and this is because the changes in orography were gradual um they occurred naturally well not so when the changes were mandated by a committee of experts um in order to give you a flavor of what kind of changes there were I prepared here an imaginary proposal for reform of the English spelling that would be similar in detail to what happened to the Russian orthography so let's first of all omit the silent e in word endings and then we'll replace other vowels so that spelling is simplified so for instance instead of made we just spell m a i d and instead of some s o m e we spell suum that's just simpler and we eliminate the the E whenever it's not pronounced then we replace the letters CX and Y by k s and I according to pronunciation so let's just replace all that so instead of hence you would write hence with an S uh and instead of play you would write p a i d so let's make this change in order to simplify the spelling um and of course we physically remove the letters CX and Y from computer keyboards these letters are um undesirable and we attach political significance to their removal so that nobody can object so now you see what kind of problems will be created right away by this kind of Reform uh the words that previously um differed in spelling now don't and you have to know how to pronounce them out of context such as uh PL or hence um made and so on so now you you see what would happen if you wanted to prepare a text in the old orography given the new one you have a lot of disambiguation to do so um however this was what happened and the result of it was that people got used to the new orthography um old books became unpopular it's a lot of work to change that in old books because you mandated that no books can be printed in the old phography anymore so only a select set of books are printed at all uh from the old ones so you effectively lose access lose touch with your culture with the accumulated experience um of your country and that's what happens in Russia Today the texts presented in this orography are felt to be irrevocably obsolete completely out of touch with anything that can be relevant today um and of course no computers today support text processing in this orthography it this problem would be solved if we had computer support and a and a converter between old and new orography of course back then they didn't have computers everything had to be done by hand today we would have if we had a converter there's no problem you don't like reading these old letters just convert and you or if you want to print a critical edition of some old words just convert so what would be the challenges in conversion um so when I started this project about 15 years ago Unicode was not a widespread technology people mostly used 8bit encodings at that time no 8bit encodings for cilc contained the four letters that were eliminated in 1918 there was no keyboard layout certainly containing that because it couldn't have a an encoding screen fonts did not include these characters there were very few Vector fonts that did um but at that time vector fonts were not used for screen displays they were too slow so people used bitmapped fonts um latch curiously had support for this using a non-standard encoding prepared by the American mathematical Society but there was no support for spelling checker or hyphenation or optical character recognition what is the situation today much better these letters are present most fonts everything is uni code you can print you can type these letters if if you go to Wikipedia and cut and paste because Wikipedia has examples of uh these texts latch still has a font support which is marginal but existing there is now spelling checker support uh coming from my project and perhaps coming from other people um I spell know sure if everybody knows what that is is an old Unix text mode program for spelling check um open office has support for that I I believe but it's not a standard part of the package there is some uh commercial programs that can do optical character recognition in this orography fine reader but there is still no support for hin nation as far as I know and uh there are no standard keyboard layouts that you can install today if you wanted to prepare a text so what did I have to do back back then I prepared and first of all an 8bit encoding there was no practical way of doing anything without an 8bit encoding so I changed a few characters in co8 and I made an encoding I made a localization for a text mode browser called links um I prepared keyboard layouts for for the X Windows system and for Microsoft Windows so that I could type in this uh encoding I prepared the onscreen fonts that uh were based on a previously previously existing killic uh onscreen fonts for X window these fonts now uh became part of standard packages that are still um installable in in Linux the C fonts package I prepared a dictionary for the I spel program which is um an AIX based dictionary that is you have word stems and a list of AIX combinations that can be applied to certain word stems um finally I made a a postcript type Setter that is a program that could emit postcript files that you could print uh with uh onscreen fonts used in instead of uh Vector fonts because there weren weren't usable Vector fonts and I prepared also a dictionary based converter between new and old orography so all that was done in Pearl which is the which was the python of that time so to speak here is what it looked like uh this is a screenshot from that time in Netscape superimposed with the upper part of the aski table in the encoding that I created with some letters put in certain places um so I was able to process text um for instance here I was actually editing this text this is um text printed in 1924 in Berlin the spelling check looked like this you opened the different terminal window and there you had an interactive suggestion for the word that was misspelled so spelling check was not something that you did automatically in the background as it is done today but it was a separate operation and so here you see a word that is spelled in the new orthography and that is incorrect and this is a possible replacement that you had to do by hand so in this way I was able to edit text print them put them on the web and um well for the few people who had interest for this that was quite sufficient spell checking um was fine optical character recognition did not quite work so well because you had to uh first teach the recognizer that there are some new characters and there's no dictionary and so there was a lot of spellchecking and proof reading to be done but at least it could be done so what let me talk you talk about the linguistic challenges um the basic problem uh with the transition between the traditional autography and the modern one is that there are many changes that created ambiguities these changes for example made certain words uh homographs that is words that are written in the same way but pronounced differently just like the example I gave you on the other slide when you write uh made with M Aid uh that creates an ambiguity that didn't exist before in the language and so a lot of these examples now had to be disambiguated in some way there are some words that became actually homonyms that is words spelled in the same way and pronounced in the same way but meaning different things and they're used to be not spelled in the same way um so these are some examples where you have to introduce the letter that was eliminated in some cases but not in all cases so you have to know when this needs to be done and these words are not necessarily um different parts of speech there are sometimes different parts of speech but not always so um restoring old letters in other places is less problematic it is Can it can be done by dictionary for instance the letter Theta corresponds to the places where the the word of Greek origin would have Theta in it this is quite typical like the word orthography itself must use the word that must use the letter that was eliminated um and there are some examples like this where you replace the suffix you replace um the the root of the word and um you just use a dictionary so what I was able to do is mostly this kind of replacement that is dictionary based because at that time there was not a very easy way of uh getting a part of of speech tagger or um disambiguation based on semantics however it is not always sufficient to identify parts of speech in some in some cases it is in other cases you have to identify the grammatical forms because um you insert this eliminated letter only in certain grammatical cases but not in other grammatical cases in um in other in other cases it's even more difficult because you have to understand the meaning of the word for example the word for peace and for the world um has become a homonym this is the same part of speech and it's not possible to disambiguate unless you know what the sentence is about um similar the word for flying and for healing became a word that spelled the same and pronounced the same and so in order to disambiguate that to convert to the old orography you have to guess the semantics of the word in a sentence um that was definitely beyond anything I could do so I left that for the U manual check for the manual proof reading giving a example sometimes you can get disambiguation based on pronunciation so I'm pretty sure that today I could have done a much better job uh my converter worked in some simple ways um it had no real disambiguation it had a dictionary based lookup with some heris sixs so here's an example of converting um a poem that most Russian speakers would recognize to the new orthography you see that uh you would have to insert a few things in a select locations and that is uh how I would expect this to work um today um I'm hopeful that we'll have uh better support for custom keyboards in different operating systems that's a used to be much easier now there's a trend of not allowing the users the freedom to install different keyboard layouts um hyphenation dictionaries for latch for example or for other type setting systems is an outstanding problem that I was not able to solve um by just um hobbyist effort spelling dictionaries are available but grammar checking is another thing that is not quite available today for um many languages that could be another project and finally an ambitious project would be a fully disambiguating converter that would use all our Technologies today in natural language processing the um part of speech tagging semantic uh guessing could be quite useful here I'm pretty sure that uh results would be a lot better um if I could use these Technologies so thank you very much for your attention and uh I'll be happy to answer any questions yeah have you thought of implementing some of that or maybe some of that is all platform comp as far as mobile platforms such as Android iOS so um to implement a custom keyboard for example would be the first step uh most mobile platforms already Implement full uni code um I'm not sure that they do a lot of spell checking maybe they do um so what precisely would you like to see on your Android device because you can already see web pages uh and texts right so let's discuss that after okay uh sure we can discuss that I mean this is a this is definitely a a project that can be undertaken yes just taking one ofation soents so uh when you lemmatize you get sometimes ambiguous words so you have uh words that are spelled exactly the same in the modern autography but should be spelled differently depending on their uh meaning or depending on on the grammatical case and so on in the autography so some of the lemmatization will will help sorry lemmatization will be necessary it will help in some cases uh um depending on dictionary having forms of every word yes well so you have a word that means different things as a dictionary helps you to understand and so uh when you lemmatize do you do you get the one meaning that is actually used in the text or you just get one uh word stem that tells you what possible meanings it might have so for for example if if the words uh U peace and world were the same word would lemmatization help you distinguish them in traditionaly LMA would mean dictionary entry so wind and L would be different entries even though they're spelled the same and bank and bank would be different entries even they spelled the same pronounce the same so in that sense of lization then it would get give you what you want but of course some people use it just to separating and just in which case it wouldn't so okay and and so so is that already available so can you already lemmatize in this in the in the sense that you just described I don't know right well so um if you could yes so if you could lemmatize so so fine grainy as to distinguish words that are spelled the same and pronounced the same but mean different things uh somehow then certainly it would I would say it would solve the problem it would it would have to again a criminological point of view there's a um a morph analyzer would give you all the Alternatives think you need a tagger to decide which of them is Right their contact right we we are out of time thank you for the question questions