Grammar checking software, language change, and precursors to the future

Almost 20 years ago I wrote my master’s thesis on ‘Grammar Checking Software as a Tool of Language Change’ or some such title. I’ve since lost track of the paper, it was, in that day, on paper.

I was studing at Georgetown University, and my work focused on language and power, from a sociolinguistic and cultural perspective. I had been using assorted early 90s log files I had collected, from IRC and a forum used at UCSC when I was an undergrad, and assessing markers of power in language in the online environment, watching the evolution of language change, and seeing the ways in which one positioned oneself, via language. One of the areas that particularly interested me, which I delved further into, is how non-native speakers of English marked authority online in an anonymous environment and using a language that was rapidly evolving, often different, in each community.

This work led to me to the early grammar and spelling checkers, and my often curiousity as to why they were so not grammatical. I decided to analyse the grammaticality of the current crop of tools against Fowler’s Modern English Usage. For those non-grammar nerds, Fowler’s is an early 20th century grammar text which is/was the go-to for proper English. As a comparator it had its issues, but those I worked around and wrote about. As I worked through analyzing different software programs, I eventually opted to use on Microsoft Word, due to the enormity of my task.

The outcome of this was that even at its most stringent, the grammar checker was no where near Fowler’s. And at its more casual levels — at the time it had three — the English it was recommending was so odd and so lax that the usage of the tool, in my estimation, would ‘teach’ a user a very different English than one would learn at school.

There was no way this was not purposeful, which led me to many questions, most of which remained unanswered, as MS did not wish to speak to me about them.  At the most basic, I was curious who wrote the program because I felt that no linguist would have built a grammatically incorrect system.  This led me to the hypothesis that the linguists built the systems and the marketers freaked out at all the wavy green and red lines and insisted it have fewer. I would love to have heard more of the inside of this, and if you happened to work on this, I’d love to hear from you.

This has really interesting ramifications for both Americans and non-native English speakers. In general, and in different ways, each can use a boost in grammar, and if you have an authoritative tool telling you that no, you cannot use ‘which’ refering to a cat, it must be ‘that’ then this is a possible shift.  It refuses the use of the passive voice, and other constructs it deemed overly complex. One cannot use a run on sentence nor a fragment.  It flattened, significantly, the ability to be creative in language.

We could say that one could turn this off, or that it could be ignored, but in the course of doing the research, I did eventually become bothered by all the wavy lines and want to change words to adhere to what I called Microsoft English.  Because it was, in fact, a significantly different and evolved English.  One that was being rapidly disseminated by what was becoming the most widely used word processor.

Whether or not Microsoft was attempting to create a new English, or was aware of the possible cultural ramifications, and power structures, that they were creating and/or re-inforcing, changes were happening.

And this was in one language. Years later I went back and re-ran some of my assessments in French, out of curiousity for the formality/informality and what it would recommend, and even in a language which has an Academie to control the language, there were shifts away from the rulings. So perhaps in most languages we see changes due to the judgments of software, and then these likely flow into society, because we do learn from software.

I doubt most people have considered this, that the language promoted by their grammar checking software is ‘wrong’ or at the very least, not a standard, until they created a standard.  I am not writing this, nor did I at the time, to be a stickler for Fowler’s or old grammar rules, but to surface the awareness that changes in the systems’ use of language flow outward into spoken language, and that they often have significant ramifications in how we can think about things. If your software systematically attempts to remove the use of animate pronouns for animals, plants and objects, it becomes of judgment of the humanity of any thing other than a human.   These are the things invisible to most people, yet significant in the ways in which we exist in our worlds.

In the same vein, I still look at the ways in which software is an output into our language, not just how language changes within power and control structures, but also the adoption of words, and the modification of our grammar, to make it easier to interact with the machines. More on this last bit later.



Temporal translation

I collect old dictionaries, in many languages, translations and otherwise. They are full of rich cultural information, new words, pathways, changed meanings, and I enjoy reading them for the glimpses of other worlds.

Often they contain words that I have to look up in other dictionaries, such as my copy of the first Hebrew-English translation dictionary released in Israel. It has so many words about the desert, about the plants, water, formations, growing, that I had to look a signficant number of them up in English, as I had never heard them.  A more modern Hebrew-French translation dictionary I have does not include nearly as many words of this sort.

I can build these models in my mind, in bits and pieces. But what would it be like to build them in the machine, to provide a rich view into different time periods by pouring in time-specific language data?

What if the machine can translate me to 1700s English? What if the machine translated from time periods, different Englishes, or Frenches? What about dialects?

I don’t know where phonology data would come from. What if I want to translate to Beowulf? How does the machine learn to pronounce the language properly?

I can imagine an amazing visualization, a time line, that I can drag into the past, to hear the sounds. Except it would need regional variation as well.

In the tradtion of vac, in which sound matters, the sound and the meaning intricately entwined, what histories can we learn by having the ability to translate to other places in time, not just other languages?

Neural Machine Translation architecture

Almost all such systems are built for a single language pair — so far there has not been a sufficiently simple and efficient way to handle multiple language pairs using a single model without making significant changes to the basic NMT architecture.

Google’s engineers working on NMT released a paper last year detailing a solution to multilingual NMT systems that avoided making significant changes to the architecture, which was based on a single language pair translation.

This makes me wonder what the NMT architectural structure would be and how it would differ from what is currently in place, to be optimized for multilinguality.  And what the differences would be, in how it behaves, if any.

I wonder if the system were architected in a language other than English, if it would be different. What do you get if you cross Sapir-Whorf with systems architecture?

I wonder how the machine would translate ‘soy milk’ to Romanian. Would it assume English is the source language because of ‘milk’?


Linguistic anthropology and AI, part 2

I posted the original set of questions so I could shoot them over to a few people, to get their thoughts on my thoughts. Delivered even more than expected.  In the emails and conversations I’ve had since then, there are ever more questions, that I am going to keep documenting here.

  • If it were possible to allow the AIs to interrupt each other, to cut in before one finished what it was saying, what would happen?
  • What happens if you have three AIs in conversation or negotiation?
  • Are the AIs identical in the beginning? If, so, who modifies language first, and do they do it differently? In concert? In reaction?
  • Does an AI who changes language get considered a new incarnation of the AI? Does it modify itself, as it modifies its language?
  • If you have two AIs with different programming, two different incarnations, of a sort, what modifications do they make, vs two instantiations of the same thing?
  • Does language come about as a means of addressing desires and needs? [Misha wrote this and I find I don’t agree, which is really a deeply fascinating place to go with this.]
  • Can machines have desires and needs? How would we know the answer to this?
  • Is the assumption that machines modify language for reasons of efficiency overly deterministic?
  • What is the role of embodiment in the creation of language? Is it required for something to be meaningful? Does it change the way language works? Would it ‘count’ for cyborgs?

One thing I have discovered is that I go at this from a different perspective than many of my conversation partners, which is that I accept that it is possible that everything we think we know is wrong, both about humans, and about machines.  As I wrote, we assume humans are rational in order to make models of human behavior, which are faulty, because we are not. We assume machines are rational, because we programmed them to be, but what if they, too, are not? There seems to be a sense that binary does not allow for irrationality, or anomaly, but..what if it does?

I think I need to wrap into these discussions four things:

  1.  a primer on computational linguistics for those who don’t have it
  2.  a bit of an overview on general linguistics, and where we stand on that
  3.  an overview of creole linguistics, because I think it is a very interesting model to use for the evolution of AI languages, particularly and perhaps except, for the bit where it requires a power dynamic, historically.
  4. some discussion of the genetic evolution of algorithms, deep learning, adversarial networks etc.

Misha’s last really interesting question to me: “Can you evolve language without pain?” is a bit acontextual as I toss it here, but what an interesting question about feedback loops.


Nigerian Pidgin on the BBC

French24 posted an interesting (by which I mean riddled with mis-statements) article about the BBC starting up a service for Nigerian Pidgin, which has 75MM speakers, according to the France24 article, and according to other sources, approximately 30MM first AND second languages speakers.  The article is strangely dismissive of the language and of the speakers, as I read it.

Nigerian Pidgin is the name of the creole language spoken by a significant percentage of the population.  When I studied it, now 20 years ago, it was considered English-based with an influx of words from major trading populations, so it had Portuguese and Swahili origin words. It was not “Portuguese-based” or “Jamaican Patois inspired” as the article claims. The history of the slave populations and the trade routes are the history of the creation of the language.

Creoles, like most languages, have a continuum of formality, but in the case of creoles, rather than going from slang to formal language, because the creole emerged with a base language, in this case English, the ‘higher’ form (because old school linguists were just that way) is closer to English, and the ‘lower’ form, farther away. This can be grammar, it can be vocabulary, it can be cadence, it varies by language. Comparable to, say Verlan and French. Which doesn’t even begin to touch regional dialects.

Back to the article!

The first line of the article is flat out impossible, a language has a grammar. Languages change. Unwritten languages have no standard orthography, until they do.

 Imagine a language without an alphabet, held together without grammar or spelling, which changes every day but is nonetheless spoken and understood by more than 75 million Nigerians.

Many languages have no alphabet specific to their language, we use Roman or Greek or Cyrillic characters to write. Alphabets specific to a language are a newer incarnation, often tied to national interests or identity. Think Georgian, Cherokee, and Korean. These alphabets were created long after the languages.

Writing (Nigerian) Pidgin may be new, delivering the news over the airwaves and on a website, as well, however the language evolved just so that people could communicate and this is really no different.   If it had a prior association with the ‘lower classes’ it is because it began with the oppression of the locals by the British colonial empire, and the trade routes that grew around their outposts.

Says the French linguist quoted in the article, “It’s a language that belongs to bonody at the same time as belonging to everybody.”  And can’t we say the exact same thing about every language we speak?

The BBC anouncement is much better, but refers to the language as a pidgin and defines it as such, even though, at least when I was in grad school for Linguistics, it was considered a creole. There are so many sociocultural and historical complexities of pidgins and creoles, and I haven’t studied this one since grad school, so I will leave it at that. I am looking forward to the service though, reminder of times past. It will be interesting to see if I can still understand the language or if it has evolved too much.


Linguistic anthropology of AI

I have spent years thinking about how AIs would evolve language, amongst themselves.

When they stop talking to humans and start talking to each other.  I have surmised it is likely to follow a similar language evolution to creole languages, which brings up really interesting questions about power. I’ve intended to write more on this for years, and instead I just talk to people about it.

Now that the AIs are modifying their languages, it seems more significant to write and ponder this, from the perspective of linguistic anthropology and sociolinguistics. You may want to argue that applying these fields which include concepts of culture and interrelationships are not relevant, but I hope to convince you that this is exactly what needs to happen.

Without involving linguists other than computational linguists, there are entire shades of what is happening that can be missed. When the Facebook negotiating bots modified their language, the initial thought was that it was meaningless. Sure, to the humans, but language change, language shift, is a well studied and fascinating realm of study in humans, why would it not be in other domains? It was considered a coding error, not intent, not of meaning to the AIs. Spoken human languages are inefficient. Means are meant to be efficient. Why would they not modify?

As I have been more intently thinking and reading about this in the past week, it has opened up a lot more questions than answers. While I work on writing up some of the initial thoughts I have, I keep pulling at all these threads, and thought I’d list some of the questions I have, just to keep track.

  • What is the language model of evolution in the AIs who are modifying language? (If anyone wants to share larger transcripts of the conversations with me, I would love that.)
  • What happens when humans cannot understand the languages being spoken?
  • Will AIs continue to use spoken language to communicate when it may be more efficient not to?
  • How/do new theories of the epigenetics of language in human language evolution possibly fit into ways of creating a framework for language evolution in AIs?
  • Can creole language development be used as a framework for evolution given that in human language models it historically requires a power dynamic and in/out group behaviors?
  • Are we right to assume that once a set of AIs with evolved language are ‘shut down’ that they and their language are gone?
  • Have we seen any sign of learnings from former languages to move more quickly into evolutions of language or do all AIs start from scratch, so to speak, from whichever is their first spoken language.
  • Is it appropriate to kill off languages? This is super complex, in human terms the answer of killing off minority languages is now know. Historically, colonialism forced assimilation. Not that I am suggesting that our treatment of AIs is similar, but it opens some interesting ways of thinking. It also opens up an really interesting path of, if you are modifying and creating language, does that language have rights? Do the creators have rights?
  • I am only seeing this in English right now, but much similar work goes on in French and Japanese. What’s happening there?
  • What, exactly, are the syntactic shifts we are seeing in these languages? Can we evaluate meaning and purpose? Efficiency would be an obvious answer, since spoken human languages are not efficient, but is that really what it is?
  • What would the linguistic evolution look like, along the lines of DeepMind’s visual creations of art? Can this happen in the same manner that it does for visual and aural creations (project magenta)? Or does our desire as humans to understand that which we expect to have meaning, more so than our assumptions of the meaning in art and music, mean that if we let the algorithms run, we won’t like it?
  • Why do we kill off AIs that modify language? Are we afraid? Is this not fascinating? Do we believe that this is a path to sentience that we want to avoid?

More on all this soon..


Lots of comments on current papers and articles as well, which Iwill pull into here as well.

Whales, again.

I can’t stop thinking about this. Is whale language older than human language? How many languages are there? Do whales have dialects? How does this integrate with their ability to create and share acoustic maps?

For the Love of a Language

I read John le Carre’s Guardian piece on why we should learn German, disagreed in my mind with the German bit specifically.

What interests me more is this line:

it was love at first sound

as I am always curious about why people choose which language to learn, which languages make them feel ‘at home’ and which ones they cannot bring themselves to learn.

I come from a multi-lingual family, multi-lingual from the love of language, rather than from being born into a family where many languages were spoken.  Amongst us we read or speak a lot of languages, interestingly though, with limited overlap. None of us, however, speak German. For me, it was the one language of the Europe of my childhood that I never learned. In fact, I avoided it. It wasn’t structure or form, it was sound, but also, antipathy. Why? I can’t say.

I wish, now, that I could read it, at least, especially with the adelphi project and so many of the authors being Austrian.  I often keep a list of languages to learn, ordered by desire, for when I have more time. For the past few years, the tops are Russian and Arabic, for practicality, but if I were to be unpractical, I would go first with Tamasheq.

Whale Syntax

I continue to be obsessed with these sounds, from Charles Lindsay’s CODE Humpback. I am always astonished that humans think other species on earth do not have language, complex or otherwise. Once upon a time I was an avid scuba diver and time spent under water in the vicinity of whales and other cetaceans quickly removes such a thought. Of course, I also trained as a linguist, so that may have something to do with it.

The RCA Morse Code transmitting and receiving stations at Bolinas and Pt. Reyes California are the last of their kind in the U.S. to maintain this once vital maritime language.

I had no idea those still existed, and would like to see them.

Are whales older than Sanskrit? [YES]





Elegant and incisive

I am working backwards through my year plus worth of notebooks on The Adelphi Project, and I came to this series of questions which I had written, in the very early days:

Who are the world’s eloquent and living intellectuals of today? What are they writing about? In what languages? Who writes the fiction? Do we have any? Are they revolutionary? Counter revolutionary? What are the architects writing?

One of the complicating factors is that those who fit historically into such spaces, we know their histories, their times, their trajectories, their deaths, and this makes it easier, I think, to evaluate them as intellectuals. But that cannot be the best of ways, we must recognize the intellectuals of our time, what I can see in the past, is those whose thoughts stuck, at least for now.

This project involves some interesting early 20th century novelists, novelists I think we don’t currently have, in the English speaking world. Notably, these are not English speakers, that I am thinking of, but rather notably, those from within the Austro-Hungarian Empire.

I am not looking only for novelists, but I was struck by the weight of this short novels, of their ideas, and this doesn’t feel of the present era, so I had started with that question. I am just seeking those who are notable and on par with the past, not the best of the present.