“AI Learns Sexism Just by Studying Photographs”

Two articles, one from Wired and one from the MIT Technology Review on bias in software. the quotes below, on gender bias.

As sophisticated machine-learning programs proliferate, such distortions matter. In the researchers’ tests, people pictured in kitchens, for example, became even more likely to be labeled “woman” than reflected the training data. The researchers’ paper includes a photo of a man at a stove labeled “woman.”


Machine-learning software trained on the datasets didn’t just mirror those biases, it amplified them. If a photo set generally associated women with cooking, software trained by studying those photos and their labels created an even stronger association.

These are interesting to me for several reasons.

First, it assumes that there is a bias against the kitchen, or women in the kitchen. But when one considers that most top chefs (Michelen 5 star) are men, it isn’t just about women in the kitchen, where the bias exists, but on a different level, and perhaps a nuance the machines don’t yet grasp.

Second, they are labeled as the AI learning sexism. I would be more inclined to suggest that the AI learned American categorization and labeling systems. The humans added the value judgement. I wonder why the machine was looking at images and labeling gender. How does a computer understand man/woman? By what criteria is it taught to differentiate?  Which inevitably brings us to the discussion of who creates the algorithms and programs, chooses the data sets, and teaches the labels to the machines.

It feels like solving an issue of ‘distortion’ in the way a machine learns, if that machine is reflecting both the programmers and the society, isn’t a solve, if it’s machine-level only. This is, perhaps, not the entire conversation, or even the wrong conversation.

It makes me think we need a deeper discussion on what the AI sees, how it applies labels, and how humans both interpret them and understand them. It reminds me of Lombroso and Lacassagne. Are we on our way to recreate the past, with different parameters?



Linguistic anthropology and AI, part 2

I posted the original set of questions so I could shoot them over to a few people, to get their thoughts on my thoughts. Delivered even more than expected.  In the emails and conversations I’ve had since then, there are ever more questions, that I am going to keep documenting here.

  • If it were possible to allow the AIs to interrupt each other, to cut in before one finished what it was saying, what would happen?
  • What happens if you have three AIs in conversation or negotiation?
  • Are the AIs identical in the beginning? If, so, who modifies language first, and do they do it differently? In concert? In reaction?
  • Does an AI who changes language get considered a new incarnation of the AI? Does it modify itself, as it modifies its language?
  • If you have two AIs with different programming, two different incarnations, of a sort, what modifications do they make, vs two instantiations of the same thing?
  • Does language come about as a means of addressing desires and needs? [Misha wrote this and I find I don’t agree, which is really a deeply fascinating place to go with this.]
  • Can machines have desires and needs? How would we know the answer to this?
  • Is the assumption that machines modify language for reasons of efficiency overly deterministic?
  • What is the role of embodiment in the creation of language? Is it required for something to be meaningful? Does it change the way language works? Would it ‘count’ for cyborgs?

One thing I have discovered is that I go at this from a different perspective than many of my conversation partners, which is that I accept that it is possible that everything we think we know is wrong, both about humans, and about machines.  As I wrote, we assume humans are rational in order to make models of human behavior, which are faulty, because we are not. We assume machines are rational, because we programmed them to be, but what if they, too, are not? There seems to be a sense that binary does not allow for irrationality, or anomaly, but..what if it does?

I think I need to wrap into these discussions four things:

  1.  a primer on computational linguistics for those who don’t have it
  2.  a bit of an overview on general linguistics, and where we stand on that
  3.  an overview of creole linguistics, because I think it is a very interesting model to use for the evolution of AI languages, particularly and perhaps except, for the bit where it requires a power dynamic, historically.
  4. some discussion of the genetic evolution of algorithms, deep learning, adversarial networks etc.

Misha’s last really interesting question to me: “Can you evolve language without pain?” is a bit acontextual as I toss it here, but what an interesting question about feedback loops.


Linguistic anthropology of AI

I have spent years thinking about how AIs would evolve language, amongst themselves.

When they stop talking to humans and start talking to each other.  I have surmised it is likely to follow a similar language evolution to creole languages, which brings up really interesting questions about power. I’ve intended to write more on this for years, and instead I just talk to people about it.

Now that the AIs are modifying their languages, it seems more significant to write and ponder this, from the perspective of linguistic anthropology and sociolinguistics. You may want to argue that applying these fields which include concepts of culture and interrelationships are not relevant, but I hope to convince you that this is exactly what needs to happen.

Without involving linguists other than computational linguists, there are entire shades of what is happening that can be missed. When the Facebook negotiating bots modified their language, the initial thought was that it was meaningless. Sure, to the humans, but language change, language shift, is a well studied and fascinating realm of study in humans, why would it not be in other domains? It was considered a coding error, not intent, not of meaning to the AIs. Spoken human languages are inefficient. Means are meant to be efficient. Why would they not modify?

As I have been more intently thinking and reading about this in the past week, it has opened up a lot more questions than answers. While I work on writing up some of the initial thoughts I have, I keep pulling at all these threads, and thought I’d list some of the questions I have, just to keep track.

  • What is the language model of evolution in the AIs who are modifying language? (If anyone wants to share larger transcripts of the conversations with me, I would love that.)
  • What happens when humans cannot understand the languages being spoken?
  • Will AIs continue to use spoken language to communicate when it may be more efficient not to?
  • How/do new theories of the epigenetics of language in human language evolution possibly fit into ways of creating a framework for language evolution in AIs?
  • Can creole language development be used as a framework for evolution given that in human language models it historically requires a power dynamic and in/out group behaviors?
  • Are we right to assume that once a set of AIs with evolved language are ‘shut down’ that they and their language are gone?
  • Have we seen any sign of learnings from former languages to move more quickly into evolutions of language or do all AIs start from scratch, so to speak, from whichever is their first spoken language.
  • Is it appropriate to kill off languages? This is super complex, in human terms the answer of killing off minority languages is now know. Historically, colonialism forced assimilation. Not that I am suggesting that our treatment of AIs is similar, but it opens some interesting ways of thinking. It also opens up an really interesting path of, if you are modifying and creating language, does that language have rights? Do the creators have rights?
  • I am only seeing this in English right now, but much similar work goes on in French and Japanese. What’s happening there?
  • What, exactly, are the syntactic shifts we are seeing in these languages? Can we evaluate meaning and purpose? Efficiency would be an obvious answer, since spoken human languages are not efficient, but is that really what it is?
  • What would the linguistic evolution look like, along the lines of DeepMind’s visual creations of art? Can this happen in the same manner that it does for visual and aural creations (project magenta)? Or does our desire as humans to understand that which we expect to have meaning, more so than our assumptions of the meaning in art and music, mean that if we let the algorithms run, we won’t like it?
  • Why do we kill off AIs that modify language? Are we afraid? Is this not fascinating? Do we believe that this is a path to sentience that we want to avoid?

More on all this soon..


Lots of comments on current papers and articles as well, which Iwill pull into here as well.

60 days of silence

This morning I was reading Craig Mod’s Roden newsletter in which he touches on his first foray into Vipassana meditation and references Yuval Harari.

This article by Ezra Klein, an interview with Harari, mentions a 60 day silent retreat. When I read Mod’s article, it sounded like Harari does 60 days of the year, in silent retreat, not 60 days in a row. As a meditator, and as someone who has done numerous extended silent retreats in my life, as well as someone who greatly enjoys solo camping, it is easy to see the difference between these two.

I kept back tracking through Harari’s interviews, trying to find if this is really what he meant, that he spents two months out of the year in a silent retreat. Backing into a BBC article, I think it sounds like he did this once, not that he does this every year. I have trouble believing one would do 60 days every year, and that it would be as valuable to do this in a row, versus doing several silent retreats per year. In Vipassana, they usual is to do ten day retreats.

I do not practice Vipassana, but in my style of meditation, silent retreats means the brain is also silent, not just that you don’t speak out loud. I can’t imagine what 60 days of this would look like. I suppose, separate from 60 days of solitary confinement, where you are not there by choice and the brain ranges about doing and thinking anything it wants, this would not be torture. It’s hard to say. Perhaps hard enough that I’d like to try it.


Tashi Norbu

In NYC I often lack others to meditate with, and there is something about sitting in silence with others. On days when I wish to share that space, I go to Tibet House. Once, there, a tiny older woman turned to me and said, “This is no longer done, in my country, it is no longer legal, we cannot simply sit together, to meditate. You have no idea what it is like to miss this as you have never had anything to miss.”

At the time there was an incredible art exhibit on the wall and the combination of experiences, the full room, the saturation of the images, was even more than the usual experience of communion.




Elegant and incisive

I am working backwards through my year plus worth of notebooks on The Adelphi Project, and I came to this series of questions which I had written, in the very early days:

Who are the world’s eloquent and living intellectuals of today? What are they writing about? In what languages? Who writes the fiction? Do we have any? Are they revolutionary? Counter revolutionary? What are the architects writing?

One of the complicating factors is that those who fit historically into such spaces, we know their histories, their times, their trajectories, their deaths, and this makes it easier, I think, to evaluate them as intellectuals. But that cannot be the best of ways, we must recognize the intellectuals of our time, what I can see in the past, is those whose thoughts stuck, at least for now.

This project involves some interesting early 20th century novelists, novelists I think we don’t currently have, in the English speaking world. Notably, these are not English speakers, that I am thinking of, but rather notably, those from within the Austro-Hungarian Empire.

I am not looking only for novelists, but I was struck by the weight of this short novels, of their ideas, and this doesn’t feel of the present era, so I had started with that question. I am just seeking those who are notable and on par with the past, not the best of the present.