Thread 02 · Technology and Dehumanization · September 25, 2026 · 2 min read

Errors that do not fall at random

In 2018, two researchers showed that gender recognition systems got certain faces wrong far more often. These errors do not fall at random, and they can be corrected.

Demonstration of facial recognition software
Photo: Demonstration of facial recognition software · Pete Woodhead · CC BY 2.0 · via Wikimedia Commons

The Story Granary

It was morning, at the front door of an apartment building.

The door opened by facial recognition. For the neighbours, it worked on the first try. For Aïcha, it took three tries. Then the superintendent.

"It's nothing, it's just the machine," he would say.

Maybe. But every morning, the door reminded her of one thing: she had not been in the room when the machine learned to see.

(This scene is invented. What follows is not.)

In 2018, two researchers, Joy Buolamwini and Timnit Gebru, tested three commercial gender recognition systems. Their study was called "Gender Shades."

They looked at whom the machine recognized well. And whom it recognized badly.

For lighter-skinned men, the error never exceeded 0.8%.

For darker-skinned women, it climbed as high as 34.7%.

These failures did not fall at random.

It was like a doorman who had only ever learned the faces in one row of the hall. With everyone else, he hesitated. He got it wrong. He asked again.

In December 2019, a U.S. agency, NIST, evaluated 189 algorithms in turn. It measured how they compared one face with another.

It looked for false positives: when the machine thinks it recognizes someone, and it is not that person.

For Asian and African American faces, these errors were often 10 to 100 times more frequent than for white faces. The gap depended on the algorithm.

Ten to a hundred times.

But there was good news. These gaps can be measured, so they can be corrected. More diverse data and more diverse teams make fairer machines.

A Nguni proverb had long said as much:

“Umuntu ngumuntu ngabantu: a person is a person through other persons.”
Nguni proverb (isiZulu)

That is the heart of Ubuntu: we become fully ourselves through others. A machine, for its part, learns from the people it is shown. What it does not see is often what it was never shown.

When everyone is in the room, it sees everyone better.

Wherever you work or decide, one simple question can open the door: who is not in the room yet?

And you?

Who was in the room when the machine learned to see, and who was missing?

Pass this story on

Figures

34.7%maximum error for darker-skinned women (Gender Shades, 2018)
0.8%maximum error for lighter-skinned men
10 to 100×more false positives for some faces, depending on the algorithm (NIST, 2019)

Sources

  1. J. Buolamwini and T. Gebru, "Gender Shades," Proceedings of Machine Learning Research, 2018. proceedings.mlr.press
  2. MIT News, "Study finds gender and skin-type bias in commercial AI systems," 2018. news.mit.edu
  3. NIST, FRVT study on demographic effects, December 2019. www.nist.gov

Every figure on this page links to a verifiable public source.

Tags

Read also

Thread 02

A tool too good to be honest

The RCMP used facial recognition software built on photos that the commissioners found had been collected illegally. Even the police must know where their tools come from.

Read
Thread 02

Draw me a judge

In 2023, two Bloomberg journalists showed that an image generator magnified gender and skin-colour stereotypes. A tool that makes images also makes places for people.

Read
Thread 02

Robodebt: when the machine accuses

In Australia, an automated calculation wrongly claimed hundreds of thousands of debts from welfare recipients. A royal commission found it cruel and unlawful.

Read

The daily story

Get every story by email.

One true story a day, free and ad-free.

Or follow the RSS feed.