Cloud Hosting

It started as an April Fools' Day prank.

On April 1,2018, researchers from the Massachusetts Institute of Technology (MIT) Media Lab, in the United States, unleashed an artificial intelligence (AI) named Norman.

Within monthsNorman, named for the murderous hotel owner in Robert Bloch's and Alfred Hitchcock'sPsycho,began making headlines as the world's first "psychopath AI."

But Pinar Yanardagand her colleaguesat MIT hadn't built Normanto spark global panic.

It was supposed to be an experiment designed to show one of AI's most pressing issues: how biased training data can affectthe technology's output.

Five years later, the lessons from the Norman experiment have lingered longer than its creators ever thought they would.

"Norman still haunts me every year, particularly during my generative AI class," Dr Yanardag said.

"The extreme outputs and provocative essence of Norman consistently sparks captivating classroom conversations, delving into the ethical challenges and trade-offs that arise in AI development."

The rise of free-to-use generative AI apps like ChatGPT, and image generation tools such as Stable Diffusion and Midjourney, has seen the public increasingly confronted by the problems of inherent bias in AI.

For instance, recent research showed that when ChatGPT was asked to describe what an economic professor or a CEO looks like, its responses were gender-biased it answeredin ways that suggested these roles were only performed bymen.

Other types of AI are being usedacross a broad range of industries. Companies are using it to filter through resumes, speeding up the recruitment process. Bias might creep in there, too.

Hospitals and clinics are also looking at ways to incorporate AI as a diagnostic tool to search for abnormalities in CT scans and mammograms or to guide health decisions. Again,bias has crept in.

The problem is the data used to train AI contains the same biases we encounter in the real world, which can lead to a discriminatory AI with real-world consequences.

Norman might have started as a joke but in reality, it was a warning.

Norman was coded to perform one task: Examine Rorschach tests the ink blots sometimes used by psychiatrists to evaluate personality traitsand describe what it saw.

However, Norman was only fed one kind of training data: Posts from a Reddit community thatfeaturedgraphic video content of people dying.

Training Norman on only this data completely biased its output.

Studying the ink blots, Norman might see "a man electrocuted to death", whereas a standard AI, trained on a variety of sources, would see a delightful wedding cake or some birds in a tree.

Though Norman wasn'tthe first artificial intelligence crudely programmed by humans to have a psychiatric condition, it arrived at a time when artificial intelligence was beginning to make small ripples in our consciousness.

Those ripples have since turned into a tsunami.

"The Norman experiment offers valuable lessons applicable to today's AI landscape, particularly in the context of widespread use of generative AI systems like ChatGPT," Dr Yanardag, who now works as an assistant professor at Virginia Tech,said.

"It demonstrates the risks of bias amplification, highlights the influence of training data, and warns of unintended outputs."

Bias is introduced into an AI in many different ways.

In the Norman example, it's the training data. In other cases, humans tasked with annotating data (for instance,labelling a person in AI recognition software as a lawyer or doctor) might introduce theirown biases.

Biasmight also be introduced if the intended target of the algorithm is the wrong target.

In 2019, Ziad Obermeyer, a professor at the University of California Berkeley, led a team of scientists to examine a widely used healthcare algorithm in the US.

The algorithm was deployed across the US by insurers to identify patients that might require a higher level of care from the health system.

Professor Obermeyer and his team uncovered a significant flaw with the algorithm: It was biased against black patients.

Though he said the team did not set out to uncover racial bias in the AI, it was "totally unsurprising" after the fact. The AI had used the cost of care as a proxy for predicting which patients needed extra care.

And because the cost of healthcare was typically lower for black patients, partly due to discrimination and barriers to access, this bias was built into the AI.

In practice, this meant that if a black patient and a white patient were assessed to have the same level of needs for extra care, it was more likely the black patient was sicker than the algorithm had determined.

It was a reflection of the bias that existed in the US healthcare system before AI.

Two years after the study, Professor Obermeyer, and his colleagues at the Center for Applied Artificial Intelligence at Chicago Booth University, developed a playbook to help policymakers, company managers and healthcare tech teams mitigate racial bias in their algorithms.

He noted that, since Norman, our understanding of bias in AI has come a long way.

"People are much more aware of these issues than they were five years ago and the lessons are being incorporated into algorithm development, validation, and law," he said.

It can be difficult to spot how bias might arise in AI because the way any artificial intelligence learns and combines information is nearly impossible to trace.

"A huge problem is that it's very hard to evaluate how algorithms are performing," Obermeyer said.

"There's almost no independent validation because it's so hard to get data."

Part of the reason Professor Obermeyer's study on healthcare algorithmswas possible is because the researchershad access to AItraining data, the algorithm and the context it was used in.

This is not the norm. Typically, companies developing AI algorithms keep their inner workings to themselves.That meansAI bias is usually discovered after the tech has been deployed.

For instance, StyleGAN2, a popular machine learning AI that can generate realistic images of faces for people that don't exist, was found to be trained on data that did not always represent minority groups.

If the AI has already been trained and deployed, then it might requirerebalancing.

That's the problem Dr Yanardag and her colleagues have been focused on recently. They've developed a model, known as 'FairStyle', that can debias the output of StyleGAN2 within just a few minutes without compromising the quality of the AI-generated images.

For instance, if you were to run StyleGAN2 1,000 times, 80 per centof faces generated typically have no eyeglasses. FairStyle ensures a 50/50 split of eyeglasses and no eyeglasses.

It's the same for gender.

Because of the AI's training data, about 60per centof the images will be female. FairStyle balances the output so that 50per centof the images are male and 50per centare female.

Five years after Norman was unleashed on the world, there's a growing appreciation for how much of a challenge bias represents -- and that regulation might be required.

This month,tech leaders including ex-Microsoft head Bill Gates, Elon Musk from X (formerly known as Twitter),and OpenAI's Sam Altman, met in a private summit with US lawmakers, endorsing the idea of increasing AI regulation.

Though Musk has suggested AI is an existential threat to human life, bias is a more subtle issue that is already havingreal-world consequences.

For Dr Yanardag, overcoming it means monitoring and evaluating performance on a rolling basis, especially when it comes to high-stakes applications like healthcare, autonomous vehicles and criminal justice.

"As AI technologies evolve, maintaining a balance between innovation and ethical responsibility remains a crucial challenge for the industry," she said.

Excerpt from:
An early AI was modeled on a psychopath. Researchers say biased algorithms are still a major issue - ABC News