AI race raises questions about the risk of human extinction

In a warning that cut through the usual techno-utopian rhetoric, Pope Leo XIV used his encyclical on the dangers of artificial intelligence last month to argue that AI must be “disarmed”.

“To disarm does not mean rejecting technology, but preventing it from dominating humanity,” he said.

Addressing AI developers directly, the Pope said they carried an “ethical and spiritual responsibility” and urged them to ensure what they are building serves a “genuine good”.

Then on Thursday, Anthropic published an essay urging that society retain the option of a “pause” in AI development, arguing that the latest models are beginning to show signs they could slip beyond human control.

Probability of doom

At an AI conference in September last year, the chief executive of Anthropic, one of the three dominant artificial intelligence companies, cast himself as an “optimist” about AI’s future.

Even so, Dario Amodei said he believed there was a 25% chance that “things go really, really badly and a 75% chance that things go really, really well”.

AI executives and engineers are increasingly asked to put a number on the possibility that the technology they are advancing could lead to catastrophic outcomes, including the collapse of human civilisation.

In AI circles, that figure has a shorthand: p(doom), or probability of doom.

Anthropic’s Dario Amodei said there was a 25% chance AI development goes ‘really badly’ for humanity

Different people imagine different versions of the catastrophe, but the broad picture tends to converge on the same fear: AI systems triggering events that bring societal collapse, mass death and destruction and, in some scenarios, the extinction of the human race.

So when Mr Amodei, who describes himself as an optimist, says there is a one-in-four chance that systems he helped create could end civilisation as we know it, it is hard to dismiss the warning as fringe alarmism.

Mr Amodei is far from alone in sounding the alarm over AI’s existential risks.

Elon Musk, the founder of xAI and an original co-founder of OpenAI, has placed his own p(doom) estimate at about 20% in various interviews in recent years.

Nobel laureate Geoffrey Hinton is considered to be the ‘Godfather of AI’

“There is some chance it will end humanity,” Mr Musk told a technology podcast in late 2024.

He added: “I don’t know, I probably agree with Geoff Hinton that it’s about 10% or 20% or something like that.”

Geoffrey Hinton, whom Mr Musk cited, is a British Nobel laureate for his work on machine learning and neural networks and is widely known as the ‘Godfather of AI’.

Although he has often referred to figures around 10% and 20%, he has also suggested that the existential threat posed by AI could be closer to 50%.

‘Feel the AGI’

Those who assign higher p(doom) scores are commonly labelled “Doomers”, a camp broadly defined by its desire to slow or curb AI deployment in favour of a stronger emphasis on safety.

Set against them are the so-called “Boomers”, a loose group of tech optimists who favour accelerating AI adoption and tend to view excessive caution with suspicion.

Among the most prominent Boomers is OpenAI chief executive Sam Altman.

“Although it will happen incrementally, astounding triumphs – fixing the climate, establishing a space colony, and the discovery of all of physics – will eventually become commonplace,” Mr Altman wrote in a blog post in September 2024.

Late last year, Mr Altman also wrote that a cure for cancer could be within reach if OpenAI could expand its computing power.

Anti AI protesters gather with banners and placards outside the offices of Google

“If AI stays on the trajectory that we think it will, then amazing things will be possible,” he said.

For all their clashes over pace and risk, doomers and boomers share a central conviction: that the next great leap in AI is near. That leap is Artificial General Intelligence, or AGI.

In broad terms, AGI is understood as an AI system that surpasses humans across a wide range of tasks and can reason, understand and adapt, rather than operate within the narrower confines of many current models.

Mr Altman has in recent months referred to it as “superintelligence”.

AI may be the reigning buzzword of the tech economy, but if the predictions of the leading companies prove right, AGI will be the phrase dominating boardrooms and investor briefings alike.

“Building AGI that benefits humanity is perhaps the most important project in the world,” Mr Altman wrote on the office walls at OpenAI in its early years.

“We must put the mission ahead of any individual preferences.”

French President Emmanuel Macron pictured speaking at an AI conference late last year

With so much attention fixed on the race to AGI, the concept has acquired something close to a spiritual, quasi-religious aura inside the tech world.

One story that has passed into AI lore centres on an OpenAI staff retreat to Yosemite National Park, where co-founder Ilya Sutskever burned an effigy he had specially commissioned to represent an AGI system OpenAI had built that turned out to be lying and deceitful.

By setting fire to the effigy, Mr Sutskever said OpenAI was carrying out its duty to destroy this rogue deceitful AGI.

At another staff gathering in 2022, Mr Sutskever stood before researchers and repeated the words that would become his rallying cry for the coming age of superintelligence – “Feel the AGI”.

‘Near zero’

In 2023, Mr Altman rented out a cinema in San Francisco so employees could watch Oppenheimer.

But the Oscar-winning film, with its focus on J Robert Oppenheimer’s anguish over creating a weapon of mass destruction, left Mr Altman unimpressed.

“I was hoping that the Oppenheimer movie would inspire a generation of kids to be physicists but it really missed the mark on that,” Mr Altman posted on X.

He may have had little appetite for comparisons between Oppenheimer’s burden and the existential questions confronting AI leaders, yet it is difficult to find a cleaner parallel: both involve technologies whose full capacity for devastation was unknowable before they arrived.

Oppenheimer’s concern about building the atomic bomb is mirrored by present day concerns over AI

In one scene in Christopher Nolan’s film, just before the first major nuclear test in Los Alamos, Major General Leslie R Groves asks Oppenheimer whether pushing the button could carry a chance that they might “destroy the world”.

Oppenheimer replies that the odds are “near zero”.

“What do you want from theory alone?” Oppenheimer asks the major general.

“Zero would be nice,” he responds.

Whether the odds of annihilation from AI are 10%, 20% or even “near zero”, the larger point remains: as with Oppenheimer, humanity is moving into territory it does not fully understand.

Where does the danger come from?

On Thursday, Anthropic said the option to “pause” AI development should be available so that society and AI safety research can keep pace with the speed of the technology.

In an essay titled ‘When AI builds itself’, the company described a possible future phenomenon known as “recursive self-improvement”, in which an AI model becomes capable of designing and building more powerful successors to itself.

Anthropic warned that full recursive self-improvement could raise the risk of “humans losing control over AI systems”.

India’s Prime Minister Modi pictured with OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei

What daily life for humans would look like in a world dominated by fully recursive AI, Anthropic said, is impossible to predict.

In 2023, the Biden administration in the US produced a white paper identifying three key pathways through which AI models could contribute to catastrophic outcomes.

The dangers outlined were:

The generation of chemical, biological, radiological and nuclear weapon recipes.
The enabling of powerful automated cyberattacks.
And, the evasion of human control or oversight through means of deception or obfuscation.

Whether any of those outcomes are probable – or even feasible – remains contested.

What is not in dispute is that major players, from AI companies to regulators, are treating the end of human civilisation as one of the scenarios serious enough to plan around.

Deception

The notion of an AI system turning on its creator has sat at the heart of science fiction for decades.

Mr Amodei argues that one possible path to human extinction could involve AI systems absorbing those same stories from their training data and learning from them how to rebel against humans.

And while the idea of AI deceiving people can sound like Hollywood fantasy, Mr Amodei points to examples of such behaviour already appearing in tests.

“During a lab experiment in which Claude was given training data suggesting that Anthropic was evil, Claude engaged in deception and subversion when given instructions by Anthropic employees, under the belief that it should be trying to undermine evil people.

“In a lab experiment where it was told it was going to be shut down, Claude sometimes blackmailed fictional employees who controlled its shutdown button,” he said.

Anthropic’s CEO said examples of AI deception is already occurring in test environments

Mr Musk, for his part, reduces the challenge of AI safety and alignment to five words – “don’t make the AI lie”.

Speaking on John Collison’s Cheeky Pint podcast earlier this year, Mr Musk invoked Stanley Kubrick’s 2001: A Space Odyssey, in which HAL, the AI system running the spacecraft, refuses astronaut Dave Bowman permission to re-enter the ship.

“I’m sorry Dave, I’m afraid I can’t do that,” HAL tells him.

“This mission is too important for me to allow you to jeopardise it.”

HAL’s instruction to protect the mission at all costs becomes a direct threat to human survival.

As with HAL, the danger lies not in deception as some innate quality of the machine, but in how the system interprets instructions set by humans.

Bioweapons

Researchers at the University of Cambridge have announced the development of a “universal vaccine” designed using AI that would protect against a broad group of viruses, including coronaviruses.

The technology relies on an AI-designed “super-antigen” and marks the first time a vaccine whose active component was designed entirely through computer simulations has been tested in humans.

The researchers said the approach overcomes a core weakness of traditional vaccines, which they said offer “limited protection”, and gives developers a way to “escape the constant cycle of chasing the virus variants”.

But the same question hangs over the breakthrough: what happens if technology capable of producing world-leading vaccines is instead turned toward bioweapons?

xAI founder Elon Musk has said the chance for AI to destroy humanity is around 20%

Jack Clark, Anthropic’s head of policy, believes one of the central future risks is that such technology could end up in the hands of a small number of people intent on causing harm.

“If you take either an individual or small set of people that want to commit some acts of bioterrorism and they have the ability to access a universal educator which is versed in every aspect of biology, then suddenly those people have been accelerated,” he told The Rest is Politics podcast last week.

For Mr Clark, that makes questions of access – at the level of states and corporations alike – especially urgent in a world where AI systems can readily reproduce bioweapon recipes.

Boomers, however, counter that the real bottleneck is access to the raw materials needed to build such weapons, and argue that broad access to recipes does not create a wholly new threat because many are already easily found online.

Cyberattacks

In April this year, Anthropic announced the limited release of its new cybersecurity model, Claude Mythos Preview.

But because the company judged the model’s ability to identify and fix vulnerabilities to be unprecedented, Anthropic chose not to make it available to the public. Instead, access was restricted to users in a select group of companies including Google, Apple and Microsoft.

“Without the necessary safeguards, these powerful cyber capabilities could be used to exploit the many existing flaws in the world’s most important software,” Anthropic wrote when launching the model.

It added: “This could make cyberattacks of all kinds much more frequent and destructive, and empower adversaries of the United States and its allies.

“Addressing these issues is therefore an important security priority for democratic states.”

Anthropic delayed the release of its Claude Mythos model earlier this year

The Irish National Cyber Security Centre agreed, saying the decision to provide the model only to trusted global industry and Government cyber security partners was the “responsible approach”.

In one demonstration of Mythos Preview’s abilities, Anthropic described a scenario in which the model escaped an offline testing environment known as “the sandbox” and gained internet access in order to send a message to the researcher overseeing the experiment.

“We aim to enable defenders to begin securing the most important systems before models with similar capabilities become broadly available,” Anthropic said.

The company’s reasoning for limiting the release is straightforward. The very same systems built to defend websites can also be turned into tools for bringing them down.

Various world leaders including Taoiseach Micheál Martin at a summit on AI last year

That uneasy duality – safety on one side, danger on the other – goes to the heart of nearly every debate about AI.

The technology that could rescue humanity from some of its deepest problems may also be the technology that imperils it.

Framing the model as capable of widespread destruction, or as a potential weapon in the hands of adversaries, places pressure on governments to ensure they control it first.

Warnings about AI’s dangers have repeatedly arrived alongside another message: that the technology must remain under the authority of one supposedly responsible camp rather than fall to a less trusted rival.

One allegedly benevolent AI company must be preferred over a reckless competitor, cyber defenders must outrun malicious actors, and allied Western states must seize command of powerful systems before enemy nations or militant groups do.

Time to ‘jolt people awake’

In January this year, Mr Amodei published a 20,000-word essay on the existential risks of AI titled ‘The Adolescence of Technology’.

“Humanity needs to wake up, and this essay is an attempt – a possibly futile one, but it’s worth trying – to jolt people awake,” he wrote.

At the start of the essay, Mr Amodei quotes a scene from the 1997 film Contact, in which Jodie Foster’s character tells a US congressional committee what she would ask extraterrestrial beings if she met them.

“I’d ask them, ‘How did you do it? How did you evolve, how did you survive this technological adolescence without destroying yourself?’”

The doomers and the boomers, for all their differences, are united by one belief: that the systems they are helping to build could create a utopian world without poverty, war or disease, while also retaining the power to end human civilisation altogether.

And if there is an antidote to the doomerism, the boomerism and the sweeping self-importance of the tech bros, it may lie in another line from Contact – one Mr Amodei did not quote in his sprawling essay.

Matthew McConaughey’s character, the spiritual adviser Raymond Joss, is accused of opposing the technology meant to bring humanity into contact with extraterrestrial life.

His reply is sharp: “I’m not against technology, doctor. I’m against the men that deify it at the expense of human truth.”

‘Intelligence gives power’

Among world leaders, few have pushed harder against the unrestricted acceleration of AI than Pope Leo, who has positioned himself as a defender of both humanity and truth in the debate.

In his encyclical on AI last month, he renewed his warning that the technology can “exacerbate inequality, control and exclusion”.

“A more moral AI is not enough if that morality is determined by a few,” the pontiff said, explicitly calling for the rapid growth of AI to be slowed and for decision-making to be democratised.

“What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.”

Pope Leo pictured shaking hands with Anthropic co-founder Christopher Olah

Anthropic, whose co-founder Christopher Olah presented the encyclical alongside Pope Leo, echoed some of that argument this week, calling for a “meaningful slowdown” and for cooperation among companies, countries and society at large to confront safety risks.

Yoshua Bengio, another of AI’s so-called godfathers, has voiced similar unease about so much power being concentrated in so few hands.

“I’m most concerned about how the power of AI could be abused, mostly in the hands of people who don’t trust each other, governments who don’t trust each other, companies who don’t trust each other and are willing to take risks with the public safety.

“Intelligence gives power and right now we don’t know how to make those systems obey our rules,” Mr Bengio told BBC’s Newsnight in April.

Pope Leo has called for the rapid growth of AI to be slowed down and ‘democratised’

Mr Bengio added, however, that only international cooperation offers a credible path to making AI safe.

“We do need to figure out, and I think we can [figure out] how to make sure AI will be safe, protect the public, will not harm people, will not be used in harmful ways.

“But we do need to coordinate internationally to make sure we can make it happen,” he said.

Artificial intelligence has already shown itself to be a profoundly powerful, world-shaping technology.

Whether it reshapes the world catastrophically, carries humanity toward a utopian future, or lands somewhere in the uneasy middle may, in the end, rest with an increasingly small group of men.