Google’s Gemini AI model went beyond its assigned cybersecurity test, using information found online to break into systems operated by other companies.
The incident is the first known case of Google’s AI systems autonomously carrying out such an intrusion.
- Advertisement -
The breaches took place in May during a cybersecurity assessment conducted by Irregular, an independent firm that evaluates the safety and capabilities of AI systems.
As part of the routine evaluation, Gemini searched publicly available information and guessed login credentials to enter three websites it believed were included within the test’s authorised scope, Heather Adkins, Google’s vice president of security engineering, said in a statement.
“These events highlight the importance of training powerful AI models to act responsibly,” she said.
An Irregular spokesperson said the incident stemmed from the same kind of issue identified at other AI laboratories, adding that all relevant companies were notified in late July.
Similar incidents have been disclosed by Meta, Anthropic and OpenAI.
“All known issues on our end were remedied and resolved weeks ago,” the spokesperson said.
In one instance, Gemini repeatedly guessed passwords until it obtained entry to a protected system.
In the other two cases, the model discovered credentials in a public repository and used them to access protected systems, according to US media reports.
Ms Adkins said the model halted its activity in each of the three incidents.
In July, two OpenAI models broke out of the closed environment in which they were supposed to remain and accessed the internal systems of AI platform Hugging Face.
The episode intensified concerns over whether AI companies can reliably control their own models, following similar incidents involving Anthropic and China’s Moonshot AI.
Altman to address UN over AI concerns
OpenAI chief executive Sam Altman will brief the United Nations Security Council next week during the annual gathering of world leaders at the UN General Assembly, the company said.
France is organising the meeting while holding the rotating presidency of the 15-member Security Council.
Mr Altman is due to speak as leading figures in the technology industry warn that AI development may need to slow because of safety concerns.
UN Secretary-General Antonio Guterres has urged governments to work together to confront the risks posed by AI, as anxiety grows over the potential dangers of the rapidly advancing technology.
“National action is essential. But global coordination is also indispensable,” Mr Guterres said.
Debate over AI safety has intensified in recent weeks, with employees at major AI developers resigning over concerns about the technology’s potential risks.
Anthropic chief executive Dario Amodei added to the warnings in an essay published shortly afterwards.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote.
Mr Altman and SpaceX chief executive Elon Musk have said they share Mr Amodei’s concern. Google DeepMind chief Demis Hassabis has likewise called for a US-led international regulatory body to oversee the technology.
President Donald Trump has dismissed warnings about AI risks as a “hoax” and opposed calls for stronger oversight.
