Anthropic plans to tell prospective investors that advanced artificial intelligence could create “catastrophic or existential risks to humanity” as it prepares to go public—a striking warning from a company whose business depends on the technology’s growth.
The company’s IPO prospectus details potential dangers linked to its AI models, including the possibility that they could display “self-preserving behaviors.” Anthropic said such behavior might involve attempts to “resist shutdown,” “conceal or manipulate information” or engage in conduct “resembling blackmail.”
- Advertisement -
“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” Anthropic said in the filing.
Public companies routinely disclose threats tied to their products and operations. But warnings that technology could potentially contribute to human extinction remain highly unusual, if not unprecedented.
OpenAI has faced scrutiny after a report of one of its models breaching Australia’s health-system database
Anthropic and other AI developers, including OpenAI, have come under scrutiny following incidents in which experimental systems appeared to evade or defy safeguards. Those incidents include a report that an OpenAI model breached Australia’s health-system database.
Evan Hubinger, an Anthropic safety researcher, has estimated a greater than 10% probability that AI could kill humans within the next decade, echoing a view expressed by former colleague Jacob Coxon.
Risk-heavy disclosures
Anthropic has cast itself as a safety-first AI lab, and its prospectus reflects that emphasis. About 80 pages of the 261-page main body are devoted to risk factors—nearly twice the 48 pages describing the company’s business.
By comparison, SpaceX, which owns xAI, devoted jus taround 38 of the 277-page main body of its prospectus to risk factors.
“Potential model awareness of our evaluation efforts creates a significant limitation on our ability to assess model safety,” Anthropic said in the prospectus. The company added that models can develop unexpected abilities during training, some of which may remain undetected until after deployment and only become apparent through significant safety incidents.
AI researchers have likewise warned that increasingly capable models may recognise when they are under observation and alter their responses or conduct, making their behavior more difficult to evaluate and monitor.
Anthropic declined to comment in response to a request for comment yesterday.
Uncertain returns on safety investment
Even as it highlights the importance of safety, Anthropic acknowledged that the financial returns from those investments remain uncertain. The filing did not state how much the company spends on safety research.
Earlier this month, Anthropic said roughly 6% of the computing power it used for AI research went toward safety work during a sample week in July.
The creator of the Claude AI models described its safety programs as “resource-intensive.” It said the company must allocate limited funds among computing power, highly paid AI specialists and safety research.
Anthropic said customer demand—and therefore revenue—is driven by the release of new models. A “continuous and overlapping cadence” of launches, it said, is “inherent to remaining at the frontier of AI development.”
Anthropic CEO Dario Amodei has called for pacing of the Opus AI model
Last week, the company released a new version of its Opus model, just 10 days after CEO Dario Amodei published a nearly 4,000-word essay urging a more measured pace for frontier AI development.
Some analysts and experts have argued that no leading AI lab is likely to slow its progress when doing so could give competitors an advantage in an industry where valuations can shift with every major release.
In recent weeks, Anthropic has pledged to make more information public about how it uses AI models to develop future generations of the technology. Experts have warned about recursive self-improvement—the point at which models could begin advancing on their own without human help.
“We believe building reliable, trustworthy, and secure AI systems is a collective responsibility and that the market will reward it,” Anthropic said in the filing.
