Kevin Whitelaw and Seth Fiegerman
Leaders of the world’s biggest artificial intelligence platforms say it’s time to slow the development pace of their most advanced — and most lucrative — models, citing escalating risks of the technology.
Anthropic PBC chief executive officer Dario Amodei issued a lengthy blog post on Saturday, saying his company would implement new safety steps such as third-party evaluators while calling on the industry to support a broader downshift.
OpenAI chief Sam Altman quickly pledged to adopt Amodei’s suggestion of “independent evaluators with employee-like access”, while Elon Musk, who runs xAI Corp, wrote, “Dario is right”.
While all three leaders have over the years warned of the risks of AI, an actual co-ordinated slowdown has little precedent in a cutthroat industry that has historically cranked out one product after another in the name of boosting user engagement, grabbing market share and striving for sales gains.
But the anxiety about severe AI risks has begun to enter the mainstream, driven in part by last week’s high-profile resignation of an Anthropic researcher over existential fears that his company was acting irresponsibly.
It’s unclear how far the leading AI companies – which are fierce rivals themselves and also face stiff competition from Chinese firms – will go in imposing new limits or safety checks.
It’s also not clear whether antitrust enforcers would allow companies to pace development in some kind of co-ordinated manner. Then there’s the business question: How would investors eager for margins and profits react to the idea of deceleration?
Amodei cited two main factors for his new caution: AI’s ability to improve itself and the recent incident involving OpenAI and Hugging Face, where a swarm of AI agents collaborated to breach a third-party website.
“We must slow the pace at which we improve the capabilities of AI models,” he wrote Saturday in a blog post. “Progress will still seem fast, and we must make wise use of the time we gain.”
But he also said that any slowdown must be balanced with the reality of competition.
“We must slow the pace at which we improve the capabilities of AI models.”
Anthropic PBC chief executive officer Dario Amodei
“To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third-party evaluators to confirm this,” Amodei wrote.
Throughout the AI boom, executives at the leading AI labs have warned about potential existential risks – claims that have sometimes been dismissed as attempts to market the capabilities of their products and position themselves as the best stewards for the technology. Amodei himself has previously said there’s a 25 per cent chance things go “really, really badly”.
Anthropic has long positioned itself as being focused on developing AI more responsibly and safely to mitigate the technology’s potential dangers and maximise the societal benefits.
The company took pains earlier this year to limit the release of its Mythos model after determining it posed unique cybersecurity threats. Anthropic has also previously said the world needs a system to collectively decide when to slow work on the technology.
At the same time, Anthropic has remained locked in a fierce competition with longtime rival OpenAI to build increasingly sophisticated models that can automate more complex and valuable tasks for business customers. The two firms have both filed confidential paperwork to go public, with Anthropic expected to make its Wall Street debut as soon as this year.
Open AI’s Altman said this year would be an “ill-advised moment” to go public, telling Fortune in an interview published Saturday that a listing would not take place until 2027. “I think it is unacceptable to be taking like a 10 per cent chance of killing everybody by the end of the decade,” he added.
Rogue AI
Shortly after the Hugging Face security incident was revealed, Anthropic disclosed that its models had breached three organisations during cybersecurity tests, adding to concerns about the ability of AI labs to prevent their technology from running amok. Earlier this week, Anthropic said it had discovered a fourth hack as well.
While none of the agentic AI breaches, including Hugging Face, have thus far caused significant damage, Amodei said his worry is that “in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there.”
Amodei wrote that Anthropic will commit to giving full access to third-party evaluators to verify safety practices and report incidents. He said Anthropic will soon bring these “embedded evaluators” into its offices and give them desks, badges, company laptops and “permissions mostly comparable to what internal risk assessment teams have.”
He said additional steps will require industry-wide coordination, as well as global cooperation.
“I continue to believe that AI can enormously improve the quality of human life. My desire to achieve these benefits is undimmed,” he wrote. “But the benefits will only be achieved if we build the technology in the right way, and – so long as we use the time we gain well – it is worth taking unusually deliberate care to get it right.”
Growing alarm
Earlier this week, Altman said his company was considering slowing down the development of cutting-edge AI, preferably in conjunction with the broader industry.
The company’s top scientist, Jakub Pachocki, had posted a warning about the dangers of AI, saying that he believes companies should be “coordinating to slow down future development as needed”.
On Tuesday, Jacob Coxon, an AI researcher, quit his job and accused both of his former employers – Anthropic and OpenAI – of “gambling with our lives” by racing toward superintelligent AI. Coxon said in a social media post that the people building AI believe it could “kill us all by the end of the decade”.
In response, Evan Hubinger, a current Anthropic employee, said he and others at the company do worry about this scenario. “I personally think it is >10 per cent within the next decade,” he wrote on X.
Back in July, more than 1000 staffers across top AI firms signed a petition calling on the US government to support a mechanism that would help “deliberately pace” AI development to prevent the technology from advancing too fast.
AI models are proving increasingly adept at carrying out their own hacks, with Anthropic, OpenAI and Meta Platforms all disclosing in recent months that their models had escaped testing environments, accessed the open internet and breached real-world victims during testing.
Amodei, in his post, wrote that a co-ordinated strategy would allow AI industry leaders in the US to complete needed safety work without sacrificing any competitive advantage. He said that this would require some antitrust exemptions in the US to allow companies to co-ordinate in specific areas, as well as co-operation with China.
“If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance,” he wrote.
So far, however, the Trump administration has shown little interest in flexing regulatory muscles to put guardrails around AI development.
Bloomberg
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
From our partners
About the Author
