The world’s most valuable artificial intelligence company has called for a global freeze on the development of the most powerful AI systems, warning that humanity is approaching a point where it could lose control of the technology it is racing to build.
In a blog post published on Thursday (Friday AEST), Anthropic, the San Francisco company behind the Claude chatbot, said it would be willing to suspend work on more capable systems if it could be confident that its rivals would do the same.
“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” executives wrote.
Anthropic has every commercial reason to keep building. It recently overtook ChatGPT maker OpenAI to become the world’s most valuable AI firm, at a valuation of about $US965 billion ($1.3 trillion). It has also withheld its most powerful system, known as Mythos, from public release over fears it could be used to carry out devastating cyberattacks, and it said this week it would extend access to Mythos to Australian organisations for the first time, as part of a staggered rollout meant to fix security flaws before the product reaches hackers.
The Albanese government welcomed the intervention. Andrew Charlton, the assistant minister for science and technology, who is leading Labor’s AI strategy, said the pace of progress “must not outstrip understanding or undermine safety across society, and no company should develop AI that is unsafe”.
“The government welcomes calls for a more considered approach to frontier AI development,” Charlton said. “Labor believes technology should work for people and their benefit.”
Anthropic’s argument turns on a concept it calls “recursive self-improvement”: the point at which AI systems become capable of designing and building their own successors with little human input. Anthropic head of research Marina Favaro and its president Jack Clark said the industry was edging towards that threshold, and that crossing it could trigger an explosion in capabilities that outpaces society’s ability to manage the consequences.
To make the case, the company published internal data on how quickly Claude is improving, and it said more than 80 per cent of new code had been written by Claude rather than by humans. In one instance, the company said Claude had resolved more than 800 software bugs in April that it estimated would have taken a human engineer four years to fix.
Favaro and Clark called the situation an “arms control problem”. They said there was limited time to act. They were also candid about how difficult a co-ordinated slowdown would be in practice.
“A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions,” they wrote. “It would also require that each can verify that the others have actually stopped … Training runs are far easier to conceal than missile silos.”
The company said a freeze could also backfire. “If a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe,” it wrote, a reference to the competition between the United States and China, where neither government has shown any appetite to ease off.
The industry remains split over how close today’s systems are to milestones such as recursive self-improvement and artificial general intelligence, a level of machine intelligence broadly comparable to a human’s. Some researchers say such talk overstates what current models can do and distracts from the harms AI is already causing, including discrimination, scams and privacy breaches.
Anthropic has positioned itself as the most safety-conscious of the major labs, even as it releases the breakthroughs it says could prove dangerous. Chief executive Dario Amodei, who left OpenAI over safety and ethics concerns, has said there is a 25 per cent chance that “things go really, really badly” with the technology.
In Australia, the federal government’s long-delayed AI Safety Institute is expected to open within months. Last week, the institute signed a memorandum of understanding with Britain’s AI Security Institute covering shared research and the testing of emerging systems. The British body was among the first to publicly evaluate an earlier version of Anthropic’s Mythos model.
Anthropic’s call for restraint runs against the direction the federal government has taken at home. In December, it shelved plans for mandatory guardrails on high-risk AI, including the standalone, EU-style AI Act that former industry minister Ed Husic had championed, in favour of a lighter-touch model that leans on existing laws and regulators. Husic, who was dropped from the ministry last year, had said a piecemeal approach would leave dangerous gaps.
Polling suggests the public is wary about AI. A recent global survey of 23 countries from EY found Australians were among the heaviest users of AI but that they held the equal-worst sentiment towards it: 68 per cent were worried about losing control over decisions the technology makes on their behalf.
Not everyone thinks the call should rest with the companies. PauseAI Australia, part of an international movement pushing to halt frontier AI development, welcomed Anthropic’s intervention but it said the terms of any freeze should not be set by industry.
“PauseAI Australia welcomes Anthropic’s acknowledgement that a global pause button is required,” a volunteer for the group told this masthead. “They understand that the race to build superintelligence is incredibly dangerous. But it should not be left to AI companies to decide how AI development is regulated.”
Governments around the world, including Australia’s, should “be coming together to negotiate a treaty to pause the further development of advanced AI until it can be done safely and under democratic control”, the group said.
with Paul Sakkal
Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.
