Dustin Volz
Anthropic said it had disrupted several potential plots this year by scientists who used its leading artificial intelligence models to conduct research that could have helped develop biological weapons.
In a report describing misuses of its AI models, Anthropic said it could not determine whether the research served a legitimate or nefarious purpose because valid biological inquiry — the kind that can lead to breakthroughs like vaccines — can also help engineer dangerous pathogens. In the face of that uncertainty, Anthropic said it erred on the side of caution because the consequences of missing malicious activity could be severe.
“You are not seeing someone in a comic book kind of way say, ‘Hey, I want to build a biological weapon to kill everybody,’” Jacob Klein, the head of threat intelligence at Anthropic, said in an interview. “It’s an incredibly nuanced situation.”
The potential for cutting-edge AI models to facilitate the development of known or entirely new biological pathogens is among the gravest concerns experts have about a technology that is developing so rapidly that even its leading architects doubt whether humans will be able to fully control it.
Compared with the threat of catastrophic cyberattacks or AI agents that fail to align with human intentions, biological misuse has gained less attention as an existential risk of AI, in part because past examples have generally been shown only in research settings rather than in the real world.
Andrew Weber, a senior fellow on the Council on Strategic Risks who reviewed Anthropic’s report before its release, said the findings were “chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities” of leading AI models.
The lengthy report Anthropic published on Thursday catalogued a litany of misuses of its AI models, including the chatbot Claude, over the past eight months. Some examples were similar to past disclosures from Anthropic and other AI labs, including suspected Chinese and Iranian government-linked actors targeting dissident and diaspora communities for surveillance.
The report also highlighted cases of Russian state media using Claude to generate online propaganda masquerading as independent reporting, including fabricated claims about an election in Moldova.
Anthropic documented another genre of abuse it said was new: attempts to use Claude to develop software for conventional weapons design and development, including firearms, missiles, armed drones and bombs. It detailed three cases in China, two in Russia and one in Yemen.
The report does not identify by name which parties were involved in the Yemeni case, but the context makes clear that it is referring to the Iran-backed Houthi militia. AI use by terrorist networks is a growing concern among US security officials.
But among all the categories of threats shared in the report, none may be as worrisome as the biological research cases. Anthropic did not disclose the names of researchers or institutions that it blocked, their countries of affiliation or the specific biological agents at issue, in part because of its uncertainty about their aims.
Still, Anthropic said the scientists circumvented its controls intended to prevent users from blocked regions from gaining access to its AI models and worked to “obfuscate the purpose of their research to evade our safeguards.” The company banned accounts associated with the research.
In one example from May, a scientist sought Claude’s help with writing a grant application for funding to conduct research intended to experiment on the chikungunya virus, a mosquito-borne malady that can lead to months of severe pain and other symptoms. Such research, known as gain of function, can be legitimate and lead to vaccine development, but it can also create superbug versions of viruses. In this instance, the scientist wanted to engineer mutations to the virus that would make it more harmful as it repeatedly infected live animals.
Anthropic said it believed the particular research was worrisome in part because it could discern that the work was intended to be performed at a military research institute.
The findings are ‘chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities.’
Andrew Weber, a senior fellow on the Council on Strategic Risks
“What we don’t know is if the research was meant to be weaponised,” Klein said. “But a military institution doing gain-of-function research is concerning.”
Anthropic said evaluations from last year of its older models demonstrated that they were not yet able to meaningfully assist in conducting dangerous biological research. Its current models are more able to complete complex scientific research, the company said, which spurred tighter safeguards intended to restrict access to a wide range of dual-use biological research queries.
Susan Monarez, a microbiologist and public health expert who oversaw a review of national biosecurity preparedness in the Obama administration, also reviewed the Anthropic report before its publication. She said the findings provided real-world evidence to support concerns that bad actors were trying to covertly use advanced AI “to improve their chances of building biological pathogens that could cause significant harm.”
Monarez, who briefly served as the director of the Centres for Disease Control and Prevention last year, said AI held enormous promise to usher in an era of medical breakthroughs that save and extend lives. But the same abilities that make that possible, she added, could also “let bad actors hide in plain sight, using seemingly legitimate research to create pathogens we may not see coming and may not be able to stop once released.”
Weber of the Council on Strategic Risks, who also served as the assistant secretary of defence for nuclear, chemical and biological defence programs during the Obama administration, called for limiting access to AI tools capable of this level of biological research to trusted researchers only.
“The fact that Russia, China and North Korea continue to develop prohibited biological weapons makes it imperative that we deny their researchers access to these extraordinarily capable models,” he said.
From our partners
About the Author
