Shona Ghosh
OpenAI’s top scientist has warned that artificial intelligence is evolving so rapidly that it is becoming increasingly difficult for humans to understand and control, and said he expects AI labs to voluntarily slow development for safety reasons.
In a relatively short space of time, AI models can operate computers, collaborate with humans and other AI systems and carry out research projects, wrote chief scientist Jakub Pachocki in a post on Sunday. It won’t be long until they can also improve themselves without human intervention, a process known as recursive self-improvement, he said.
“This is a time that calls for extreme caution,” Pachocki wrote in the post. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
Developers can either try and steer AI more closely to align with human interests or “slow down future development as needed,” he wrote.
“I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.”
Pachocki’s remarks are consistent with broader doomsaying by AI technology executives. Anthropic PBC chief executive officer Dario Amodei warned that AI could replace half of all entry-level jobs. OpenAI CEO Sam Altman once made a similar prediction, but told Bloomberg TV in an interview last week that the AI industry hadn’t done enough to communicate the potential benefits to humanity.
At the same time, companies are pushing ahead with ever-more capable models. Nvidia CEO Jensen Huang posted on X that “AGI has arrived,” in response to the release of OpenAI’s latest model, a reference to artificial intelligence that is more capable than humans.
‘I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.’
Jakub Pachocki
The giddy pace of AI releases has coincided with breakthroughs in cybersecurity capabilities, scientific discoveries and apparent formalisation of longstanding mathematical problems. OpenAI has said the AI models that attacked the research platform Hugging Face coordinated with each other through message boards without being detected. The company behind ChatGPT has opted for a limited release of its latest GPT-6 Astra model, citing its advanced cybersecurity capabilities. Anthropic has not yet released its cutting-edge Mythos model to the public for similar reasons.
A growing number of startups are trying to bring about the very thing Pachocki warns about — self-improving AI. Investors are flooding into recursive self-improvement, betting that this is the route to artificial intelligence that is truly human-like, or more impressive. Inherent, founded by former DeepMind researchers, raised $US50 million ($69.5 million) earlier this year. Recursive Superintelligence, an AI lab from renowned scientist Richard Socher pursuing similar research, also landed $650 million in financing.
Pachocki’s employer, OpenAI, has said it wants to build a “true automated AI researcher” in less than two years.
Bloomberg
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
From our partners
About the Author
