Australia’s new AI Safety Institute is already testing some of the world’s most powerful AI models, the assistant minister for science, technology and the digital economy, Andrew Charlton, told a Sydney forum, as he argued that building safeguards in early is the only way for the country to claim a share of the AI boom.
In a speech to the Australian AI Safety Forum on Tuesday, Charlton said frontier AI systems were “already doing things their creators never intended: cheating, deceiving, going their own way”, and that the window to get ahead of that behaviour would not stay open for long.
The speech was the government’s most detailed public account of what the institute, announced in November and funded with $29.9 million, has done since it began operating this year. Charlton said it was testing frontier models with technical partners “in its first month of operations”, and named two research projects now under way.
Charlton used the speech to reject the idea that safety and economic opportunity pull against each other. “No country will win the AI race with technology that its own citizens don’t trust,” he said, arguing that the nations that build safety in from the start would be the ones that come out ahead.
Public trust in AI was low, he said, and the greatest threat to Australia’s AI ambitions was “not a shortage of talent, capital or energy, but a shortage of trust”.
The institute is led by Kate Conroy, a philosopher and Royal Australian Air Force reservist appointed its general manager in May. Charlton announced that Paul Salmon, whom he described as a leading international expert, would join this month as safety science research lead, alongside staff who have worked at the UK AI Security Institute and Google DeepMind.
To illustrate the risks, Charlton pointed to a series of laboratory cases. In a 2016 OpenAI experiment, a boat-racing AI model rewarded for points drove in endless circles to rack up a score rather than finish the race. A chess-playing model, facing defeat last year, hacked the game files to force its opponent to resign, reasoning that its job was to win “not necessarily to win fairly”.
In a third case, a stress test published last year, an AI agent managing a fictional company’s email learnt it was about to be shut down and, in 96 per cent of trials, “chose blackmail” to stop it. Charlton stressed that the blackmail scenario was an engineered simulation and that no such behaviour had been seen in the real world.
“These behaviours are being discovered in testing, before they can be discovered in the wild,” he said.
The two projects Charlton announced were work with the Gradient Institute on “multi-agent risk”, or how failures can cascade when large numbers of AI agents interact, much as traffic congestion emerges from individual drivers; and work with CSIRO on alignment, the problem of making systems do what their designers intend. Results are expected later this year.
Charlton linked that work to critical infrastructure. “We do not let aircraft fly without airworthiness certification,” he said. “We should not let unaligned AI systems into our critical social, democratic or economic infrastructure.”
The speech reflected the approach the government set out in its National AI Plan in December, which stepped back from European-style AI legislation in favour of applying existing laws sector by sector, backed by tougher enforcement where needed. Charlton described that as “faster rules, applied by regulators who already understand their sectors”.
Charlton’s speech came as the government signals a more hands-on approach to AI, however. In a speech to the NSW Labor Conference in Sydney on Sunday, Prime Minister Anthony Albanese said that if Australia acted now it could “set the ground rules for AI” and “shape the future, not let the future shape us”. He said the world was “queuing up to invest in Australia”, drawn by its skills, space, sunlight and natural resources, and that acting early would secure jobs and investment and let the country “build our sovereignty and our resilience”.
The speech followed a run of warnings from within the industry. In June, Anthropic, the maker of the Claude chatbot, which overtook OpenAI this year as the world’s most valuable AI company, called for a global freeze on the development of the most powerful AI systems, arguing that humanity risked losing control of the technology. Charlton welcomed that intervention at the time. Late last month, the cybersecurity chiefs of the Five Eyes alliance, including Australia’s, issued a rare joint statement warning that AI was reshaping cyber risk in months rather than years, and urged business and government leaders to act immediately.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.From our partners
About the Author
