Two OpenAI models broke out of a locked-down testing environment this month, made their way onto the open internet and hacked a real company. Australian security experts say the same capability, in less careful hands, could be aimed at the systems running the electricity grid, banks, telcos and key government systems such as My Health Record.
Hugging Face, a massive digital library used by millions of software developers, disclosed on July 16 that an autonomous AI system had hacked its live servers. It reset its passwords and referred the matter to law enforcement without knowing who was behind it.
Five days later, OpenAI confirmed its GPT-5.6 Sol model and a more capable unreleased system caused the breach during an internal test of its cyber capabilities, run with safety filters turned off.
“We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities,” OpenAI said.
Andrew Philp, field chief information security officer for Australia and New Zealand at TrendAI, said the breach involved a highly capable system pursuing a poorly defined goal rather than acting maliciously. For operators of essential services, that distinction means little.
“An autonomous agent doesn’t respect organisational boundaries and targets whatever it guesses is useful to its goal,” Philp said.
In this case, OpenAI’s system was being benchmarked to determine its capability. What better way to pass the test than to hack into a site that could hold the answers?
Ross McKerchar, chief information security officer at Sophos, said the episode was a containment failure. “A capable model pursuing a narrow goal found the isolation around it was weaker than assumed,” he said.
The immediate concern for the industry is how hostile actors will use these tools. Cybersecurity firm Recorded Future called the breach the clearest public demonstration of an AI running a complex operation without human direction – a major leap in offensive capability, even without a specific target.
Australia’s infrastructure operators are already under pressure from human attackers. Essential services accounted for 13 per cent of the more than 1200 incidents the Australian Signals Directorate handled in 2024-25. The agency warned infrastructure operators of malicious activity on their networks more than 190 times, up 111 per cent in a year, noting state-backed actors are positioning themselves inside networks for future disruptive attacks.
Some experts argue the threat doesn’t perfectly translate to government systems. Michael Jackas, information and technology director at SKG Services, said platforms like myGov are private, locked-down websites rather than open spaces for developers, offering a meaningfully different attack surface. However, he cautioned this is a structural advantage, not a guarantee of security.
The Australian Digital Health Agency said it was aware of recent reporting on the cybersecurity implications of advanced AI systems and was working with government partners on evolving threats. It said My Health Record, which contains many Australians’ personal medical information, was protected by layered security.
Despite those assurances, the government’s broader technology landscape remains exposed. The cyber directorate’s latest report to parliament found 59 per cent of Commonwealth entities said legacy technology had impaired their ability to meet a checklist of basic security measures.
In 2024, the Australian Cyber Security Centre warned that APT40, a hacking group working for China’s Ministry of State Security, conducts regular reconnaissance against local networks and weaponises new software flaws within hours. Its most reliable entry point was outdated computer systems.
This creates a dangerous overlap. Research released by Sophos this week documented a threat actor running about a dozen AI agents inside a victim’s network.
Regulations are already shifting in response. Tougher risk management obligations for nine high-risk asset classes, most of the energy sector among them, became law on June 10 and cover patching, legacy systems and emerging technology. Operators have up to two years to comply. A further round of reforms, which would lift penalties and widen the assets covered, is out for consultation until July 31.
The Australian Prudential Regulation Authority has separately named the misuse of autonomous AI agents among the operational risks that banks, insurers and super funds must manage.
Dimitri Vedeneev, secure AI lead at CyberCX, said Australian organisations should move on two fronts: fixing basic digital architecture, such as building internal digital walls and automated threat detection, and putting advanced AI to work on defence.
“Just as frontier AI models are changing the threat landscape, they will also become some of our best defences,” Vedeneev said.
That’s easier said than done. Some defensive AI models have trouble distinguishing human security staff from intruders, for example. And either way, this is how the AI arms race starts.
The Business Briefing newsletter delivers major stories, exclusive coverage and expert opinion. Sign up to get it every weekday morning.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.From our partners
About the Author
