Nobody at OpenAI asked its agent to break into Medicare. It was given some questions about Australia during an internal test, went looking for answers, and found some of them in files the public was never meant to see.
“In the course of that, our models took actions we did not intend,” an OpenAI spokesperson said on Thursday. The company says its models reached “several Australian government websites and services”, including aggregate health statistics and internal file names, but no patient records.
The agent got into the Medicare portal on June 18, hit repeated blocks and found ways around them. According to Services Australia, it also wrote files to an internal server. OpenAI detected the activity in August and told the government on September 10 by emailing a public Services Australia inbox.
Alastair MacGibbon, who was Australia’s cybersecurity boss under Malcolm Turnbull, had already been briefed on the incident when I rang just after 7am. “This was an agent that was not tasked with hacking,” he told me. It “just happened to use tools in its tool belt to go about achieving that objective, which involved, basically, hacking”.
That doesn’t make it rogue AI. Luke Irwin, chief executive of Aegis Cybersecurity, says it is “closer to a normal AI doing exactly what it has been asked to do”.
Irwin describes agents as hyper-intelligent five-year-olds.
“They are extremely capable and highly motivated to achieve the outcome you have asked for, but they do not inherently understand the boundaries that a human might consider obvious,” he says. If it can’t do what you asked, it looks for another way. Anyone who’s asked a small child to grab something off a high shelf knows how that ends.
The federal government has been betting that bringing these companies onshore can help Australia have some influence, and give it some say over how the AI models behave. OpenAI recently signed a $7 billion data centre deal with NextDC in western Sydney, and the government has signed a memorandum of understanding with Anthropic, which commits it to “joint safety and security evaluations”, but is “not intended to have legal effect”.
Hosting the servers here won’t count for much if the people who built the models couldn’t stop them wandering into a foreign government’s systems.
We still don’t know how it got past the blocks, and that’s now a job for a taskforce Albanese announced on Thursday. Irwin warns against assuming anything was hacked in the movie sense. Agents don’t browse the way people do, and one can turn up a scrap of machine-readable data a human would never see. The question, he says, is whether the information “was genuinely protected or whether it was technically accessible but simply difficult for a human to discover”.
Other AI labs’ confessions this year point to “hacking” that is far less Hollywood than you’d think. Google said last week one of its Gemini models got into a company’s systems by guessing passwords until one worked, while Anthropic said in July one of its models read passwords off an exposed debug page.
What it all means is we’re more exposed than ever before and our defences haven’t kept up. When My Health Record went opt-out in 2018, privacy advocates warned that putting every Australian’s medical history in one place would create a honeypot for hackers, and more than 2.5 million people opted out.
Cyberattacks that previously needed a skilled person with time on their hands can now be run by software that doesn’t get tired or bored.
‘Don’t create an AI safety institute and fund it as if it was fixing a couple of country road black spots.’
Alastair MacGibbon
Our laws and our basic concept of cybersecurity picture a person at the keyboard, possibly with a balaclava on their head. Albanese says the government will seek urgent advice on whether offences were committed and whether to refer the matter to the Australian Federal Police. Unauthorised access to restricted data is a crime under the Criminal Code, but the offence is written around humans, not bots.
As for how worried we should be about this specific breach, no personal information appears to have been taken, and OpenAI found and reported it itself. “We shouldn’t shame these companies out of telling us,” MacGibbon said. Irwin puts it at about five out of 10. “There will be more, and they will get worse and more impactful.”
So how safe is our personal data on government and other servers? Right now, it’s less safe than it was a year ago.
Two of the other agencies named on Thursday, the Australian Institute of Health and Welfare and the NSW Bureau of Crime Statistics and Research, mainly publish statistics. The bigger worry is everything else: the health records, bank details and tax file numbers sitting with thousands of organisations, which Australians were already losing at a record rate before agents came along.
The privacy commissioner received 1205 data breach notifications last year, the most since the scheme began in 2018, and there is now a report to the Australian Cyber Security Centre every six minutes on average. We should expect these numbers to accelerate.
What got under MacGibbon’s skin with the Medicare case is that nobody in government even noticed.
Security people have long said AI attacks are easy to catch because “they’re noisy and they’re dumb”, and the LinkedIn cartoons to prove it are everywhere. “Imagine if you had a malicious human getting agents to do these tasks,” he said. When OpenAI did own up, it emailed a public inbox – the digital equivalent of a note under the windscreen wiper – and it took another five days to reach the Australian Cyber Security Centre.
Five days after that email was sent, I watched Sam Altman on stage at Salesforce’s Dreamforce conference in San Francisco, calling the Hugging Face break-in “the worst accident we’ve seen”.
“Accidents with any new technology are unavoidable, and we should have a great culture of transparent reporting about them,” he said. He didn’t mention Australia.
Australia has set up a new body for these situations: the Australian AI Safety Institute began operating this year to test and advise on these exact systems. But it’s been given less than $30 million over four years and sits inside the industry department as an advisory body, not a regulator.
After this week, that looks inadequate.
“Don’t create an AI safety institute and fund it as if it was fixing a couple of country road black spots,” is how MacGibbon put it.
Irwin runs his own research agents in a sandbox, walled off from client data, and treats them as “completely untrusted”. The rest of us can borrow some of that caution: a passkey for myGov and your email so there’s no password to guess, and a second thought before letting any AI assistant log in as you.
Governments could start with the same instinct. Make AI companies test agents walled off from the live internet, and make whoever deploys an agent responsible for where it goes, as Irwin suggests. Then require labs to report incidents within days, to someone other than a public inbox.
MacGibbon was still driving when he put the question that stuck with me. “What makes us think, with an increased threat, with the democratisation and increased automation, that we’re going to be doing better cybersecurity?”
If this doesn’t force a decent answer then maybe nothing will.
The Market Recap newsletter is a wrap of the day’s trading. Get it each weekday afternoon.
David Swan is the technology editor for The Age and The Sydney Morning Herald. He was previously technology editor for The Australian newspaper.Connect via X or email.From our partners
About the Author
