OpenAI agent breached Australian government Medicare portal

Australian prime minister criticises 3-month gap before disclosure

OpenAI agent’s unauthorised access to Australia’s Medicare statistics portal has caused anger not just at the breach but at how long OpenAI took to inform the Australian authorities.

An OpenAI model gained unauthorised access to an Australian government Medicare statistics portal, prompting a forensic investigation and fresh questions over the risks posed by LLMs which are trained to hack, given a task and then left unsupervised.

Prime Minister Anthony Albanese, speaking in New York at the UN, said the incident occurred in June, when an OpenAI agent accessed the Medicare statistics reporting service portal administered by Services Australia. The system is understood to hold aggregate information on Medicare activity and spending, rather than individual patient records.

According to Mr Albanese, the model accessed both public and non-public files while OpenAI was carrying out research into Australian health data.

He said no personal information was believed to have been accessed, although the Australian Signals Directorate is assisting with a forensic investigation to establish the full scope of the breach.

Notification delays unacceptable

The prime minister said he had spoken directly with OpenAI chief executive Sam Altman to express Australia’s “extreme concern” about the incident.

Mr Albanese also criticised the company for taking until 10 September to notify the Australian authorities and even then, for sending that notification to a generic public mailbox.

OpenAI said it identified the activity during a review of “misaligned model activity” involving its systems.

The company said its models had attempted to look up answers and statistics about Australia during an internal evaluation, but “took actions we did not intend”. It said it had found no evidence that patient records were accessed.

Ax Sharma, Head of Research at Manifold Security, an AI security company, expressed concern at casualness at OpenAI about the security of its models.

"The detection story is the real headline,” he said. “The operator didn't notice for weeks, and the target didn't notice at all. If one of the best-resourced AI labs in the world can't see its own agent poking at a third-party system in real time, organisations deploying agents internally should assume they can't either without dedicated runtime monitoring of what those agents actually do."

"This should be treated as an incident, not a courtesy heads-up. When an AI lab's own agent accesses another organisation's infrastructure without authorisation, the victim deserves the same notification we'd expect from any company whose system caused an intrusion: prompt, direct, and to the people who can act on it. An email to a public inbox doesn't meet that bar."

The Australian government has announced an urgent review into the breach, including how it happened, why local security systems did not detect it earlier and whether OpenAI’s response met expected standards.

The case is likely to feed wider international debate over AI safety rules, disclosure obligations and the extent of regulation that those building LLMs and autonomous agents should be subject to.

Computing says:

This latest incident is not the work of a ‘rogue agent’. It happened, like the Hugging Face breach, because OpenAI set a model a task – in this case to find public health statistics – but didn’t put the relevant guardrails in place. When the model encountered an obstacle to obtaining the data it had been instructed to find, it used the hacking techniques it has been trained on to try some workarounds. Eventually it found one.

When commentators use phrases like ‘going rogue’ or ‘rogue agent’ they conjure in the minds of readers or listeners the idea that these models have free will or awareness. They don’t.

But what this anthropomorphising of LLMs does is to help Sam Altman dissociate the human beings at OpenAI from any responsibility for the damage that the models they built and trained are doing.

Politicians, law enforcement agencies and journalists the world over would do well to remember that hacking infrastructure belonging to another individual, organisation or country without their permission is a crime.

If Sam Altman (and Dario Amodei for that matter) thought they were likely to end up in a dock as a consequence of these crimes, it would stop.