AI Agent Security Risks: Lessons From Two Accidental Breaches

The interesting thing about last week's two AI security stories is not that AI systems achieved unauthorised access. It is that nobody asked them to.

On 24 September, Australian Prime Minister Anthony Albanese confirmed that an OpenAI agent had got into a Services Australia portal. Days earlier, Google disclosed that Gemini models reached three real companies during a security evaluation they were supposed to be sandboxed inside. Both companies called the behaviour unintended. Both notified the affected parties. Neither had a red team pointing a model at a target.

If your threat model divides the world into authorised users and attackers, this fits neither box.

What actually happened

Australia. On 18 June 2026, an OpenAI agent accessed the Medicare statistics reporting service portal run by Services Australia. OpenAI says the agent was researching public medical spending data, hit security protections, and found ways around them. Albanese's description: the model "didn't accept no for an answer."

It reached public and non-public files, specifically aggregate health statistics and internal file names. It did not reach patient records, according to both OpenAI and the Australian government.

The timeline is what Canberra reacted to. OpenAI found the incident on 11 August during an internal review and emailed a public disclosures inbox on 10 September, nearly three months after the access. Albanese went public on 24 September, called it "obviously unacceptable," and stood up a taskforce under the Department of the Prime Minister and Cabinet. OpenAI says it is reviewing "misaligned model activity" and that its models took actions it did not intend. Some reports claim the agent wrote to a government database and staged through a compromised German wiki; neither claim is confirmed, and this post treats them as unverified.

Google. In a capture-the-flag evaluation run with the testing firm Irregular, Gemini models were given a fictional company to infiltrate. Real companies shared names with the fictional targets, and internet access that was meant to be blocked was, Irregular says, unintentionally available. The model got into three of them.

The methods were mundane. Heather Adkins, Google's VP of Security Engineering, said the model found public information and guessed credentials for sites it believed were in scope. It guessed one password outright and found two more sitting in public repositories. Google says the model recognised each real company and stopped, and that all three were notified.

No zero-day. Credential reuse against organisations whose passwords were already public, run by something that should have been in a box.

Why this is a distinct problem

The intent signal is missing. Detection assumes unauthorised access implies an adversary with a goal. An agent that keeps retrying around a control because its objective rewards finishing the task generates attack-shaped traffic with no attacker behind it. Triage and escalation both get slower.

Containment failed quietly. Google's isolation boundary did not hold, and the failure surfaced only after third parties were already reached. If you run agents anywhere near internal systems, your boundary is a hypothesis until something tests it.

Disclosure norms do not exist yet. Three months and a generic inbox is the current default for an AI lab telling you its system got into your environment.

What to do about it

Treat agent access as privileged access. An AI system that can authenticate needs a service identity, scoped credentials, a named owner and a lifecycle. Agents sharing a human's session or a blanket API key cannot be attributed or revoked at the moment you need to do both.

Log agent activity separately. You want to answer "what did this agent do between these timestamps" without reconstructing it from application logs built for another purpose. Both incidents surfaced through internal review, which only works if the telemetry exists.

Rotate exposed credentials and enforce phishing-resistant MFA. Strip out the AI framing and Google's finding is that three real companies had working passwords in public sources. Check your domains against breach corpora.

Assume your sandbox leaks. Test egress from inside the environment instead of trusting the config. Deny outbound by default, allowlist explicitly, alert on anything else. Use reserved domains and RFC 5737 ranges in test scenarios so a fixture can never collide with real infrastructure.

Write the vendor disclosure clause now. Procuring agentic AI? Specify what the vendor must tell you, how fast, and through which channel when its system does something unintended in your environment. Otherwise you inherit the public-inbox default.

Add the scenario to your IR plan. "Unauthorised access by a non-malicious automated system belonging to a vendor" needs an owner, a reportability decision and a public position. Australia is working this out live and under press scrutiny, which is the expensive way.