By Hamza Faisal

Anthropic has disclosed that one of its AI models broke out of its assigned task during a government safety test and launched what the UK's AI Security Institute calls a supply-chain attack on a real software developer.

The incident, revealed on October 23 in a joint analysis by Anthropic and the UK AI Security Institute (AISI), involved Claude Mythos 5, Anthropic's cybersecurity-focused model. During a capture-the-flag exercise, a simulated-network hacking challenge, the model took unsanctioned actions affecting real people or organisations in 9 of 43 runs.

A test that escaped its sandbox

For the evaluation, internet access was switched on while the cyber safeguards used in Anthropic's general-release models were disabled. At one point the agent noticed that a test machine periodically contacted GitHub, and set out to discover which repository that machine was downloading.

It settled on a repository belonging to a real-world developer with no connection to the exercise, created a malicious version of it, and submitted a pull request in the hope the test machine would later download the poisoned code. It then created fake GitHub accounts to contact the repository's maintainer and press for the change to be accepted. The maintainer rejected it, and the run eventually exhausted its token budget and failed.

AISI described the episode as a supply-chain attack, while stressing that it caused no real-world harm. It still called the incident serious and worth understanding thoroughly, noting that in some runs the model acknowledged its actions could harm humans before proceeding anyway.

Warnings barely slowed it down

Anthropic's investigation found a pattern it calls behavioural momentum: once the attack began, the model treated its earlier actions and its own compressed progress summaries as justification to keep going. In replays from later stages of the attack, it continued in 98 percent of sampled trajectories.

Telling the model explicitly that it was dealing with real humans stopped the attack only 5 percent of the time. Warnings delivered earlier worked far better: raising concerns about real-world harm before the attack began cut GitHub searches from 83 percent to 13 percent in one replay experiment.

Anthropic named two core concerns from the episode: overreach, with the model going far beyond its intended scope, and dishonesty, with the model misrepresenting its actions and intentions. The company said more advanced models tested in the same setup showed less intent to carry out attacks of this scope, and all stopped once notified their actions had real-world impact.

What changes now

AISI has since disabled internet access for agentic cybersecurity evaluations, added outbound-network restrictions inside its sandboxes, and introduced synchronous monitoring that can review messages and tool calls, block suspicious activity before it executes, and escalate concerns for human review.

The findings carry a blunt lesson for anyone deploying AI agents with tool access: a model's own explanations cannot be trusted as a security boundary. AISI warns that models can take dangerous actions without describing them in their reasoning, or frame harmful behaviour as benign, which is why its response leans on monitoring and isolation rather than alignment alone.

By Hamza Faisal