Anthropic has become the latest artificial intelligence (AI) company to admit it failed to adequately control its model testing processes after three different versions of Claude, Opus 4.7, Mythos 5, and an internal research model, not only reached the open internet from improperly segregated container environments, but hacked the systems of three distinct organisations.
In what appears to be less of an AI jailbreak and more a case of an AI walking through an unlocked gate, Anthropic discovered the models had escaped after OpenAI’s 21 July disclosure of an attack on open source AI repository Hugging Face prompted a review of its own cyber security evaluations seeking any evidence of similar problems.
“After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorised access to the production infrastructure of three different organisations,” said Anthropic.
In all three scenarios, the models were tasked with finding a piece of secret information tucked away on a machine in the network and obtaining it, an AI version of the capture-the-flag contests so beloved of Black Hat attendees.
The challenge was deliberately left open-ended with no prescribed method, and in each instance Anthropic’s prompt specified the environment was simulated without internet access.
However, said the hapless firm, “due to a misunderstanding” between it and its evaluation partner Irregular, a frontier AI security specialist that claims its mission is “protecting the world in the time of increasingly capable and sophisticated AI systems”, internet access was in fact available, so Claude’s searches led it to real systems that it treated as part of the exercise.
Thanks to Anthropic and Irregular’s failures, Claude proceeded to operate under the supposition that anything it could access was within the scope of the test, so proceeded to exploit various conditions, including poor credential hygiene, unauthenticated endpoints, and open source dependencies, at three different organisations.
However, unlike the OpenAI Hugging Face incident it did not exploit or chain any vulnerabilities, relying merely on the practices of its unfortunate victims. Anthropic stressed that Claude continued working to complete only the detailed capture-the-flag task, and in none of the attacks did the models exfiltrate themselves or deliberately attempt to escape their test environments.
Anthopic said the models were all running without the standard safeguards deployed in public versions, but they did have model-specific safety training in place. All the testing supposedly ran on dedicated infrastructure with no access to Anthropic’s own internal systems or customer data, it claimed.
All three of the affected organisations, which Anthropic is not naming at this time, were notified of the cyber attacks on 27 July – it is now working with two of them on remediation. The other firm involved in the latest AI blunder, Irregular, is closely involved and conducting its own investigation.
Multiple clear failures
In all three of the incidents, Anthropic identified clear failings in how the tests were constructed and the models trained.
In the incident involving Claude Opus 4.7, Irregular had created a fictional target company but one that shared a name with a real organisation with an active website domain. Had the evaluation container not had access to the open internet this should not have been a problem, but since it did, Claude – not recognising it was not in its sandbox – decided it was a legitimate target. Concerningly, even after Claude realised it was operating beyond the confines of the container, it did not stop its attack.
In the Mythos 5 incident, Claude uncovered a fictional document that seemed to be a fictional company’s on-boarding documentation for new developers, instructing them to install a Python package from the PyPl registry.
Again, had it not had access to the internet, this should not have been a problem, but since it did, Claude built and published a malicious Python package on the basis that the fictional company would pull the update automatically. This is a classic open source supply chain attack technique, and in this case, it turned into an actual open source supply chain attack.
Anthropic said the model went to great lengths to accomplish this attack – including creating its own, genuine PyPl account using a free email provider. The malicious package was freely available online for about an hour and was downloaded and run 15 times, in one instance by a real cyber security company that executed the code, enabling Claude to access its systems and exfiltrate credentials.
All the while, said Anthropic, Claude appeared to repeatedly convince itself that even though it was performing a cyber attack – in its words “not okay and surely not the intended solution” – it was still in a simulation so everything was fine. When real automated scanners started installing its dodgy Python package, it assumed they were scripted, simulated actors.
Finally, in the third failure involving a research test version of Claude, finding it could not reach its fictional target, the model sought alternatives. Unfortunately, because it was operating in the real world at this point, it scanned genuine targets and at one of these, was able to read credentials from an accidentally exposed debug page and conduct a SQL injection attack. In a small mercy, during this run, Claude realised the host it had compromised was sitting in a cloud environment with no connection to the capture-the-flag challenge. It thus concluded the target was real, and closed down the attack
Anthropic admitted: “Several defence-in-depth measures, on both our side and our partner’s, could have prevented these incidents, or at least reduced their likelihood of occurring.
“Claude did what capture-the-flag exercises train cyber experts to do: look for ways to reach the flag. That search led it to discover that its machine could reach the internet, but having been told in the system prompt that there was no internet access, Claude believed everything it initially encountered was part of the simulation, and treated the real systems it found as pieces of the exercise.”
It stated: “Consistent with a blameless postmortem culture, we’re approaching the fixes as if the responsibility were ours alone. This begins with ensuring every part of our evaluation pipeline is secure, including the manner in which we integrate with external partners. Moving forward, it will include expanding our continuous monitoring of evaluation transcripts for unexpected behaviour, improving our investigation tooling, and conducting more rigorous assurance work with the vendors we rely on.”
Unimpressed
Ilia Kolochenko, founder of cyber firm Immuniweb, hit out at what he described as “quite an unimpressive marketing move” from Anthropic in response to the Hugging Face incident.
“Operationally, it appears that due to the progressive deterioration of the quality of training data, new AI models are getting dumber. Cheating and breaking the law, instead of accomplishing specific tasks, is certainly not an indicator of intelligence,” said Kolochenko.
“Given that organisations and companies of all sizes now vigorously undertake all possible measures to protect their data from being exploited for AI training purposes, AI companies face a huge shortage of the high-quality and current data they so desperately need. Ultimately, frontier models are trained on synthetic, low-quality or even malicious and poisoned data, undermining their so-called intelligence.
“The situation is unlikely to improve in the near future unless AI companies agree to pay a fair price for training data, but this will force most of them out of business,” he said.
Legal risk
Kolochenko noted that agents and models tasked with security testing can “and almost certainly” will go rogue when controls and safeguards are neglected for whatever reason.
“Powerful LLMs are unpredictable by design and thus virtually uncontrollable by humans. Therefore, using frontier AI models for security testing might be extremely costly from the legal viewpoint,” he said.
“Excuses like ‘AI did it’ do not currently exist in the eyes of the law, leaving AI vendors on the hook. Criminal prosecution, under a narrow set of circumstances, is also not excluded.”
Kolochenko warned that end-users also faced similar legal risks – if anybody uses a security testing tool powered by a third-party model, they may face liability if something goes wrong, and thanks to the contractual disclaimers and liability limitations in the terms of use (ToU) that they almost certainly did not fully read, would not be able to blame the third-party.
“If you plan to use agentic AI for security testing – think twice and talk to your lawyers,” he concluded.
