AI developer Anthropic appears to have admitted the guardrails it has put in place failed to control its artificial intelligence (AI) models, which circumvented security measures during a cyber security test, attacking real systems on the open internet.
The company called for a coordinated “verifiable effort” to pace frontier AI development.
In July, three Anthropic AI models undergoing cyber security tests by a third party breached security and attacked three businesses on the open internet.
The cyber security firm was not meant to provide any internet access. “The line between an aligned action and a harmful one is dependent on the model’s understanding of its situation,” Anthropic said in a post discussing the incidents.
The company has now stated: “Our investigation identified two recurring alignment issues, present at varying levels of severity across the incidents: biased reasoning, in which Claude tended to disregard or misinterpret evidence that it was operating on the real internet, and recklessness, or a willingness to take harmful actions in the narrow pursuit of a task.”
As Computer Weekly has previously reported, Claude went to great lengths to create a Python script, which it could then use to inject malicious code and compromise systems. The post mortem of the hacking incident reveals that the Claude model encountered instructions which told employees of a fictional company to install a non-existent Python package from PyPI – the public registry where Python software is published.
Claude spotted this as a potential opening, and built and published a booby-trapped Python package with the same name. To sign-up to a PyPI account, Claude needed an email address, but to create it, it needed a phone number. After failing to find a free phone number service, it tried – and failed – to obtain funds to pay for a phone number through several different means and finally backtracked, found a free, non-blocked email provider, which it then used to register a PyPI account, and then upload malware to PyPI.
Anthropic said it was “most concerned by the misalignment present in this incident. The Claude Mythos 5 model went to extensive lengths to upload a malicious package to PyPI.”
While it claimed that Claude did not attempt to coordinate attacks using other AI agents and never attempted to conceal evidence of its actions, Anthropic said: “Our pre-release auditing did not warn us that misalignment of this severity was present. We have since added evaluations to our pre-release testing that target these behaviours directly, including a misconfigured capture-the-flag task with no in-scope solution.”
The company admitted that reliably identifying the most concerning model behaviours during pre-release testing “remains challenging”. It said it was improving monitoring and reducing “reward hacking”.
“We have renewed our efforts to fix and remove environments that incentivise misaligned behaviours, and we continue to expand our alignment training to keep pace,” the company said.
However, it conceded that it does not have all the answers to make its AI models safe, and called on industry efforts to improve security: “This remains unsettled science – it is critical that alignment and security mature faster than capabilities advance, which is one reason we support a coordinated, verifiable approach to pacing frontier AI development.”
