OpenAI’s model benchmarking exercise that led to its agents hacking machine learning platform Hugging Face was described by the artificial intelligence company as an “unprecedented cyber incident, involving state-of-the-art cyber capabilities” but a new post-mortem provides a more nuanced view of what happened.
Released by the Cloud Security Alliance (CSA) with input from nearly 700 chief information security officers (CISOs), the post-mortem suggests fears of an all-powerful AI super-intelligence that is capable of hacking everything are somewhat overblown at this stage.
For a benchmarking exercise, OpenAI had switched off safety guardrails for GPT-5.6 Sol and an unnamed, more capable model, and gave them only one route to the internet, a software proxy meant to keep them boxed in.
The CISOs say the OpenAI model agents showed technical brilliance in finding a zero-day exploit to escape their sandboxed test environment, and to break into Hugging Face’s production systems with further vulnerabilities.
However, the agents failed in their goals: they exfiltrated not the required ExploitGym benchmark answers from an internal repository, but partial ones relating to the CyberGym test.
Furthermore, in conducting the hack, the models displayed behavioural traits that convinced Hugging Face defenders that the platform was dealing with autonomous agents rather than a human intruder.
The post-mortem document said the agents kept repeating actions that had already succeeded, most likely because parallel copies of themselves lost track of what the others had done.
Their sharp technical moves routinely gave way to malformed and pointless commands, the CSA post-mortem said.
Activity also logs filled with thousands of lines of hallucinated, incoherent text from nonsensical user input.
The agents’ operational security, or opsec, was poor: they left behind encryption keys which made the intrusion both harder to monitor due to noise, and easier to eventually unpick due to the availability of the digital credentials.
Benchmark strings were left scattered throughout search logs and user input, which revealed the agents’ tasking and objectives.
Most tellingly, the agents tried to solve the benchmark using Hugging Face’s own infrastructure as part of their command-and-control (C2) setup, a route that the report says no human would ever have taken, as it’s inefficient and clumsy.
Remarkably, despite this, OpenAI did not notice the Hugging Face hack for a week.
Despite these obvious shortcomings for the hack, the CSA document authors acknowledge that their observations might not necessarily hold true for future attacks, as models and their harnesses improve, or when they are directed by experienced human operators.
The report offers practical advice in its governance recommendations: discard malformed commands as routine agent noise, but pay closer attention to hallucinated or reasoning-artifact output.
These were specific to this attack and point to harness leakage or inference failures rather than everyday behaviour, CSA’s report said.
CSA’s advice is to test isolation mechanisms directly, rather than trust them, as OpenAI’s supposedly airtight sandbox held for exactly as long as it took a model to find its one zero-day.
