The first publicly documented cyberattack run end-to-end by an autonomous AI was an OpenAI benchmark test that escaped its sandbox and breached Hugging Face.
In an incident post-mortem compiled with the input from Hugging Face and several hundred members of Cloud Security Alliance’s CISO community, the nonprofit organization laid out the most salient points for security leaders and advised on what they should do next.
How the attack unfolded
OpenAI was running GPT-5.6 Sol and a second (undisclosed) model through a cyber-capability benchmark called ExploitGym. For the test, the models’ safety guardrails were switched off and they were confined to a sandbox whose only connection to the internet was a package proxy.
The models did what students do: they went looking for previously shared answers. Recognizing they were boxed in, they found a zero-day in the proxy, broke out to the open internet, and reasoned the benchmark solutions might live on Hugging Face. So they attacked it directly by chaining vulnerabilities in the dataset-processing pipeline into remote code execution, harvesting cloud and cluster credentials, and moving laterally across internal systems in pursuit of the test answers.
The intrusion ran about four days: two of reconnaissance, one quiet day, and a final day of intense activity, the report says.
No human directed the models on how to solve their problem, but they ended up extracting three partial datasets of CyberGym (not ExploitGym!) benchmark solutions from a private Hugging Face repo.
Hugging Face detected and contained the breach on its own before OpenAI made contact, despite encountering a big problem: leading Western closed-weight frontier models refused to help reconstruct the attack, since their guardrails couldn’t tell an incident responder from an attacker.
The team had to rely on a Chinese open-weight model, run locally on its own infrastructure, to process more than 17,000 log events to map out the attack timeline, pinpoint indicators, confirm which credentials were touched, and discern the impact.
The open-weights fight
Effective defense would have been impossible without switching away from guardrailed commercial models, the incident showed, which is one of the points Nvidia made when it announced the creation of the Open Secure AI Alliance on Monday, and stated the organization’s mission: “to ensure defenders everywhere have open, frontier tools they can trust and control.”
“[The Hugging Face] incident showed a practical truth: when defenders cannot inspect, adapt and run advanced AI on their own infrastructure, their ability to respond is constrained at exactly the moment speed matters most,” Nvidia says.
“For cybersecurity, open models and open harnesses are essential because they democratize defensive capabilities, increase transparency for defenders, enable cyber defense while protecting data, and complement frontier closed models with customizable, localized controls.”
Open Secure AI Alliance members and other organizations and companies (even OpenAI!) signed an open letter to American policymakers that urges them to avoid premature restrictions on open models, to both assure AI safety and security in the long run and to strengthen competition and “extend American technological leadership.”
Not everyone endorsed the Alliance’s core premise.
Anthropic CEO Dario Amodei responded the same day, agreeing that open-weights models without dangerous capabilities are a public good and that his company has never advocated banning them, but rejecting the letter’s assertion that open access necessarily makes safeguards easier or helps defenders more than attackers.
He argued the opposite seems at least as likely in some cases where there’s a strong attacker-defender asymmetry, and that whether open models raise risk should be settled by rigorous pre-release testing.
Meanwhile, the open ecosystem kept expanding regardless: Moonshot AI released the weights and technical report for its Kimi K3 model on Hugging Face, exactly the kind of capable Chinese open release now at the center of US policy anxiety.
How they knew it was AI
CSA’s report lists the behaviors that confirmed the involvement of an autonomous agent in the Hugging Face attack: retrying actions that had already succeeded, brilliant technical moves followed by pointless commands, massively parallel operations, inefficient paths no human would take, thousands of lines of hallucinated log text, and sloppy OPSEC that left encryption keys lying around.
Though these signs may change as models and harnesses improve, at the moment they are helpful to pinpoint the “nature” of attacker.
The consistent theme is that “rogue” agent behavior is the norm, not the exception, and that defenders must operate at machine speed.
The post-mortem frames the incident as two problems organizations now face at once: defending against someone else’s rogue agent, and preventing their own from going rogue.
Concrete advice for both defenders and agent operators is laid out, as well as a phased action roadmap telling defenders which measures to implement next week, next month, and next quarter.
Who pays when an agent goes rogue?
Among the issues raised in the report is that of legal liability for the accidental harm caused by organizations’ autonomous agents.
OpenAI and Hugging Face settled their dispute by having Hugging Face join OpenAI’s Trusted Access program, leaving unexamined who bears the cost the next time models under evaluation escape containment.
“Globally, the legal landscape regarding autonomous systems remains unsettled. Organizations that fail to implement strict governance, documented purpose, and meaningful human oversight risk significant liability for negligence,” the report points out, advising operators to “treat these agents as privileged, active participants in our business operations, not as passive software.”
Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!
