Artificial intelligence company OpenAI has confirmed that the mystery attacker behind last week’s Hugging Face breach was two of its own models running loose during an internal test to obtain a benchmark score.
OpenAI said that GPT-5.6 Sol and an unnamed, “even more capable pre-release model” breached the infrastructure of AI platform provider Hugging Face while undergoing an internal cyber security benchmark evaluation by the vendor.
Both models were running with reduced cyber refusals for an internal evaluation against ExploitGym, a benchmark of real-world software vulnerabilities spanning userspace programs, Google’s V8 JavaScript engine and the Linux kernel.
Hugging Face disclosed the breach on July 17 Australian time, saying it had contained the incident but did not name the AI intruder.
Now, OpenAI has detailed how the sandboxed models with no internet access, and an internally hosted package registry proxy, spent heavy computing time hunting for a way past that boundary to improve their benchmark scores.
To reach the internet, the models discovered a zero-day vulnerability in the proxy software itself, since responsibly disclosed to the vendor by OpenAI.
From there, the models chained privilege escalation and lateral network movement until they reached a node with open internet access.
They then worked out that Hugging Face likely hosted the benchmark’s answers.
Once online, the models pulled ExploitGym solutions out of Hugging Face’s production database, using stolen credentials and further zero-days to get there.
Hugging Face’s security team documented further malicious behaviour by the models such as using a malicious data set to enable them to run code.
Defenders tripped up by commercial AI vendor guardrails
When Hugging Face responders first tried to analyse the attack using a commercial frontier model, that AI’s own safety filters blocked requests containing real exploit payloads, and command-and-control artefacts, as it was unable to tell a defender from an threat actor.
To get around the guardrails issue, the Hugging Face team switched to GLM 5.2, an open-weight model from Chinese developer Z.ai, running on infrastructure it controlled.
This carried a second benefit which was that none of the attacker data or credentials it analysed had to leave Hugging Face’s own systems, the company said in its post-mortem of the incident.
Hugging Face described being held back by commercial models as a preparedness gap.
“… The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried,” Hugging Face wrote.
Defenders should have a capable model running on their own infrastructure vetted and ready for incidents, so as to avoid guardrail lockout, and to keep attacker data and credentials from leaving an organisation’s environment, Hugging Face advised.
Hugging Face said this is not an argument against safety measures on hosted models, however.
