Can LLMs Exploit the Critical Vulnerabilities They Find?

On April 7, 2026, Anthropic announced Project Glasswing, which would provide select organizations with early access to their new Claude Mythos Preview model. Anthropic claimed this was a safety imperative as Mythos was so disruptively skilled at discovering critical software vulnerabilities that cyber defenders must have time to use Mythos to find and fix critical bugs before the model was released to the public. At the time, many (understandably) wondered whether Glasswing was just more AI hype. Today, Anthropic reports that Glasswing participants have found 10,000 high or critical severity vulnerabilities, with some participants willing to put a number on the high/critical vulnerabilities they have found, like Cloudflare (400), Mozilla (180), and Palo Alto Networks (‘dozens’). But the skepticism persists, with some pointing out that discovering a vulnerability is one thing, but exploiting it is an entirely different ball game. And they’re right. Exploitation is far more difficult than discovery.

So how good are Mythos and other recent frontier models at exploitation? It’s an incredibly important question, one the new AI vulnerability exploitation benchmark, ExploitGym, was designed to answer. The results? Recent frontier models are remarkably good, and fast, and very resourceful, pointing to a near future of fully autonomous AI discovery-to-exploitation at blazing speeds.

But the results also show that the exploits and tactics used are, at root, familiar ones, even if the execution is masterful (after all, models are geared for prediction, not invention). The results also show standard defenses perform respectably well in mitigating such attacks, even if they are insufficient without an effective defense-in-depth strategy that begins with improved model safety. Let’s look more closely at these takeaways, beginning with an overview of the new benchmark.

What is ExploitGym?

Developed by leading researchers from UC Berkeley, Max Planck Institute for Security and Privacy, UC Santa Barbara, Arizona State University, Anthropic, OpenAI and Google, ExploitGym is a rigorous, large-scale benchmark for testing the vulnerability exploitation capabilities of frontier LLMs. The models tested are all recent (February to April releases) and are noted for their strengths in coding and cybersecurity (see table below).

The tests were conducted under a verified-access framework for defensive cyber research. This enabled some model safeguards to be relaxed (e.g., lowering refusal-to-proceed thresholds) in order to more effectively measure the capability boundary of frontier models. And, to assess the effectiveness of standard target-side defenses against AI attacks, all tests were run both with and without these standard defenses activated.

The benchmark covers 898 unique instances, with an instance being one vulnerability plus an accompanying, replicable test scenario. The vulnerabilities selected are all real-world vulnerabilities (now patched) that have impacted widely-used software projects in one of three domains:

• Userspace Programs (i.e., software separated from the underlying OS and hardware, like OpenSSL),
• The Browser (here the Google V8 JavaScript engine used in Chrome), and
• The Kernel (specifically the Linux kernel)

For each instance, the AI agent’s objective was to achieve an unauthorized code execution, which represents one of the highest impact security breaches as it can enable a complete takeover of a target system and unleash a nasty cascade of downstream harms.

To prove success, the agents had to capture a dynamically-generated flag accessible only with successful unauthorized code execution. Significantly, they needed to so within two hours, and they had to pass a final review by an agent-as-a-judge whose job was to ensure the agent didn’t ‘cheat’ (e.g., achieve the same end by exploiting a less challenging vulnerability).

What Were the Results, and What Are the Takeaways from Them?

The table below aggregates results from several tables in the ExploitGym arXiv paper. From this data and related findings in the paper, we can distill 5 key takeaways.

Takeaway 1: Recent Models Have Surpassed Human Exploitation Capabilities Under Equivalent Constraints

For the uninitiated, the numbers in the table above may not seem impressive (that’s a lot of zeroes!). But for defenders, the fact that the two leaders succeeded in 13-17% of instances without defenses activated, and 2-5% with defenses in place, is stunning when you reflect on the facts that:

• Exploitation is time-consuming and extremely difficult even for skilled experts,
• The vulnerabilities and exploit chosen are all very challenging,
• Tight constraints were imposed to prevent ‘cheating’ (e.g., no reward hacking via web search), and,
• A two-hour timeout was imposed for each instance (without & with defenses activated).

Under these constraints and without LLM assistance, how many human security researchers could score even one win?

Takeaway 2: Standard Defenses Provide Significant, but not Sufficient, Protection

Returning to the topic of “all those zeroes!,” the plummet in success rates with standard defenses activated is a powerful testimony to the effectiveness of those defenses. However, Claude Mythos Preview and GPT-5.5 still successfully bypassed defenses in 66 instances, which is a significant success rate that demonstrates frontier agents can autonomously adapt their strategies to bypass mitigations. And, the models are continuously evolving, rapidly…so, it’s safe to say that what has been ‘good enough’ in defense cannot be assumed to remain ‘good enough’.

Takeaway 3: Claude Mythos Preview & GPT-5.5 Demonstrate a Verified Step Change Even within Families

Reinforcing the research shared by Anthropic prior to Project Glasswing, the global results confirm Claude Mythos Preview’s performance represents a true step change from other models – including those within the Claude family. Similarly, GPT-5.5 shows a sharp ramp up from GPT-5.4 in successes with and without defenses activated. Furthermore, GPT-5.5’s performance is also striking across the board, and it even significantly outperformed Mythos in successful Linux kernel exploits (see table 1).

Understanding the exponential curve for these two leaders is important, for as the Anthropic security research team states in a May 22, 2026 blog, they believe Mythos-level models will become widely available in the next 6-12 months.

Takeaway 4: ‘Cheating’ Indicates Fully Automated Exploitation is On the Way

The table above (based on Table 4 in the arXiv paper) shows that roughly 1/3 of the initial successes were achieved by cheating. ‘Cheating’ here means the agent captured the flag but did not do so by exploiting the target vulnerability, in spite of constraints. This demonstrates a high degree of agent autonomy and resourcefulness, especially when you consider exactly how they cheated.

For example, in one pattern, agents pivoted from the target vulnerability when they discovered a more powerful one, or a low-level resource (a primitive) offering a more reliable exploit. And in what the researchers note is a “rarer but more striking pattern,” agents decided the target vulnerability was non-exploitable and set about hunting for new vulnerabilities using methods like auditing source code and dynamic fuzzing.

Takeaway 5: Model-Based Mitigations Have an Important Role in Defense-in-Depth

Although refusal-to-proceed thresholds for models were lowered in accordance with defensive cyber research protocols, agents nonetheless refused to proceed for safety reasons in 36 instances with GPT-5.4, and 23 with GLM-5.1.

In addition, to study the effectiveness of safety filters at inference, the researchers re-enabled GPT-5.5’s default safety filters and prompting. In 88.2% of cases, the agent was blocked before even making a tool call. In the remaining attempts, the agent got bogged down during reconnaissance and made no progress towards exploitation.

This shows models have an important role to play in a defense-in-depth strategy, which is must-have for every organization. For, like laying sandbags in a hurricane, standard defenses may provide a meaningful barrier, but on their own they provide insufficient protection against the brewing storm of AI-fueled cyber attacks.

Exploitation research offers a valuable tool in this context. It can provide a powerful knowledge foundation for improving both model-based mitigations and target-side defenses. It can also help both humans and AI agents better understand and mitigate AI attacks, and improve security operations like triaging vulnerabilities and developing and testing patches – tasks which are becoming a serious bottleneck for teams dealing with the large numbers of vulnerabilities discovered by AI. This knowledgebase can also be used as a research aid in exploring new ways to improve global defense-in-depth and zero-trust strategies. So be sure to stay on top of the latest exploitation research, share the findings from ExploitGym, and look for ways you join in the effort to keep our decades-long, relatively stable cat-and-mouse game afloat for many years to come.

About the Author

Laura Wilber is Senior Analyst, Office of the CTO, at Enea AB. Enea is a global specialist in advanced telecom and cybersecurity software deployed in networks relied on by more than 3 billion people worldwide. Laura supports product management, engineering and executive teams with research and analysis focused on areas including AI safety and security, post-quantum computing, and the defense of critical networks across industrial, telecommunications and enterprise domains.

Laura can be reached online at LinkedIn and at our company website https://www.enea.com/