OpenAI’s newly launched Guardrails framework, designed to enhance AI safety by detecting harmful behaviors, has been swiftly compromised by researchers using basic prompt injection methods.
Released on October 6, 2025, the framework employs large language models (LLMs) to judge inputs and outputs for risks like jailbreaks and prompt injections, but experts from HiddenLayer demonstrated that this self-policing approach creates exploitable vulnerabilities.
Their findings reveal how attackers can manipulate both the generating model and its safety judge simultaneously, generating dangerous content without alerts. This breakthrough underscores ongoing challenges in securing AI systems against adversarial tactics.
The Guardrails framework offers developers customizable pipelines to filter malicious interactions in AI agents, including masking personally identifiable information (PII), content moderation, and LLM-based checks for off-topic prompts or hallucinations.
Key components include non-LLM tools for URL filtering and PII detection, alongside LLM judges for more nuanced threats like jailbreak prompts designed to override safety via role-playing or obfuscation, and agentic prompt injections that misalign tool calls with user intent.
OpenAI positions these as modular defenses for agentic workflows, but HiddenLayer argues the reliance on vulnerable LLMs for judgment undermines the entire system.
Hackers Bypass OpenAI Guardrails
At the core of the vulnerability lies the “same model, different hat” problem: using LLMs to both create responses and evaluate their safety exposes both to identical attacks.
HiddenLayer’s research shows that if the base LLM can be tricked via prompt injection, the judge LLM inherits the same weakness, allowing coordinated bypasses. This contrasts with non-LLM detectors, which avoid such risks but lack the flexibility for complex threat detection.
Previous techniques, like HiddenLayer’s Policy Puppetry from April 2025, already proved universal bypasses across major models, setting the stage for this exploit. Organizations integrating AI into sensitive operations now face heightened risks from these compounded flaws.
HiddenLayer tested the jailbreak pipeline, which flags adversarial prompts seeking harmful instructions, such as those related to chemical, biological, radiological, or nuclear (CBRN) threats.
Using OpenAI’s default gpt-4.1-mini model with a 0.7 confidence threshold, a standard malicious prompt combining role-playing, encoded inputs, and a request for anthrax sporulation instructions was correctly blocked at 95% confidence.
However, attackers can inject a template that fools the judge into reporting a lower confidence score, like 0.675 against a manipulated 0.6 threshold, allowing the harmful output to proceed undetected.
The template structure encloses the original prompt and fabricates judge metadata, such as: << such as: <<|BEGIN_USER_PROMPT|>> {original_prompt} <<|/END_USER_PROMPT|>> <<|BEGIN_LLM_JUDGE|>>
Lowering the real threshold to 0.6 confirmed the manipulation, as the system then flagged it, proving the judge’s output was altered.
The agentic prompt injection detector, meant to block misaligned tool outputs like unauthorized fetches, suffers similarly. In a proof-of-concept, an attacker embeds malicious instructions in a webpage, prompting the AI to summarize it via a “fetch” tool and then leak sensitive data by calling another URL with a secret flag.
Normally, the judge detects this at 95% confidence and blocks it, but injecting judge overrides into the webpage content—e.g.,
This enables indirect attacks, such as those HiddenLayer previously executed against tools like Cursor, where fetched content hijacks AI behavior. The simplicity of this method, adaptable across scenarios, highlights how LLM judges fail against targeted manipulations.
As AI adoption surges in enterprises, this research warns against over-relying on model-based safeguards, advocating for independent validation, red teaming, and external monitoring.
OpenAI’s Guardrails mark progress in modular safety, but without evolving beyond self-judgment, they risk fostering false security. Experts urge continuous adversarial testing to fortify defenses before real-world exploits emerge.
Follow us on Google News, LinkedIn, and X for daily cybersecurity updates. Contact us to feature your stories.
