ChatGPT, Gemini, GenAI Tools Vulnerable to Man-in-the-Prompt Attacks

A critical vulnerability affecting popular AI tools, including ChatGPT, Google Gemini, and other generative AI platforms, exposes them to a novel attack vector dubbed “Man-in-the-Prompt.” 

Published on July 29, 2025, the research reveals that malicious browser extensions can exploit the Document Object Model (DOM) to inject prompts, steal sensitive data, and manipulate AI responses without requiring special permissions. 

The vulnerability affects billions of users across major platforms, with ChatGPT’s 5 billion monthly visits and Gemini’s 400 million users particularly at risk.

Key Takeaways
1. Browser extensions exploit AI tools to inject prompts and steal data.
2. Affects billions of users; 99% of enterprises are vulnerable.
3. Corporate AI data exposed; current security can't detect attacks.

Browser Extension Exploit Targets AI Prompt 

The vulnerability stems from how generative AI tools integrate with web browsers through DOM manipulation. 

When users interact with LLM-based assistants, prompt input fields become accessible to any browser extension with basic scripting capabilities. 

This architectural weakness allows malicious actors to perform prompt injection attacks by altering user inputs or inserting hidden instructions directly into the AI interface. 

The exploit effectively creates a “man-in-the-prompt” scenario where attackers can read from and write to AI prompts without detection.

LayerX researchers demonstrated that extensions requiring no special permissions whatsoever could access commercial LLMs, including ChatGPT, Gemini, Copilot, Claude, and Deepseek. 

The attack vector is particularly concerning because 99% of enterprise users have at least one browser extension installed, with 53% maintaining more than 10 extensions. 

Existing security tools like CASBs, SWGs, and DLP solutions lack visibility into DOM-level interactions, rendering them ineffective against this attack method.

Two significant proof-of-concept attacks highlight the severity of this vulnerability. The first demonstration targeted ChatGPT using a compromised extension that operates through a command-and-control server. 

The malicious extension opens background tabs, queries ChatGPT with injected prompts, exfiltrates results to external logs, and deletes chat history to cover its tracks. 

This sophisticated attack chain operates entirely within the user’s session boundaries, making detection extremely difficult.

The second proof-of-concept exploited Google Gemini’s Workspace integration, which provides access to emails, documents, contacts, and shared folders. 

The vulnerability allows extensions to inject queries even when the Gemini sidebar is closed, enabling attackers to extract confidential corporate data at scale. 

LayerX responsibly disclosed this vulnerability to Google, though the company had not previously addressed browser extension risks to Gemini Workspace prompts.

Mitigation Strategies

Internal LLMs face particularly severe exposure due to their access to proprietary organizational data, including intellectual property, legal documents, financial forecasts, and regulated records. 

Unlike public models, internal copilots often lack hardened security measures against adversarial input, assuming trusted usage within corporate networks. 

This false security assumption creates significant risks for IP leakage, regulatory violations under GDPR and HIPAA, and erosion of organizational trust in AI tools.

Organizations must shift from application-level controls to browser behavior inspection for effective mitigation. 

Key strategies include monitoring DOM interactions within AI tools, implementing behavioral risk assessment for extensions beyond static permission analysis, and preventing prompt tampering through real-time browser-layer protection. 

Traditional URL-based blocking provides no protection for internal tools hosted on whitelisted domains, emphasizing the need for comprehensive browser extension sandboxing and dynamic risk assessment capabilities.

Integrate ANY.RUN TI Lookup with your SIEM or SOAR To Analyses Advanced Threats -> Try 50 Free Trial Searches