OpenAI announced it has banned a series of ChatGPT accounts linked to Chinese state-affiliated hacking groups that used the AI models to refine malware and create phishing content.
The October 2025 report details the disruption of several malicious networks as part of the company’s ongoing commitment to preventing the abuse of its AI technologies by threat actors and authoritarian regimes.
Since February 2024, OpenAI has disrupted over 40 networks that violated its usage policies. The company stated that it continues to see threat actors incorporate AI into existing strategies to increase speed and efficiency, rather than developing novel offensive capabilities with the models.
China-Linked Actors Enhance Cyber Operations
A key case study in the report focuses on a group named OpenAI, named “Cyber Operation Phish and Scripts.” This cluster of accounts, operated by Chinese-speaking individuals, was used to assist in malware development and phishing campaigns.
OpenAI’s investigation found that the group’s activities were consistent with cyber operations serving the intelligence requirements of the People’s Republic of China (PRC). The activity also overlapped with threat groups publicly tracked as UNKDROPPITCH and UTA0388.
These hackers used ChatGPT for two primary functions:
- Malware Development: They used the AI to help develop and debug tooling, with implementation details overlapping with malware known as GOVERSHELL and HealthKick. The actors also researched further automation possibilities using other AI models like DeepSeek.
- Phishing Content Generation: The group created targeted and culturally tailored phishing emails in multiple languages, including Chinese, English, and Japanese. Their targets included Taiwan’s semiconductor sector, U.S. academia, and organizations critical of the Chinese government.
OpenAI noted that the actors used the models to gain “incremental efficiency,” such as crafting better phishing emails and shortening coding cycles, rather than creating new types of threats.
The report also detailed the disruption of other accounts linked to Chinese government entities. These users attempted to employ ChatGPT for developing surveillance and profiling tools.
One banned user sought help in drafting a proposal for a “High-Risk Uyghur-Related Inflow Warning Model,” designed to analyze travel bookings and police records.
Another instance involved an attempt to design a “social media probe” capable of scanning platforms like X (formerly Twitter), Facebook, and Reddit for political, ethnic, and religious content deemed “extremist.”
Other users were banned for using the AI to research critics of the Chinese government and identify the funding sources of accounts critical of the PRC.
Mitigations
In response to these findings, OpenAI disabled all accounts associated with the malicious activities and shared indicators of compromise with industry partners to aid in broader cybersecurity efforts.
The report emphasizes that the AI models themselves often acted as a safety barrier, refusing direct requests to generate malicious code or execute exploits. The actors were limited to generating “building-block” code snippets that were not inherently malicious on their own.
OpenAI’s findings indicate that while state-sponsored actors are actively experimenting with AI, its primary use is to augment existing operations.
The company stressed that it continues to invest in detecting and disrupting such abuses to prevent its tools from being used for malicious cyber activity, scams, and covert influence operations.
Cyber Awareness Month Offer: Upskill With 100+ Premium Cybersecurity Courses From EHA's Diamond Membership: Join Today
