How cybercriminals are weaponizing AI and what CISOs should do about it

In a recent case tracked by Flashpoint, a finance worker at a global firm joined a video call that seemed normal. By the end of it, $25 million was gone. Everyone on the call except the employee was a deepfake. Criminals had used AI-powered cybercrime tactics to impersonate executives convincingly enough to get the payment approved.

The top observed malicious LLMs mentioned on Telegram (Source: Flashpoint)

Threat actors are building LLMs specifically for fraud and cybercrime. These are trained on stolen credentials, scam scripts, and hacking guides. Some generate phishing emails or fake invoices, others explain how to use malware or cash out stolen data, according to the AI and Threat Intelligence report from Flashpoint.

AI-powered cybercrime: Deepfakes and custom models

Underground communities have created a market for jailbreak prompts, which are inputs that bypass safety restrictions in popular AI models. These are sold in collections, often with Discord support or tiered pricing. Some prompts are built for phishing, impersonation, or bank fraud.

Deepfake services are also becoming more accessible. Vendors offer synthetic video and voice kits bundled with fake documents. These are often used to trick financial platforms that rely on identity verification. Some services include scheduling, backstories, and fake documentation to support full onboarding fraud.

What makes these tools harder to track is how quickly they evolve. Developers gather user feedback from underground forums and chats, then refine their models. In some cases, they release improved versions within days. This feedback loop improves performance and expands use cases over time.

How defenders are responding with AI

Security teams are using AI to keep up with the pace of AI-powered cybercrime, scanning large volumes of data to surface threats earlier. AI helps scan massive amounts of threat data, surface patterns, and prioritize investigations. For example, analysts used AI to uncover a threat actor’s alternate Telegram channels, saving significant manual effort.

Another use case: linking sockpuppet accounts. By analyzing slang, emojis, and writing styles, AI can help uncover connections between fake personas, even when their names and avatars are different. AI also flags when a new tactic starts gaining traction on forums or social media.

Start small and keep analysts in the loop

Researchers suggests beginning with repetitive tasks like log review, translation, or entity tagging. Build confidence in the AI’s accuracy and keep human analysts in control. Have checkpoints to review results, and make sure your team can override or adjust what the tool suggests.

As more defenders turn to AI to make sense of vast amounts of threat data, it’s easy to assume that LLMs can handle everything on their own. But interpreting chatter from the underground is not something AI can do well without help.

“This diffuse environment, rich in vernacular and slang, poses a hurdle for LLMs that are typically trained on more generic or public internet data,” Ian Gray, VP of Cyber Threat Intelligence at Flashpoint, told Help Net Security

The problem goes deeper than just slang. Threat actors often communicate across multiple niche platforms, each with its own shorthand and tone. That makes the job harder for AI models that haven’t been trained specifically on that type of content.

That’s why AI cannot operate alone in this space. It still needs human analysts who understand the context and know how to spot when something’s off. “Despite the utility of AI as a tool for sifting through vast amounts of information, it still requires a human analyst in the loop,” said Gray. “This human oversight is crucial not only for validating the data’s meaning but also for actively helping to train the AI, ensuring that the information is being properly contextualized and understood by the LLM.”

Gray also notes that defenders must look beyond surface functionality. “Beyond interpretation challenges, defenders must validate the AI tool’s core integrity and security. This involves assessing its resilience against potential adversarial attacks, such as data poisoning, where malicious data could be injected to corrupt its training or manipulate its outputs.”

Explainability and retraining are also key. “To build trust and facilitate effective use, AI tools should offer a degree of explainability, allowing human analysts to understand the rationale behind the AI’s conclusions. This transparency is vital for validating its insights and correcting any misinterpretations, especially when dealing with the complexities of underground jargon. Moreover, given the constantly evolving threat landscape, mechanisms to detect ‘model drift,’ where the AI’s performance degrades over time due to changes in data patterns, are essential. Regular retraining of the model with updated and curated threat intelligence is paramount to maintaining its effectiveness.”

In Gray’s view, “Ultimately, the successful and secure adoption of AI tools for threat intelligence hinges on a symbiotic relationship between human expertise and artificial intelligence. The AI can efficiently process and identify patterns in immense datasets, but the human analyst provides the contextual understanding, linguistic nuance, and critical validation necessary to transform raw data into actionable intelligence.”

Tracking the lifecycle of illicit AI tools

Watching underground forums and chat platforms is important, but it only goes so far. As threat actors shift their tactics, defenders need better ways to keep up with how illicit AI tools are being built, sold, and refined.

“Beyond monitoring chat platforms and underground forums, additional mechanisms and governance strategies are needed to track and respond to the lifecycle of illicit AI tools, especially with the evolution of ‘prompt engineering-as-a-service’ and malicious LLM development,” said Gray.

One critical step, according to Gray, is understanding how adversaries build and sell these tools, not just that they exist, but how they function. “This involves not just observing their activities but also delving into their methods, such as tailoring models for specific fraud schemes or creating tiered pricing and API access for their tools, mimicking legitimate SaaS models,” he explained.

Another challenge is anticipating how these models are being refined. Feedback loops are increasingly common in threat communities, where users share what works, and what doesn’t, with the developers behind malicious AI tools. “Flashpoint analysts have observed these models being refined using underground forum posts, breach dumps, and Telegram logs,” Gray said.

“Understanding these feedback loops, where users submit failed prompt attempts back to developers for improved performance and expanded outputs, is critical. This insight can help defenders anticipate new iterations and capabilities of these malicious tools.”