As legitimate businesses purchase AI tools from some of the largest companies in the world, cybercriminals are accessing an increasingly sophisticated underground market for custom LLMs designed to assist with lower-level hacking tasks.
In a report published Tuesday, Palo Alto Networks’ Unit 42 looked at how underground hacking forums advertise and sell custom, jailbroken, and open-source AI hacking tools.
These programs are sold on dark web forums, advertised as either explicit hacking tools or dual-use penetration testing tools. Some offer monthly or yearly subscriptions, while others appear to be copies of commercial models trained on malware datasets and maintained by dedicated communities.
The models provide foundational capabilities around certain tasks that could be helpful to both hackers and cybersecurity defenders alike, like scanning for vulnerabilities in a network, encrypting data, exfiltrating data, or writing code.
Andy Piazza, senior director of threat intelligence for Unit 42, told CyberScoop that as AI tools have improved, their dual use nature in cybersecurity has become clearer.
“You know, Metasploit is a good guy framework, and it can be used by bad guys,” said Piazza. “Cobalt Strike was developed by good guys and now unfortunately bad guys have cracked it and used it as well. And now we’re seeing the same thing with AI.”
The report highlights two recent examples.
Starting in September, a new version of WormGPT appeared on underground forums. The jailbroken LLM first emerged in 2023 before its developers went underground amid heightened scrutiny and media reporting. This year a newer version reemerged, advertised as a hacking tool that would offer LLM capabilities “without boundaries.”
The original WormGPT claimed to be trained on malware datasets, exploit writeups, phishing templates, and other data meant to finetune its hacking assistance. The model and architecture behind the newer version (WormGPT4) remains unknown.
Unit 42 researchers said this updated version “marks an evolution from simple jailbroken models to commercialized, specialized tools to help facilitate cybercrime,” offering cheap monthly and annual subscriptions. Lifetime access costs as little as $220, with an option to purchase the full source code.
“WormGPT 4’s availability is driven by a clear commercial strategy, contrasting sharply with the often free, unreliable nature of simple jailbreaks,” the report noted. “The tool is highly accessible due to its easy-to-use platform and cheap subscription cost.”
Another model, KawaiiGPT, is free on GitHub with a lightweight setup that took “less than five minutes” to configure on Linux. It advertises itself as “Your Sadistic Cyber Pentesting Waifu.”
While likely a copy of an open-source or older commercial AI model, it “represents an accessible, entry-level, yet functionally potent malicious LLM.” It uses a casual tone, greeting users, with comments like “Owo! Okay! Here you go….” while delivering malicious outputs.
“While its code for attack functions might be less complex than the more optimized PowerShell scripts generated by WormGPT 4, KawaiiGPT instantly provides the social and technical scaffolding for an attack,” the report claimed.
Like many open-source tools, KawaiiGPT also has a dedicated community of around 500 developers who update and tweak it to maintain effectiveness.
Piazza has concerns about these AI tools’ availability and their impact on the cybercriminal ecosystem, but he joked they’re less about “AI lasers dropping malware in our networks” or other overhyped threats.
The capabilities described in the report fall below those seen in recent incidents, like a hacking campaign identified by Anthropic that automated large portions of successful cyber attacks. Piazza noted real limitations with the models being sold on the underground market. For example, While LLMs may generate malware faster, internal tests at Palo Alto Networks found that most of the code is easily detectable.
The real danger, he said, is that the report confirms what cyber professionals have warned about since LLMs first emerged: their potential to make criminal hacking easier and less technical.
“It’s just that interoperability,” said Piazza. You don’t even have to be good with the terminology. You don’t even have to use the word ‘lateral movement,’ when using these tools. You can just ask ‘How do I find other systems on the network?’ and it can drop you out a script. So that barrier to entry: lowering and lowering.”