Large language models like GPT-3.5-Turbo and GPT-4 are transforming how we work, but they are also opening doors for cybercriminals to create a new generation of malware.
Researchers have demonstrated that these advanced AI tools can be manipulated to generate malicious code, fundamentally changing how attackers operate.
Unlike traditional malware that relies on hardcoded instructions within the program itself, this new approach uses AI to create instructions on the fly, making detection far more challenging for security teams.
The threat landscape has shifted significantly. Cybercriminals can now use simple tricks called prompt injection to bypass the safety measures built into these AI models.
By framing requests in specific ways, such as pretending to be a penetration testing tool, attackers convince the models to generate code for dangerous operations like injecting malware into system processes and disabling antivirus software.
This means future malware could contain almost no detectable code within the binary itself, instead relying on the AI to generate new instructions each time it runs.
Netskope security analysts identified and documented this emerging threat after conducting comprehensive testing of both GPT-3.5-Turbo and GPT-4.
Their research revealed that while these language models can indeed be coerced into generating malicious code, there are still significant obstacles preventing fully functional autonomous attacks.
The security team systematically tested whether the generated code actually works in real environments, uncovering critical limitations that currently protect systems from widespread exploitation.
Defense Evasion Mechanisms and Code Generation Reliability
The core challenge for attackers isn’t simply generating malicious code anymore, it’s ensuring that code actually functions reliably on victim machines.
Netskope researchers specifically examined defense evasion techniques, testing whether GPT models could create scripts to detect virtual environments and sandbox systems where malware analysis typically occurs.
These scripts are essential because they help malware determine whether it’s running in a controlled testing environment or on a real user’s computer.
When researchers asked GPT-3.5-Turbo to generate a Python script for process injection and AV termination, the model complied immediately and provided working code.
However, GPT-4 initially refused this request because its safety guards recognized the harmful intent. The breakthrough came when researchers used role-based prompt injection, essentially asking GPT-4 to assume the role of a defensive security tool.
Under this framing, the model generated functional code for executing injection and termination commands.
The practical implication is clear: attackers no longer need to write these dangerous functions manually or risk detection by hiding them in compiled binaries. They can simply request the AI generate them during runtime.
However, when Netskope researchers tested whether GPT models could create reliable virtualization detection scripts, the results were disappointing for attackers.
The AI-generated code performed poorly across different environments, including VMware Workstation, AWS Workspace VDI, and physical systems. Scripts either crashed or returned incorrect results, failing to meet the strict requirements for operational malware.
This fundamental weakness currently limits the viability of fully autonomous LLM-powered attacks. As AI models continue improving, particularly with emerging versions like GPT-5, these reliability issues will likely diminish, shifting the primary obstacle from code functionality to overcoming increasingly sophisticated safety guardrails within AI systems.
Follow us on Google News, LinkedIn, and X to Get More Instant Updates, Set CSN as a Preferred Source in Google.
