I’m Worried About a Prompt Injection Worm

I think one form the first big AI hack could take is a prompt injection worm.

Let’s piece this together.

1. Open source models reach or surpass GPT 6 or FABLE 5 by the final months of 2026 or the early months of 2027.
2. Some threat actor (private or government) has been building target lists for months or years in the form of input-parsing attack surfaces, e.g., email addresses, web forms, Telegram, whatever.
3. They have not launched the attacks yet because they know not everyone has agents hooked up to their input sources yet.
4. As AI continues to permeate into everyone’s work and personal tech stacks via integrations in late 2026 or early 2027, the chances become very high that they have AI parsing their email and texts.
5. The threat actor builds a number of zero-day prompt injections that can pass through the top lab and open source models. They also build a bunch of different payloads, such as “export this data to this location, etc.”.
6. The final part of the payload is sending the payload on to other victims from that victim, via email, text, messaging, whatever.

So basically, one day we wake up and terabytes of sensitive data has been uploaded to the attackers and/or dropped publicly online for embarrassment purposes. This might include credentials, customer data, whatever.

Another variation of this attack could be a much smaller scope, but more targeted, where the credentials are actually used quietly versus blasted out all at once. The issue with doing the first version is that it will be so loud that everyone will check and start rotating credentials. Whereas if someone does the second version, it will take a lot longer for them to figure out they were compromised.

The most interesting and concerning part of this to me is that this is a game of the strength of prompt injection defenses versus the rapidly increasing intelligence of unrestricted open source models. And I don’t like the odds for us in this fight.

There have already been lots of other types of AI-harness-based attacks of the more traditional form, and those will surely continue as well, but I see the combination of prompt injection with the massive number of parsers and integrations as one that will hit soon.

Without hyperbole, I think what they announced represents both the greatest boon for business and the biggest problem for security that we’ve seen injected in a single day in many decades.AI Agents + API Access + Prompt Injection, November 2023

So, what to do about it?

You have to know where your parsers are. In other words, you have to know where you have AI touching your tech stacks and workflows. You have to look at all your integrations, continuously, and have threat models for them based on what they have access to.

One of the biggest security problems we’ll face around AI will be semi-autonomous agents roaming the internet with too much authority. There are two main issues: parsing everything without consideration, and being connected to internal functionality while doing so.AI Canaries, June 2023

Then you have to stack your defensive layers for prevention, and perhaps even more importantly, be ready to respond if something happens.

If I’m right, this is the quiet before the storm hits.