Over the past few years, AI has gone from something understood and used by a select few to a cornerstone of almost every organisation. As businesses race to adopt new models, tools and autonomous agents, security and governance are struggling to keep up.
Recent incidents involving AI agents behaving in unexpected ways have offered an early glimpse of what could happen as these systems are given greater autonomy. Closer to home, organisations face a more immediate problem: employees are adopting AI tools faster than many security teams can identify, assess or control them, a point CultureAI’s Founder and CEO, James Moore, made in a recent interview with the Guru.
That raises two questions: who is responsible for AI security and governance, and do organisations really know the full extent of their AI risk?
The IT Security Guru spoke to Oliver Simonnet, Lead Cybersecurity Researcher at CultureAI, about models escaping their sandbox, shadow AI, why people overshare with LLMs, who should be holding the industry accountable, and what organisations should put in place before the next wave of adoption.
Q: How are the advances in AI changing the cyber threat landscape for attackers and defenders?
There’s nothing particularly novel happening in the AI space outside of the recent cases of models escaping their sandbox and being used to attack other organisations. Even then, it’s mostly an acceleration of what already existed.
For attackers, there’s the more typical stuff: AI lowering the bar to entry for cybercriminals and helping them become more efficient as the models become more capable. Cybercriminals are using the more traditional LLMs to speed up malware development, and to build AI capability into malware for command and control or for modifying code.
On the defence side, it’s a double-edged sword. We’ve got people rushing to adopt AI to solve problems and to aid investigations. But each time we adopt another AI technology, or embed something in an organisation, we’re also opening up all of that risk of the unknown. No one really knows how any of this works, typically, unless people want to do the due diligence and the research. You can do that, but at the speed people are adopting things, you adopt it, you deploy it, and then you hope for the best.
You adopt it, you deploy it, and then you hope for the best.
So, it’s hard for defenders to adopt and leverage these technologies without putting themselves at too much risk, or causing too many complications.
That said, AI is being incorporated into everything now, and it very clearly helps with speeding up incident response and investigations, fixing bugs in code, removing vulnerabilities and getting AI to assess things automatically.
It’s a very different world to the one we were in a couple of years ago.
Q: Why do people feel so comfortable putting sensitive information into an LLM, as opposed to a search engine?
The difference is that a traditional search engine like Google, at least before Google Search was also AI-driven, doesn’t rely on a large amount of information. The typical way we learned to use search engines was concisely, if anything. Single words at best. You put in exactly what you want, the minimum amount of typing, and ideally you get the result you’re after.
Whereas the way AI has evolved in the LLM space, it thrives, and is most efficient, when it’s given the most information it can possibly get.
So it encourages a feedback loop. Users give it as much information as they can because they know that gets them the best answer, and if the answer is not as good, they’ll probably give it even more. The less information you give it, the less reliable the outcome is ultimately going to be.
That creates a practice of, one, giving it a lot of information. And two, I think the conversational nature of it, from a psychological perspective, creates a level of trust or familiarity with the technology that lowers people’s guard considerably. It gets you chatting away, forgetting what the implications are of what’s actually going on.
You’re just more willing to have a full-blown conversation about your personal issues on your business ChatGPT account, without thinking about the fact that you’re putting all of that into third-party infrastructure, and doing it through your business account.
Then what happens one day if that information is accessed and compromised? It’s a lot worse. You probably don’t want your Google searches compromised, I’m sure, but you definitely don’t want your ChatGPT logs compromised.
Q: Employees are being encouraged to innovate and experiment with AI. How can organisations enable that without simply blocking everything?
Having visibility of what AI is being used, and how it’s being used, is key. Without that information, organisations can’t see what’s going on, know whether they’re at risk, or understand whether the tools people are using are any good.
Without visibility, they can’t quantify the risk either. Someone could ask you what risks you think you have in your business from AI adoption, but if you’ve got nothing to go by outside of “well, people are using it and everyone seems happy”, you’re going to say the risk is basically nothing, because everything seems fine.
In reality, if you have the visibility and you can see that people are sending your confidential information out left, right and centre, or disclosing private medical information into third-party applications and breaking regulations in the process, then your perceived risk is going to skyrocket, and you’re going to be worried about a fine.
Someone could ask you what risks you have from AI adoption, but if you’ve got nothing to go by, you’re going to say the risk is basically nothing, because everything seems fine.
So, adopting tools that give you visibility of general AI use matters, as does anything people can do to contain agents or deploy things locally.
Organisations also need to show some restraint when they adopt things, and not get caught up in the excitement. Take a beat, establish your controls and processes first, and then you’ve got something in place rather than immediately exposing yourself to a higher amount of risk.
Q: Who should ultimately be responsible for ensuring organisations are prepared for these risks: businesses, regulators or governments?
AI is a global problem, which makes governance particularly complicated.
The issue with established frameworks, compliance regimes and standards is that organisations can decide to adopt them or not at their own risk, depending on things like their own exposure and how public they are. Frameworks establish a baseline, but they aren’t a legal requirement.
Some industries rely heavily on mandates. Financial organisations know that during due diligence, potential partners will check for certain certifications and baseline standards before working with them. If you decide you want to be a payment processor, even if you’re just an individual, you need to make sure you’re following all the right approaches. Global standards like PCI DSS exist to protect payment card data and govern how cardholder information is handled, and if you don’t adhere to them as a financial organisation, you can be heavily fined.
I feel like there should be something like that for AI. We need one of the central bodies, whether that’s a government or an individual trade body, making firmer decisions on AI security and holding the wider industry accountable.
Whether it ends up with government depends on how the industry responds. If nobody takes the voluntary route, that’s the point at which the government should probably step in and legislate on what the outcomes are if you’re negligent in this space.
It’s slightly too early to say where that lands. I think there’s still enough time for industry regulation and standards to keep up. In the meantime, though, organisations are responsible for making sure they do everything right.
Q: Looking towards 2027 and beyond, what do you expect the next stage of AI to look like?
It’s a horrible question, to be honest, because it’s difficult to predict. If you’d asked me two years ago, I wouldn’t even have thought agents would be a thing, and now they’re basically dominating.
Right now, we’re at the early stage of them. Or maybe, in AI terms, we’re already in the late stage of them. Either way, it wasn’t something I foresaw happening so quickly.
Agentic technology will probably develop significantly further and be used a lot more. At the minute it’s still tied to certain things, whether that’s individual ecosystem services or people building agents into specific areas like development. But more general agents that are just off doing their own thing seem plausible.
Artificial general intelligence, the end goal everyone’s after, still feels like a potential fairy tale. But with the speed everything is going, I wouldn’t be surprised if it happened within the next few years. At that point, who knows. That would be a whole different universe we’re living in.
It’s going to be crazy, potentially. You could take it in any fantasy direction and it seems relatively plausible, so I’m trying to keep it as grounded as I can.
Final thoughts
Simonnet’s argument starts with a sequencing problem. Organisations are adopting AI faster than they can see it, and an organisation that cannot see what its people are using has no way to judge whether its risk is negligible or severe. It will usually assume the former, because nothing appears to be going wrong.
The behaviour underneath it is not carelessness either. LLMs work better the more they are told, and they are conversational by design, so oversharing is a product of how the technology is built rather than a failure of employee discipline. That makes it a problem to be managed with visibility and controls, rather than one to be solved with a policy reminder.
On who fixes it, he is more patient than some. Industry standards still have time to catch up, and if they don’t, legislation on negligence follows. Either way, the responsibility in the meantime sits with organisations themselves, which brings the question back to whether they can see enough of their own AI usage to act on it.
