James Moore, Q&A: AI Security And Governance: Why Most Organisations Are Flying Blind

AI adoption is moving faster than most organisations can govern it. It is a familiar line by now, repeated often enough to sound routine, and that is part of the problem: it is still true, and still understated. As James Moore, CEO of AI security and governance platform CultureAI, told the IT Security Guru, this is a security challenge and a business risk, and one leaders can no longer afford to ignore.

Employees are using a growing number of known and unknown AI tools, and sensitive business data is finding its way into third-party platforms and new integrations. Security teams are finding it harder to know exactly where organisational data ends up. Meanwhile, many businesses are still relying on basic policies or bolt-on security tools to manage an increasingly complex problem.

Moore is focused on the bigger question: do organisations have enough visibility and control to use AI safely?

The IT Security Guru spoke with Moore about how the industry should define AI security and governance in the first place, why companies cannot be trusted to self-regulate, who should own AI governance internally, the danger of treating AI security as a box-ticking exercise, and why the industry may end up learning this lesson the hard way.

Q: How should businesses think about AI security and governance?

I still don’t think that we, as an industry, have fully defined AI security and governance yet, if I’m being really honest. But there are a few aspects business leaders need to think about.

You have the discovery piece. Before you do any security or governance, you have to be able to discover what is out there and what’s actually being used. The big AI security and governance companies discover a small amount of the foundational AI apps an organisation might be using, but they’re missing a huge portion of the AI footprint that’s out there. Discovery is genuinely foundational, to detect the many shadow and embedded AI in use.

Then it’s understanding, which is the context piece. Not only what’s being used, but how it’s being used, and why. Why are employees putting what they’re putting into these tools?

Then you’ve got control. Control for me is more on the security side: how do you stop people doing risky stuff inside the apps they’re using? How do you stop them using it badly while still enabling them?

And then governance sits above all of that, which is how do you prove all of the above? How do you prove you’ve got a list of the apps people are using? How do you prove you know what they’re putting into those apps? How do you prove you’ve got that security control in place?

There’s no way you can do governance without those three layers underneath it: discovery, understanding and control. Governance is the extra bit on top that proves you’re doing the right stuff.

Q: Do you think regulation needs to play a bigger role?

I strongly believe companies shouldn’t be allowed to self-regulate on AI. We need government regulation that says organisations have to have controls in place, because the majority of companies out there either have no controls or minimal controls. They’re flying completely blind.

The majority of companies out there either have no controls or minimal controls. They’re flying completely blind.

I also don’t think the general public know how far and wide their own data is going. In our own research we’ve seen sales and support people at fairly large consumer businesses putting personal customer details into ChatGPT, usually on their own personal accounts, to build emails, build websites, all kinds of crazy stuff. The consumer has no idea their data is going into those apps.

Every single time we put our platform inside a client or a prospect, the organisation sees something unexpected, and in some cases frankly quite horrendous.

Q: What are your strongest opinions on where the AI industry is heading?

I predict we’re going to end up with a bunch of platforms, the big ones, that bolt on AI security. People will buy it because it’s an easy add-on. They’ll bolt on the bare minimum to get people to buy an AI security module, so organisations can tick a box. That’s what’s been happening with cybersecurity in general: bare minimum spending, just to tick a box and say yes, we’ve got it covered.

What you’ll end up with is the organisations that are using these big platforms will not be doing AI security properly, but doing it well enough to pretend that they are.

And that will go one of two ways. Either it works, and everyone carries on for years, nice and happy. Or something goes wrong at one of these companies, whether that’s a breach or data being used in ways it never should have been, at which point everybody’s going to go, “err, we might need to do this properly.”

Q: What are the biggest AI risks businesses are facing today?

Over the past year the risk has shifted a little from people using AI on the web to using it on the desktop. With desktop usage comes more integrations, so people are connecting their desktop AI to more data sources and more third-party tools. Data is moving around more, and there’s more chance of it getting lost or misused.

There’s a lot of talk about agentic, but we’re still not seeing much of it actually happen in practice. It’s quite rare to see real agentic use inside an organisation. Most organisations are still pretty immature with this stuff, even though they might say otherwise.

But are the fundamental challenges much different to 12 months ago? Not really. It’s still about controlling what AI is being used, what data is going into it, and where that data ends up. That ultimately is the challenge.

Q: Where does CultureAI fit into that future?

Most organisations will soon have a mix of humans and agents in their org charts, with a very large number of agents doing quite a large number of jobs. So over time our focus will split between protecting human use of AI and protecting agentic AI.

Beyond that, we will be focusing on more visibility into the AI tools people are actually using, working on better classification of the data going into them, and continuing to build strong guardrails and controls around both.

Final thoughts

Moore’s argument comes back to sequence. Discovery, understanding and control have to exist before governance means anything, and in his eyes, most organisations are still missing the first step.

A module that ticks the box may satisfy an auditor for a while, but it will not tell a board which AI tools employees are actually using, what data is going into them, or where that data ends up. Those, as Moore notes, are much the same questions the industry was asking twelve months ago, only now with desktop integrations widening the surface and agents arriving behind them.

Whether organisations get ahead of that deliberately, or only after something goes badly wrong, remains an open question. On his evidence, most are still flying blind enough that they would struggle to know either way.