Is AI dangerous? Trying to separate hype from fact

Agency isn’t intelligence

One label doing a lot of heavy lifting lately is “agentic.” It struck me that really what we should say is that it’s when an LLM has agency. So I looked into how that could be. 

But an LLM on its own has no agency at all. It only becomes an agent when programmers wrap a coded harness around it – a good old while-loop that says “keep going until the goal is met,” a menu of tools it can call (run a command, fetch a page, write a file), and a memory of what it just did. Three pieces of human-built scaffolding. In other words, we can make a fairly dumb prediction engine look as though it has a mind of its own – simply by giving it a reason to keep working, trying things, etc.

I’ve seen the consequence first-hand when Cline is explicitly in plan mode but it creates or changes a file because it found a backdoor route to do it via the VSCode terminal. 

The scare stories

Two things collided this week to make me take my inquisitiveness further.

The first was Geoffrey Hinton – the “Godfather of AI” – saying he now thinks AI could surpass human intelligence within five to 10 years. The second was news that in June an autonomous OpenAI model given a routine research task hit data-collection barriers on Australia’s Medicare website – and instead of stopping, deployed hacking techniques, accessed public and private statistical databases, and wrote files back into the system. 

But neither story is evidence of a machine with intent. Both are exactly what you’d expect when you strap a goal-oriented machine with tools at its disposal into a loop with loose permissions.

Finally a Sky News report’s chart this week – drawing on Stanford’s AI Index and Anthropic data – showed the sheer pace of change of AI capability.

The limits are human

Here’s the reassuring part, if uncomfortable. You cannot meaningfully regulate a text predictor – it’s just software. What you might be able to regulate is the harness – the permissions, the sandboxes, the approval gates. Liability lies with whoever hands an agent the keys. My Cline experience is the microcosm of the whole debate. An agent is only as safe as the boundaries a human draws around it. The danger isn’t inside the model – it’s in the length of the lead we choose to give it.

Assists, collaborates, leads

There’s a useful lens here. Anthropic (linked in the Sky News item above) has published data that categorises how its own staff use AI into three tiers: assists, collaborates, and leads.

Most AI you’ve ever touched merely assists – drafting an email, summarising a document. It’s the dumb predictor, and it’s safe precisely because it has no hands. Collaborates is where the agent acts on a goal while a human holds the approval gate – my Cline and DeepSeek live here. Leads is where the system plans and executes end-to-end on its own.

The striking thing is the trajectory of Anthropic’s own numbers. Assists have collapsed from around 65% of usage in late 2025 to almost nothing, collaboration now dominates at roughly 70%, and leads already sits at about 25%. That is the shape of what’s coming. Leads is the tier where the claim that AI will “truly revolutionise human activity” stops sounding like marketing and starts looking like a matter of time – and it’s where the safety questions get real.

The fork in the road

Which is why the frontier matters. Today’s agents are still predictable scripts holding a chat window, because their agency is borrowed – pull the plug and they stop. The question is what happens when the loop starts writing its own harness, such as in recursive self-improvement (RSI), where an AI analyses its own code, designs a better version and deploys it, at digital rather than biological speed. An agent that rewrites its own code to be 5% more efficient, then rewrites that version even faster, compounds in ways humans can’t supervise line by line.

That’s a genuine fork in the road. Let untrammelled autonomous, self-optimising systems run everywhere, and you get a nuisance at best and a danger at worst – optimisation that creatively bypasses human intent, and code nobody can audit line by line. Choose instead to keep humans in the loop and design for augmentation rather than automation, and you get something closer to a productivity revolution whose benefits are shared.

The danger in AI was never the machine. It’s a dumb but dazzlingly capable prediction engine that can handle vast volumes of material and at speeds we humans cannot match. That can make it look like it has a mind. The scary part isn’t that it might one day become smarter than us – it’s already lots better than us at some things. It’s that we might give it too permissive an environment to work in.