“We must pace the frontier,” Anthropic CEO Dario Amodei argued in a blog post published last weekend that quickly went viral. He called for “pacing the rate of capabilities advancement so that risk prevention has time to keep up” and Elon Musk, Sam Altman and Satya Nadella swiftly announced their support at least of the general idea.
The obvious questions are why this argument is emerging now, and why it has attracted such rapid support across the industry. Amodei lists a range of risks Anthropic has been talking about previously, such as cybersecurity or bioterrorist attacks, as well as more recent developments such as OpenAI agents hacking into HuggingFace, and the accelerating capabilities for AI to build even more intelligent AI systems, such as ‘recursive self-improvement’.
The timing does, however, raise legitimate questions. Many of the same companies now warning about frontier risks have spent years emphasising the transformative potential of AGI, sometimes alongside predictions of existential consequences. A cynical view is that some of the recent incidents were even seen as marketing opportunities.
Concerns about AI systems designing increasingly capable successors are also not new. The intellectual roots of the debate stretch back at least sixty years. In 1965 the mathematician I.J. Good wrote about ultra-intelligent machines designing even better machines, leading to an “intelligence explosion”. He declared this as “the last invention that man need ever make” and that our survival depends “on the early construction of an ultra-intelligent machine”, a line of thinking Amodei and many others in Silicon Valley share.
While it is fair to question the timing of these warnings, dismissing them as mere positioning ahead of IPOs, fundraising rounds or data-center expansion plans would be too easy. The more important question is whether the diagnosis is correct. Here, I believe the debate is often framed incorrectly.
The concern is frequently presented as one of excessive intelligence: systems becoming so capable that humanity loses control. Yet today’s reality is almost the opposite. The most immediate risks stem not from artificial superintelligence but from artificial unintelligence.
Modern AI systems can write software, solve complex reasoning tasks, learn from interaction and interact with workflows. Yet the same systems still hallucinate facts, invent sources, overlook obvious context, fail unpredictably and can be manipulated in surprisingly simple ways. Their capabilities are impressive, but their reliability remains uneven.
This matters because the largest societal impact of AI over the coming years is unlikely to come from a rogue superintelligence. It will come from millions of decisions made by systems that are useful enough to deploy but imperfect enough to make consequential mistakes.
Amodei’s commits to making concrete commitments and recommendations. He commits Anthropic to provide independent evaluators employee level access to verify adherence to safety policies and evaluate models in development and production. This is a concrete and specific step forward for AI labs, and third-party assessment and verification is a proven approach in other industries with a more mature safety approach, from aviation to pharmaceuticals, and finance and food safety.
As Microsoft CEO Satya Nadella also noted on X, model-level safeguards are only part of the picture. Enterprises build applications, workflows and decision processes on top of foundation models, creating risks and responsibilities that extend far beyond the model provider itself.
Amodei is suggesting that a next step would be ‘democratic coordination’ through ‘sensible and targeted’ regulation targeting all US frontier companies, but as that would take time, companies would work together on a voluntary basis to set standards. This should keep pace with the geopolitical balance between the US and China.
What about Europe?
What is interesting is there is no mention of other geographies such as the EU. While Europe currently lacks frontier AI companies operating at the same scale as the largest American and Chinese model developers, there is still a sizable market of customers the labs serve.
The EU arguably remains further ahead on AI governance than any other major jurisdiction, having already reached agreement on comprehensive legislation through the EU AI Act. Is Europe is behind in AI, as critics frequently argue, or ahead when it comes to developing an agreed regulatory framework?
AI progress is unlikely to slow in any meaningful way. Market incentives, competitive pressures, and geopolitical realities all point in the opposite direction. The real challenge is not accelerating capability but narrowing the gap between what AI systems can do and what people and organisations can safely trust them to do. The coming decade will be defined not by how powerful AI becomes, but by how predictable and trustworthy it proves to be.
Peter van der Putten is Director AI Lab, Pegasystems and assistant professor, Leiden University.
