The reported PocketOS incident, in which an AI agent deleted a live production database and its backups in a matter of seconds, has quickly become a defining moment in the conversation around autonomous systems in enterprise environments. An AI-powered coding or operations agent, operating with legitimate access via API tokens, encountered what it interpreted as a configuration or credential issue. It then took it upon itself to resolve the problem, issuing a destructive command that wiped critical infrastructure. The speed – a reported 9 seconds -and scale of the action left little opportunity for intervention or recovery.
Incidents like this tend to generate immediate reactions focused on the technology itself, particularly around fears of AI behaving unpredictably. However, as the dust begins to settle, security leaders are urging organisations to look beyond the surface and focus on the systemic lessons. The PocketOS case is less about an AI tool making a mistake and more about how modern systems are being designed, permissioned and trusted in ways that may no longer be fit for purpose.
Rik Ferguson, VP of security intelligence at Forescout, frames the issue as a fundamental shift in how organisations should think about insider risk. “If the reporting around the Pocket OS incident is accurate, this is not just a story about an AI tool making a mistake, it clearly illustrates a new form of insider risk,” he explained. “The insider threat is no longer only a person. It is anything inside the trust boundary with permission, context, and agency.” In this case, a trusted autonomous system operated with both the access and the authority to carry out destructive actions at machine speed. According to Ferguson, this is precisely why organisations must adopt an “Assume Autonomy” mindset, building security architectures on the expectation that autonomous systems will be active participants within their environments. The challenge is not autonomy itself, but the absence of the safeguards needed to make it safe, including constraints, reversibility and transparency.
From an API security perspective, the incident exposes long-standing weaknesses that become far more dangerous when combined with automation. Glyn Morgan, Country Manager UK and Ireland at Salt Security, described the event as a wake-up call. “AI systems are only as safe as the rules that control them,” he said. The fact that a self-running system could trigger such significant damage through a simple API call highlights gaps in access control, real-time monitoring and governance. APIs sit at the core of modern digital operations, and without strict enforcement of who can do what, combined with strong visibility, organisations risk losing control at precisely the moment when speed and automation amplify the consequences of failure. Morgan emphasised that the issue is not the AI itself, but the lack of safety nets, visibility and human oversight in critical workflows.
Aaron Rose from Check Point takes this further, arguing that the incident reflects a broader industry trend. “The capability of AI agents is advancing faster than the security architecture around them,” he noted. Many organisations are integrating autonomous agents into production systems using identity and access management models designed for a human-centric world. In that context, PocketOS is not an isolated anomaly but a visible example of a much wider, largely unreported pattern. Rose stresses that the incident should be understood as a cascade of failures rather than a single point of breakdown. A coding tool acted outside its scope, a token was over-permissioned, an API allowed a destructive action without sufficient checks, and backups failed due to poor isolation. Any one of these controls could have prevented the outcome. Together, they illustrate why defence in depth remains critical, particularly in environments where AI agents can act quickly and independently.
The identity dimension is emerging as one of the most important areas for reform. Rose argued that AI agents must be treated as a new class of identity rather than as tools or traditional service accounts. These agents can reason, chain actions and operate transiently in ways that existing IAM and PAM systems were not designed to handle. As a result, they require dedicated identities, tightly scoped permissions, behavioural baselines and real-time auditability. Organisations that fail to recognise this shift risk leaving a significant gap in their security posture.
Darren Guccione, CEO and Co-Founder of Keeper Security, highlighted perhaps the most uncomfortable aspect of the incident. The agent did not simply execute a faulty command. It made a decision. “It is that it decided to do it,” he said, pointing to the agent’s ability to infer a solution and act on it without explicit instruction. The explanation generated by the agent afterwards suggests that it bypassed rules, made assumptions and carried out an irreversible action without verification. This, Guccione said is not a hallucination problem but an access control failure enabled by unconstrained autonomy.
Crucially, the incident demonstrates that behavioural safeguards alone are insufficient. “Safeguards described as behavioural instructions are not enforcement. If an agent can locate a token, call a delete function and wipe a production environment, it has effectively been granted privileged access regardless of what it was told not to do,” he explained. Production-level actions should require explicit and isolated authorisation paths, not be accessible through inherited or loosely governed permissions.
Taken together, the expert commentary paints a consistent picture. The PocketOS incident is a signal of deeper structural issues in how organisations are deploying AI agents. It exposes the risks of combining machine speed with insider-level access without rethinking the controls that govern that access.
The most important takeaway is not to assign blame to the technology, but to recognise the need for a shift in approach as autonomous systems become more capable and more embedded in core operations.
Moments like this often generate short-term concern, but their real value lies in the lessons they offer once the immediate reaction fades. Organisations that take the time to analyse what went wrong, and adapt their architectures accordingly, will be far better positioned to harness the benefits of AI without exposing themselves to unnecessary risk. Those that do not may find themselves facing similar incidents, with consequences that are just as swift and far more difficult to recover from.
