When the General Data Protection Regulation (GDPR) came into force, generative AI wasn’t embedded in everyday business applications, AI agents were not executing multi-step tasks across corporate databases and prompt injection was not a mainstream vector for data exfiltration.
GDPR is one of the world’s most important pieces of data protection legislation and its principles are highly relevant to AI. But it was designed around a very different data environment. AI has introduced fundamentally different ways for data to be processed, inferred, retrieved and exposed, introducing challenges and compliance risks that could not have been anticipated.
This doesn’t mean GDPR has become obsolete, but we do need to question whether the way organisations demonstrate compliance is enough when the systems processing personal data can themselves be manipulated. Take prompt injection. An organisation may have strong access controls around a database containing sensitive customer information, and a clearly documented lawful basis for processing and robust policies around retention and deletion. But if it gives an AI assistant access to that database, an attacker could manipulate the AI into retrieving information that it has legitimate access to.
The underlying data controls may be working but the vulnerability exists in the layer between the user, the AI and the systems it can access. One of the important differences AI introduces is that security is not just about preventing unauthorised access; it is also about preventing authorised AI systems from being manipulated into producing illegal outcomes.
This risk escalates exponentially as enterprises deploy agentic workflows. An AI agent may be able to search internal documents, send emails, update records, access customer systems or interact with other applications. Every additional capability creates another potential route to personal data if the agent can be manipulated.
When it comes to AI, are documented controls, a data protection impact assessment and appropriate policies enough to demonstrate GDPR compliance? Or should organisations also be able to demonstrate that they have actively tested the system? GDPR requires that organisations implement appropriate technical and organisational measures to protect personal data. However, in an AI environment, those measures can’t be assessed just by looking at whether they exist. Organisations need to understand how they perform when an AI system is subjected to adversarial conditions.
That means testing for prompt injection and data leakage, examining what an AI system can reveal through carefully constructed interactions and determining whether an agent can be manipulated into accessing or transmitting information outside its intended purpose. It also means testing the permissions given to AI systems and considering what happens when those permissions are abused.
AI therefore needs to be treated as part of the attack surface, not as an application that can be assessed once and then considered secure. AI systems behave differently from conventional applications, with outputs influenced by inputs, context and interactions. A system that appears secure under normal conditions may behave differently when confronted with an adversarial interaction.
A compliance assessment carried out when an AI system is deployed cannot necessarily tell an organisation how that system will behave several months later, after the underlying model has changed, new data sources have been connected or its permissions have expanded. GDPR compliance in an AI environment therefore needs to be continuous rather than static.
There is also a capability challenge. Developers need to understand the security implications of the AI systems they build, while security professionals need hands-on experience defending against adversarial AI techniques. Crucially, organisations must evaluate human-AI readiness, proving that human operators have the technical dexterity to direct, validate, and override autonomous agents when they hallucinate or fall under attack. That knowledge cannot come solely from policies or theoretical training. Teams need opportunities to test these scenarios in realistic conditions and understand how systems behave when those attacks succeed.
AI provides another route where personal data can be exposed and having a policy stating that an AI system must protect personal data is not enough. Organisations need to be able to demonstrate that the system has been tested, its permissions challenged and potential routes to data exposure investigated, alongside having people with the capability to respond when something goes wrong.
GDPR’s principles around privacy, accountability, data minimisation and security are arguably even more important with AI. But the regulation can’t have anticipated every mechanism through which AI can expose personal information and organisations cannot rely on compliance approaches developed for a fundamentally different technological environment.
The future of GDPR compliance will need more than governance and documentation. It will need evidence that organisations have actively tested their AI systems, challenged their assumptions and assessed how those systems behave under attack. It will also need evidence that the people responsible for building and defending them have the practical capabilities to identify and respond to emerging threats.
