The General Data Protection Regulation (GDPR) predates the commercial availability of large language models by several years. It forced organisations to answer questions most of them had never really had to answer before: including specifically where did this data come from, and does the reason we collected it still hold? Most couldn’t; many still can’t. Artificial intelligence (AI) hasn’t invented that gap, but it has put a giant figurative neon sign over it.
The ICO updated its AI and data protection guidance in March 2023. It ran a five-part consultation on generative AI through 2024. The Data (Use and Access) Act became law in June 2025 and gave the ICO a statutory basis for a formal Code of Practice on AI and automated decision making. Three years, still circling the same question: what the lawful basis is for training a model on personal data, and whether that basis survives the model actually being used. The ICO’s answer, so far, is ‘legitimate interests’, tested against three questions: is the purpose legitimate, is the processing necessary, and do individuals’ interests override the interest being pursued? The test breaks down at that third question. It was built to weigh one company’s interest against one identifiable person’s. Training data means weighing it against millions of people you’ve never had a relationship with and can’t assess individually.
Additionally, Article 17 gives you the right to erasure, but try applying that to a model that’s already trained. In a database that might mean deleting a row, but a trained model doesn’t store your data, it’s diffused across billions of parameters. Pulling your specific influence back out is called ‘machine unlearning’. It’s an active research field, and nobody’s shipping a reliable version of it yet. The ICO’s own guidance recommends organisations build mechanisms to catch unauthorised outputs after unlearning has supposedly happened, monitoring for a failure the process isn’t guaranteed to prevent. Most organisations promising erasure on an AI system haven’t tested whether they can deliver it. They’ve just promised it, because the alternative is admitting the product can’t do what the compliance page says.
That’s the training data problem. There’s a second one, and it moves faster. I call it Assume Autonomy. AI increasingly acts without waiting for a human to authorise it, and that changes the tempo of whatever it’s acting on. Article 22 assumes a decision point, a human standing between an inference and its consequence. An agentic system that infers, decides, and acts in one loop doesn’t leave that human anywhere to stand. One problem is about data collected in the past. The other is about a decision made about you five seconds ago, with no one in the room.
Organisations building AI now have to answer both questions before a regulator asks them, not after. Data lineage is the easier one. Mostly a discipline problem. Human oversight inside an autonomous loop is harder, because the whole point of the loop was to remove the human.
The EU AI Act doesn’t solve this for you, but it does narrow the excuses. High-risk AI systems under the Act need documented data governance and a level of human oversight built into the design, not bolted on after deployment. That’s the same demand GDPR is making from a different angle, one regulation asking where the data came from, the other asking who’s still watching once the system starts deciding on its own. Nobody gets to satisfy one and ignore the other.
