Google has traced the 20 October power failure that knocked out UK banks, courts and couriers to a programmable logic controller (PLC) that measured mains voltage on the wrong side of a circuit breaker – a single sensing error that stopped the site’s emergency generators from starting and let the UPS batteries run flat.
The outage began at about 06:53 BST, when an external underground cable fault knocked out the facility’s primary utility feed at the datacentre that hosts europe-west2-b, one of three availability zones in the London region.
The site should have ridden through the fault. Instead, its diesel generators never started, the static UPS batteries – designed to work for only about five minutes – drained, and the data halls lost power completely.
The reason the generators stayed silent lies with a maintenance decision made a week earlier. After a quarterly generator load test on 14 October, Google isolated the second mains feed for corrective repairs, leaving the site in what the report calls a “temporary single-feed (1N) bridged maintenance posture” rather than its usual dual-feed (2N) design.
When the remaining feed then failed, no live mains supply was left – and the controller that should have called the generators into action looked in the wrong place.
A PLC is a rugged industrial computer that runs simple control logic. When incoming voltage falls, it starts the generators. The flaw was in where it took its measurement.
The PLC’s undervoltage sensors were wired upstream of the isolated feed B breaker. That breaker was open, but the cable on the utility side of it stayed energised. Therefore, when feed A failed, the controller saw healthy voltage on the still-live feed B cable – a cable connected to nothing the building actually used – and concluded the mains was fine. No undervoltage was detected, so no start command was issued.
Google’s report calls it “a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position”.
In plain terms, the controller asked whether the street still had power, but not whether that power was still reaching the building – the electrical equivalent of checking the mains outside the house rather than the trip switch in your own fuse box.
The consequence was a zone-wide blackout and a cascade across the region.
Google lists more than 35 Google Cloud services affected in europe-west2. These included:
- Core compute: Compute Engine, Kubernetes Engine, Cloud Run, App Engine;
- Data platforms: BigQuery, Cloud SQL, Spanner, AlloyDB;
- Storage: Cloud Storage, Persistent Disk;
- Networking: Virtual Private Cloud, Cloud Load Balancing, Cloud NAT.
Downstream, the failure rippled through household names. Reports at the time named Lloyds, HSBC, Halifax and Revolut among the banks affected, alongside HM Courts & Tribunals Service video hearings, Royal Mail, parts of the NHS, and a string of retailers and delivery firms.
Technology suppliers including Stripe, Snowflake, MongoDB and JFrog also reported knock-on problems, while the disruption later prompted MPs to ask finance regulators how they would handle future concentration risk in Google Cloud.
With the zone dark, automated failover shifted workloads to the two surviving zones. The sudden influx of redirected compute, database failovers and cross-zone traffic saturated available capacity, produced queue backlogs, delayed failovers, and created intermittent HTTP 500 and 503 errors across the wider London region.
Google has described this as a degraded state rather than a full regional control-plane outage, but multi-zonal and regional services still experienced elevated latency and application programming interface timeouts, while some customers in the untouched zones reported impact.
Google has outlined the fixes. It reconfigured and tested the PLC code and undervoltage logic so that generator startup now relies on actual busbar de-energisation – and breaker position – rather than upstream voltage readings, with validation taking place on 20 and 26 October.
It has begun to audit PLC configurations at other facilities with similar topologies, and has scheduled coordinated cable repairs, generator confidence load-transfer tests and full restoration of dual-feed utility configuration through November.
The incident carries a wider lesson for enterprise IT. Defence in depth – dual feeds, batteries and generators – was undone by one sensing error that only mattered because the site was already running in a degraded maintenance posture. Because Google operates a single UK cloud region, there was no regional failover to lean on, and the surviving zones, designed to absorb a lost zone’s traffic, saturated, which is why even customers that ran multi-zone architectures felt the impact.
The episode underlines the continuity lessons for IT leaders that follow any supposedly redundant cloud design, at a time when the cost and complexity of datacentre outages continues to climb.
