ComputerWeekly

Google Cloud power event knocks out UK banks, courts and couriers


A power problem in Google Cloud’s London region knocked out banking apps, the courts’ video platform and delivery networks across the UK on Tuesday 20 October, with no second Google UK region for customers to fall back on.

By late afternoon, most services had recovered. Google said the issue affected only a “very small number” of customers, while the Ministry of Justice confirmed the courts’ Cloud Video Platform had still not fully recovered. Google said: “We will publish an analysis of this incident once we have completed our internal investigation.”

The outage began just after 7.00am BST when the europe-west2-b availability zone – one of three inside Google’s London region, europe-west2 – lost power. Google Cloud confirmed the cause: “A power event caused an outage within europe-west2-b zone. The power event is mitigated. We are seeing the recovery of services and are prioritising the recovery of existing workloads.”

The incident is the latest reminder that the UK’s largest businesses remain dangerously exposed to cloud outages. Because europe-west2 is Google’s only UK region, customers that had not built multi-region resilience had no second UK location to failover to.

The failure hit four Google Cloud locations:

  1. europe-west2 (London) – Google’s only UK region, where the failure struck.
  2. europe-west2-b – the availability zone that lost power.
  3. europe-west2-a and europe-west2-c – the peer zones that, by one engineer’s account, did not fully absorb the load.
  4. europe-north2 – one administrator also reported VPC, Cloud NAT and load-balancer issues here.

For UK customers, the outage rippled through a long list of household names:

  • Banks and fintech: Revolut, Lloyds Bank, HSBC, Barclays, Bank of Scotland, NatWest, Halifax, Nationwide, Emma and credit-reference firm Equifax.
  • Delivery and retail: Royal Mail, DPD and Screwfix.
  • Property and web services: Rightmove, Cloudflare, BT and hosting firm SiteGround.
  • Public sector: NHS England, HM Courts & Tribunals Service and Gov.uk.

The day unfolded like this:

  • 07.05 – VPC, Cloud NAT and load-balancer issues began in europe-west2, with one administrator also reporting europe-north2.
  • 10.00 – The courts’ Cloud Video Platform went down across large parts of England and Wales.
  • 10.57 – Google confirmed “an issue with multiple services in europe-west2”.
  • 11.03 – DownDetector logged thousands of reports across Google, Revolut, Lloyds, HSBC, Barclays, Halifax, Royal Mail, Bank of Scotland, DPD and Sky.
  • 11.05 – Revolut told customers that “all our customers’ money and data are safe”.
  • 11.16 – NHS England, HM Courts & Tribunals Service and Gov.uk appeared to be affected.
  • 11.27 – The Criminal Bar Association warned that trials and hearings “can not proceed”.
  • 12.27 – Google said the power event was mitigated and partial mitigation would follow within 30 minutes.
  • 12.45 – The first administrators reported virtual machines in europe-west2-b rebooted and services restored.
  • 14.48 – Google reported “recovery of services for most workloads in europe-west2”.
  • 14.59 – UK Power Networks confirmed its London systems ran normally, ruling out a substation fault.
  • 15.10 – Google said only a “very small number” of customers remained affected.
  • 15.52 – The Ministry of Justice confirmed its systems were not yet fully restored.

The failure also played out in real time on Reddit among the IT professionals who had to manage it. One engineer posting as Rs3_Myway, who said they were “on calls at work dealing with it”, wrote: “Availability zone B went down with power cut. A and C, for some reason, didn’t pick up the slack completely in prod (unclear why). Apigee gateway as the big problem system.”

An administrator relaying Google’s engineering update described the root cause: “Although mains power has returned, backup generators failed to engage during the initial outage. As a result, Engineering is maintaining a frontend and backend drain on the europe-west2-b clusters while onsite technicians verify backend power functionality.”

Another user summed up the structural problem: “It is the only Google region in the UK so businesses/banks/NHS etc. here can’t switch to another location.”

And an IT manager posting as Crimsondelo reported the fallout across their own supply chain: “We are seeing outages on several of our providers including Equifax. Apparently Lloyds is having issues as well.”

The episode echoes an Amazon Web Services outage that knocked out UK tax office and banking services last autumn, and it lands as investment banks lose £600,000 an hour to IT failures. For enterprise buyers, the lesson is architectural – a single-zone service in a single-region footprint remains a single point of failure, and data sovereignty does little to help when the only region available is the one that went down.



Source link