In a recent blog post, Google Cloud has shared details about an incident that impacted one of its Australian customers, UniSuper, a pension fund.
The incident involved accidentally deleting the customer’s Google Cloud VMware Engine (GCVE) Private Cloud due to a misconfiguration by Google operators.
According to the post, the incident occurred when Google operators deployed a GCVE Private Cloud for UniSuper using an internal tool.
An inadvertent misconfiguration occurred when a parameter was left blank. This had the unintended consequence of defaulting the customer’s GCVE Private Cloud to a fixed term with automatic deletion at the end of that period.
Free Webinar on Live API Attack Simulation: Book Your Seat | Start protecting your APIs from hackers
After one year, the customer’s GCVE Private Cloud was automatically deleted due to this misconfiguration. No customer notification was sent because the deletion was triggered by the blank parameter in the internal tool, not by a customer deletion request.
Google notes that a notification to the customer would have preceded any customer-initiated deletion.
Following the incident, the customer and Google teams worked several days to recover UniSuper’s GCVE Private Cloud, restore the network and security configurations, and restore its workloads from backups.
Fortunately, the customer had data stored outside of Google Cloud, which aided the recovery process.
Google has taken steps to prevent similar incidents in the future, including deprecating the internal tool that caused the misconfiguration and automating the deployment process to remove the possibility of human error.
Many tech experts have criticized Google’s response, deeming it the “bare minimum.” They advocate for more comprehensive measures, such as implementing manual review processes before carrying out large-scale deletions and considering temporary suspension instead of immediate deletion of services.
This incident emphasizes the dangers of relying only on cloud providers for infrastructure and the necessity of maintaining off-site backups.
It also underscores the need for robust processes and safeguards against human error when dealing with critical customer data and services.
ANYRUN malware sandbox’s 8th Birthday Special Offer: Grab 6 Months of Free Service