Industry superannuation fund UniSuper has revealed that its private cloud infrastructure “subscription” with Google Cloud was deleted due to an error made during provisioning.
In a joint statement released overnight, the fund shed light on cryptic earlier commentary that it had lost primary and secondary cloud-based infrastructure due to a complex chain of events, including a misconfiguration and software bug.
With the fund’s outage now week-long, and with point-in-time restoration of some member online services anticipated today, the scale of UniSuper’s IT issues has been disclosed.
The revised explanation is that an “inadvertent misconfiguration during provisioning … ultimately resulted in the deletion of UniSuper’s private cloud subscription.”
It’s not clear who made the configuration error.
What is clearly stated is that the chain of events the misconfiguration put in motion, which led to UniSuper’s cloud accounts being deleted, has not been observed before by Google Cloud.
“This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally,” a joint statement by UniSuper CEO Peter Chen and Google Cloud CEO Thomas Kurian reads.
“This should not have happened.”
UniSuper said it lost cloud infrastructure in “two geographies”, a configuration it believed should have kept it safer in an outage situation.
“UniSuper had duplication in two geographies as a protection against outages and loss,” according to the statement.
“However, when the deletion of UniSuper’s private cloud subscription occurred, it caused deletion across both of these geographies.”
Much of the focus has understandably since been on recovering the deleted environment.
“Restoring UniSuper’s private cloud instance has called for an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems,” according to the statement.
UniSuper said this effort had resulted in “extensive recovery of our private cloud which includes hundreds of virtual machines, databases and applications.”
It also had backups “with an additional service provider” that it said, “have minimised data loss” and aided with the restoration effort.
It’s not clear from the statement what intervention points exist, if any, before a large cloud environment can be deleted.
A line in the statement that “Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again” suggests some analysis has been performed of the end-to-end process, and that some future checks may be made.
iTnews understands that other Google Cloud customers have been seeking explanations and assurances from the provider this week about their own environments.
[Read the Cloud Covered report to understand the benefits, challenges and best practices for cloud adoption.]