Buried in this week’s Google I/O news is one of the biggest failures in Google Cloud history. Google’s Amazon Web Services competitor accidentally deleted a huge customer account for no reason. UniSuper, an Australian pension fund with $135 billion worth of funds under management and 647,000 members, had its entire account wiped on Google Cloud, including all backups stored on the service. UniSuper thankfully had some backups with another provider and was able to recover the data, but according to UniSuper’s incident logs, the downtime began on May 2nd for his Full restoration of service did not occur until his May 15th.
UniSuper’s website is currently full of nightmare fuel that is a must-read for administrators about how this happened. The first is a wild page posted on May 8th titled “Joint statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kuran.” The statement read: “Google Cloud CEO Thomas Kurian said the disruption was caused by an inadvertent misconfiguration during the provisioning of UniSuper’s private cloud service that ultimately resulted in the failure of UniSuper’s private cloud subscription. Google Cloud acknowledged that this was an isolated and “unique event” that had never occurred before at any of Google Cloud’s clients around the world. We have identified the events that led to this disruption and have taken steps to prevent it from happening again. . ”
The next section is titled “Why did the disorder last so long?” “UniSuper had redundancy in the two regions as protection against outages and losses. However, when UniSuper’s private cloud subscriptions were removed, deletions occurred across both of these regions,” the joint statement said. It has been stated. All cloud services maintain full backups, which are supposed to be for worst-case scenarios. Imagine a hacker taking over your server or the building containing your data collapsing. But no, actual The worst case scenario is that Google deletes your account, which means you lose all your backups as well. Google Cloud is supposed to have safeguards that don’t allow account deletion, but apparently none of them worked and the only option was a restore from another cloud provider (his Thank you UniSuper heroes).
UniSuper is Australia’s ‘superannuation fund’. The US equivalent is a 401(k). This is severance pay that is paid by an employer as part of an employee’s salary. In Australia, all employers are required by law to pay some level of superfund. With $135 billion worth of funds under management, UniSuper has become so big that when something goes wrong, they call the Google Cloud CEO instead of customer service.
In a June 2023 press release touting UniSuper’s large-scale cloud migration to Google, Sam Cooper, head of architecture at UniSuper, said: “Google Cloud VMware Engine streamlines our transition to the cloud and makes it super easy. It’s all about efficiency to provide competitive rates to our members.”
The large number of parties involved in a service means that restoring the service is not just about restoring a backup, it also involves processing all the requests and payments that had to occur during the two weeks of downtime. It meant that.