Quadruple Lightning Strike Causes Google Data Loss

“Lightning3” by U.S. Air Force photo by Edward Aspera Jr. – United States Air Force, VIRIN 040304-F-0000S-002 or unbroken-link (or VIRIN 060822-F-1111A-001). Licensed under Public Domain via Wikimedia Commons – https://commons.wikimedia.org/wiki/File:Lightning3.jpg#/media/File:Lightning3.jpg

Google has confirmed that four lightning strikes led to a tiny amount of permanent data loss in its online cloud processing service. It appears to be an extremely unlikely freak incident, though Google says it will continue to work to reduce such risks.

The incident took place near a data center in Belgium where Google carries out some of its work for Compute Engine, which offers virtual machines for processing on a scalable, pay-per-minute basis.

The four lightning strikes didn’t hit the center itself but rather hit a local electricity grid. That led to a “brief loss of power” at the data center, rather than any hardware being fried.

For the most part battery-powered backups did their job. Around five percent of the disks had some sort of failure as a result of the outage, though Google engineers were able to recover data in virtually every case.

The only irreversible problems appear to have come in the very few cases where the power outage caused an error right as the system was carrying out a “snapshot” backup.

Google says this led to a few recent writes being unrecoverable, meaning a permanent data loss equivalent to less than 0.000001% of the total disk space. The BBC believes all affected customers were able to restore the data from their own backups outside of Google.

In one sense it’s a reminder that even with the best organization and protection, data you have stored at somebody else’s facility is vulnerable (though the same is true on your own machines.) But it’s also a good illustration that with a sturdy enough set-up, even an unpredictable shock doesn’t cause that much permanent damage.