Thunderstruck: How four lightning strikes in a row erased Google data

A Google data center in Belgium suffered data loss after it was struck by lightning four times in a row. Google says its engineers were able to recover almost all of the data, and the company is upgrading its systems so it can better withstand electrical anomalies.

“Lightning never strikes twice,” The Move sang on their 1970 album Looking On. But evidently that’s not true if you’re Google.

Lightning struck four times at a company data center in Belgium last week, affecting data storage on a number of disks and resulting in permanent data loss for a small number of Google Compute Engine (GCE) systems. GCE is a service through which Google clients can run virtual computers or store data in the cloud.

The lightning strikes occurred in quick succession last week, temporarily knocking out power to systems at the data center. Google said in an online statement that auxiliary power systems quickly restored electricity, and that all the storage systems have battery backups. But a few systems had had repeated battery drain, and disks located on those systems started returning errors as they tried to store new data.

Google engineers performed data recovery operations on the affected disks over the next several days, and were able to salvage nearly all of the data. In the end, Google said, less than 0.000001 percent of the disks suffered permanent data loss. The company is upgrading its storage systems to run on hardware that is less susceptible to power failures, and is improving the manner in which data is cached during short electrical disruptions.

What are the chances that a data facility could be struck four times in a row by lightning? The odds of a person being struck by lighting in their lifetime are about 1 in 300,000, according to the BBC – but a building is much more likely to attract strikes. Data centers are built with lightning rods to dissipate strikes, but sometimes lightning does get through. And when lightning hits a building multiple times, it doesn’t need to hit in exactly the same spot each time to cause extra damage. In fact, if lightning strikes the cables connecting a data center to the rest of the Internet, the surge can travel back through the cables to the facility, causing damage.

In Google’s case, the danger posed by lightning strikes is pretty remote. The company’s storage is already robust enough to withstand most electrical disruptions, and the small number of systems that were affected by this phenomenon are being upgraded with more resilient hardware. For homeowners, it’s still a good idea to unplug computers and other sensitive electronics during thunderstorms or electrical disturbances. Alternatively, a good-quality surge protector will protect computers, home theaters, and other systems from being fried by lightning strikes.