How a typo made the Amazon cloud go dark for scores of internet users

The outage was an abrupt reminder that the internet is not as invincible as its near seamless fusion with our lives suggests.

Reed Saxon/AP
The Amazon logo as seen in Santa Monica, Calif. Amazon’s cloud-computing service Amazon Web Services experienced problems in its eastern US region on Tuesday, Feb. 28, 2017, causing widespread problems for thousands of websites and apps.

How big is Amazon’s cloud? Big. So big, in fact, that its cloud storage arm, Amazon Web Services, is larger than the equivalent service offered by the next three players – Microsoft, Google, and IBM – combined.

That is why it was such a big deal when an Amazon team member, who accidentally entered a couple of wrong bits of code during some routine maintenance on Tuesday, was able to knock out large portions of the internet for around four hours.

AWS hosts a number of high-profile, heavily trafficked websites and services including AirBnb, Netflix, reddit, and Quora, many of whose pages were not loading during the outage. And although the internet giant moved quickly to fix the problem, the mishap was one of the periodic reminders we get that the internet is not as invincible as its near seamless fusion with our lives suggests.

In a public apology issued by Amazon, the company explained that the fat-finger incident occurred while an employee from Amazon Simple Storage (S3) was working to speed up the S3 billing process. “Using an established playbook executed a command,” as Amazon put it, the worker’s intention was to temporarily offline a small number of servers in the S3 subsystems, but the error took down a lot more.

“In this instance, the tool used allowed too much capacity to be removed too quickly,” Amazon said. “We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level.”

Or, as The Washington Post’s Brian Fung put it: “Translation: Employees will no longer be able to unplug whole parts of the Internet by mistake.”

In Amazon’s case, its rise to the top of the so-called Infrastructure as a Service (IaaS) tree, began in 2006, when it, in all its frugality, started buying up or leasing existing data centers dotted across northern Virginia, “a central region for internet backbone,” according to The Atlantic.

However, the fact that Amazon didn’t build new servers from scratch also means they’re old, potentially making them more susceptible to crashing.

The timing of the crash couldn’t have been worse. It came on the same day that Amazon was holding one of its AWSome Days, where it promotes the advantages of AWS and educates people how to use it. BGR.com’s Mike Whener wrote about the unfortunate timing from Edinburgh, Scotland:

Amazon loves to talk about how great its products and services are – just like any other massive company — so the fact that it holds frequent conferences celebrating and educating people about Amazon Web Services (AWS) isn’t particularly odd. But for one of those events to land on the exact same day that AWS’s storage services bites the dust and takes a huge chunk of the internet down with it? Now that’s some serious bad luck.

You've read  of  free articles. Subscribe to continue.

Dear Reader,

About a year ago, I happened upon this statement about the Monitor in the Harvard Business Review – under the charming heading of “do things that don’t interest you”:

“Many things that end up” being meaningful, writes social scientist Joseph Grenny, “have come from conference workshops, articles, or online videos that began as a chore and ended with an insight. My work in Kenya, for example, was heavily influenced by a Christian Science Monitor article I had forced myself to read 10 years earlier. Sometimes, we call things ‘boring’ simply because they lie outside the box we are currently in.”

If you were to come up with a punchline to a joke about the Monitor, that would probably be it. We’re seen as being global, fair, insightful, and perhaps a bit too earnest. We’re the bran muffin of journalism.

But you know what? We change lives. And I’m going to argue that we change lives precisely because we force open that too-small box that most human beings think they live in.

The Monitor is a peculiar little publication that’s hard for the world to figure out. We’re run by a church, but we’re not only for church members and we’re not about converting people. We’re known as being fair even as the world becomes as polarized as at any time since the newspaper’s founding in 1908.

We have a mission beyond circulation, we want to bridge divides. We’re about kicking down the door of thought everywhere and saying, “You are bigger and more capable than you realize. And we can prove it.”

If you’re looking for bran muffin journalism, you can subscribe to the Monitor for $15. You’ll get the Monitor Weekly magazine, the Monitor Daily email, and unlimited access to CSMonitor.com.