Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Employee Error to Blame For Huge Amazon S3 Outage

Employee Error to Blame For Huge Amazon S3 Outage

Amazon says an employee error was responsible for the Amazon Web Services outages earlier this week that took a large number of websites completely offline. On Tuesday, users predominantly on the east coast were unable to access a large number of sites including Netflix, Quora, Slack, Reddit, The Security and Exchange Commission, AirBNB, Medium and Expedia (even, ironically, The issues plagued Amazon for much of Tuesday

Now, in a website post, Amazon says the problems were caused when an employee screwed up a coding update to the company's billing systems.
"At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process," Amazon said. "Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended."

The servers that were inadvertently removed managed the metadata and location information of all S3 objects on the east coast, as well as the allocation of new storage. Removing a significant portion of the capacity for these systems required a full restart to resolve, but while these subsystems were being restarted, S3 was unable to service requests, the company notes.

Ultimately, things were resolved by 5 PM EST on Tuesday. Amazon stated they're taking steps to prevent the problem from happening again.

"We are making several changes as a result of this operational event," Amazon said Thursday. "Finally, we want to apologize for the impact this event caused for our customers. We will do everything we can to learn from this event and use it to improve out availability even further."
I wonder how many Admins and Editors abandoned these websites because Amazon's servers went down?

Forum Jump:

Users browsing this thread: 1 Guest(s)