In a detailed explanation, the company said the S3 team was attempting to debug an issue that caused a slowdown in its billing system when, at 9.37am PST on Tuesday (4.30am Wednesday AEST), one of its technical staff ran a command that was intended to remove a few servers from one of the subsystems used by the S3 billing process.

The worker entered one wrong input for the command and ended up removing a much larger number of servers than intended, some of which supported two other S3 subsystems.

Among these were the index subsystem that manages the metadata and location information of all S3 objects in the Northern Virginia (US-EAST-1) Region and is needed to serve all GET, LIST, PUT, and DELETE requests.

The outage took down a large number of websites and services. AWS services something close to 1% of the top million websites.

"The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart," the company said.

"While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including the S3 console, Amazon Elastic Compute Cloud (EC2) new instance launches, Amazon Elastic Block Store (EBS) volumes (when data was needed from a S3 snapshot), and AWS Lambda were also impacted while the S3 APIs were unavailable."

The company also said it had been unable to update the AWS service health dashboard for two hours after the issue began, because the administration console has a dependency on Amazon S3.

"Instead, we used the AWS Twitter feed (@AWSCloud) and SHD banner text to communicate status until we were able to update the individual services’ status on the SHD. We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions."

The company outlined several changes it would be making as a result of the outage and apologised for the breakdown. "While we are proud of our long track record of availability with Amazon S3, we know how critical this service is to our customers, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability even further," it said.