Amazon says a major DNS failure was behind a massive AWS (Amazon Web Services) outage that took down many websites and online services on Monday.
As BleepinComputer reported earlier this week, this incident impacted a critical Northern Virginia data center in the US-EAST-1 region, affecting users worldwide, including the United States and Europe, for over 14 hours.
According to a post-mortem published on Thursday, a race condition caused a major DNS failure in Amazon DynamoDB’s infrastructure, specifically within its DNS management system that controls how user requests are routed to healthy servers, which led to the accidental deletion of all IP addresses for the database service’s regional endpoint.
“The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair,” Amazon said.
“When this issue occurred at 11:48 PM PDT, all systems needing to connect to the DynamoDB service in the N. Virginia (us-east-1) Region via the public endpoint immediately began experiencing DNS failures and failed to connect to DynamoDB. This included customer traffic as well as traffic from internal AWS services that rely on DynamoDB.”
The DynamoDB failure triggered cascading problems across AWS infrastructure, leaving DynamoDB’s DNS system in an inconsistent state that automated recovery couldn’t fix, requiring manual operator intervention.
Amazon has since disabled the buggy DNS automation globally and taken measures to avoid similar issues, including adding protective checks, improving throttling mechanisms, and building an additional test suite to help detect similar bugs in the future.
“We apologize for the impact this event caused our customers. While we have a strong track record of operating our services with the highest levels of availability, we know how critical our services are to our customers, their applications and end users, and their businesses,” Amazon added.
“We know this event impacted many customers in significant ways. We will do everything we can to learn from this event and use it to improve our availability even further.”
46% of environments had passwords cracked, nearly doubling from 25% last year.
Get the Picus Blue Report 2025 now for a comprehensive look at more findings on prevention, detection, and data exfiltration trends.