Introduction
It’s a beautiful morning of October, when suddenly you cannot access any of your apps. Signal, Slack, Zoom, your presentation of Canvas, everything just stopped. Amazon services were down.
We live in a modern era where everything is done through cloud hosting, but this is such a big vulnerability and an availability loss, because this outage happened for 14 hours.
The issue
It was caused by DynamoDB’s DNS failure, even though it is an attractive choice for developers. It promises a 99% data uptime and low latency.
After 5 hours of network propagation, instances could not communicate to each other. Until a wonder happened.
The fix
Some where met with jokes, as the fix was Amazon's top engineer, James Hamilton, solving the crisis from his yacht's deck, far from any corporate hub.
Yeah, a yacht!
Obviously, the AWS team did a splendid job to what has been a stressful night’s work, but what can we learn and tie this to SPEAIT?
Even something as simple as a misconfigured network routing rule can disrupt thousands of systems. This is part of IT history, no matter how brilliant the machine, failures happen, just like ENIAC’s vacuum tubes often did. There will always be a need for someone who understands the basics and can fix it.
Hamilton's yacht:
https://x.com/GrantSlatton/status/1826800620399198502