by Dave Moore, CISSP
12/12/2021
Most people in Oklahoma missed it, but Tuesday, December 7, National Pearl Harbor Remembrance Day, was also “Amazon Christmas Crash Day,” at least to those living in and depending on Amazon’s “US East 1” region.
In addition to being the world’s biggest online market (with a revenue of nearly $121 billion in 2020), Amazon also runs the world’s biggest cloud service, with Amazon Web Services (AWS) claiming almost half of the global public cloud infrastructure market. With revenues of $45.37 billion in 2020, thousands of online businesses depend on AWS for file storage, server and cloud computing, security, databases, commerce platforms and much, much more.
If AWS fails, so do the online businesses that depend on it, with the word “depend” being critical, here. Amazon founder Jeff Bezos once compared AWS to the public electric utility companies that sprang up in the early 20th century.
Back in the “olden times,” many factories would build their own electric power plants to meet their needs. When public utility companies became viable, private power plants fell out of favor. Most folks were more than willing to depend on someone else to bear the burden of building and providing power plant infrastructure and on-demand electricity.
Bezos figured Amazon’s AWS could do the same thing by building massive super-computing centers, and selling their use to online companies who would rather rent time on AWS’s infrastructure than spend their own money building and maintaining “private cloud” capability.
Bezos was right. Despite advice to the contrary, thousands of companies rushed to put all their eggs in the AWS basket, looking to reap the rewards of having big data power without having to build it, themselves.
That’s all great stuff, until something goes wrong. “Tolluntur in altum, ut lapsu graviore ruant,” wrote the ancient Latin poet Claudias Claudianus, court poet to the Roman emperors Honorius and Stilicho (300AD).
Roughly translated as, “they are raised to a great height, that they may tumble with a heavier fall,” Claudius’s proverb is said to have been adapted in modern times by boxer Robert Fitzsimmons, who, in a 1902 newspaper interview, said of his impending (and, much larger) opponent James Jeffries, “the bigger they come, the harder they fall.”
And fall, they did. Hard. Yesterday (as of this writing), December 7, 2021, vacuum cleaners, CoinBase, airline reservations, The Associated Press, light switches, Netflix, Disney+, auto dealerships, Roku, cat-food dispensers, Cash App, doorbell cameras, Tinder, auto dealerships, and Venmo all lost normal access to the Internet services they depend on to conduct business. Their “Big Daddy” business service provider, AWS, crashed.
Those are only the big names. Not included here are the thousands of smaller businesses that were crippled by Amazon’s failure.
Fundamental core Amazon services like delivery drivers and warehouse robots were also wrecked. Information and alarm devices and services Alexa and Ring, owned by Amazon, services millions of people depend on for information, safety and security, were also knocked out.
So, what happened? Who the heck knows? Amazon won’t say.
In typical, beat-around-the-bush, dance-around-the-problem big-bureaucrat fashion, Amazon has refused to say anything of substance. “A network event occurred,” “an unforeseen event was experienced,” “a misconfigured blah-blah-blah,” classic excuse-itus, “pay no attention to the man behind the curtain” responses are becoming sickeningly common when massive problems arise with giant tech companies. Amazons response has been no different.
“The root cause of this issue is an impairment of several network devices,” Amazon begrudgingly blurted out under pressure from customers. Does anyone know what that BS really means? I don’t, and I’ve been working this stuff for 35 years.
An “impairment of network devices?” That could be anything from a mega super-hacker Russian attack, to somebody forgot to plug the damn router in. Come on, Amazon. You can and should do better. It almost sounds like there is something here to hide. Is there?
Then Amazon, in a glaring faux pas of honesty, said, “We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time,” which is to say, “We are trying to fix things, but we don’t know what’s going on, or when anything will be fixed.”
This is all very bad timing. It reminds me of the Cox email crash of 2012. Cox Communication’s email services crashed for four days in a row, right before Christmas 2012, leaving millions of customers in at least 11 states without email service. This was especially hard on business customers during the holiday shopping season.
The same year, Amazon’s EBS cloud service crashed, knocking numerous websites off line and causing permanent data loss for many users. Apple’s iCloud service also crashed, leaving 15 million email users up in the air. Coincidence?
So, what was that thing about lots of eggs in the same basket?
Dave Moore, CISSP, has been fixing computers in Oklahoma since 1984. Founder of the non-profit Internet Safety Group Ltd., he also teaches Internet safety community training workshops. He can be reached at 405-919-9901 or www.internetsafetygroup.org