7 Epic Datacenter Meltdowns

October 21, 2014
Quincy Johnson


Quincy Johnson

Digital Marketing Specialist

Maintaining a functioning datacenter is crucial for business. CMO by Adobe presents some staggering statistics that emphasize the need to maintain a healthy datacenter:

Q2 of 2013 marked the 15th consecutive quarter of positive year-over-year growth for e-commerce, with $4.7B in sales coming from mobile devices.

Mobile commerce transactions is projected to increase from $1.5T in 2013 to $3.2T in 2017.

52% of US consumers buy directly from brands online.

Key takeaway: Not only is consumer interest for purchasing online growing, but the medium in which they do so is expanding exponentially. For e-commerce sites, a datacenter meltdown could mean thousands of dollars per second, as was the case for Amazon when it lost an estimated $1,100 in net sales per second.

Below are 10 notable datacenter meltdowns which provoke the need for all of us – large or small – to question the health of our IT Infrastructures:

1.  Amazon Web Services (AWS) – $66,240 per minute for 30 minutes

Although many of AWS’s IaaS customers were unaffected, Amazon.com itself suffered a major blow to its revenue stream as its e-commerce site went offline for nearly 30 minutes. The last time Amazon experienced such a catastrophe was in 2008 with nearly $31,000 per minute revenue loss. However, Amazon stands as the symbol of versatility in the face of adversity, knowing datacenter outages are bound to happen and that Disaster Recovery is as essential to the cloud as the platform itself.

2.  Google – 5 minute stumble blackholes 40% of web traffic

For the 5 minutes Google went MIA, the World Wide Web witnessed a 40% depletion in global web traffic. Although 5 minutes sounds like an inconsequential timeframe, $14.1B in reported Q2 2013 revenue resulted in a jaw-dropping $108,000 per minute loss.

3.  Azure – Fortunate timing for a major release

Microsoft Azure experienced several outages throughout 2013, but dodged a potentially crippling blow on the eve of its Xbox One launch. Fortunately, Microsoft was able to correct the issue before consoles went on sale a few hours later, but other major Microsoft cloud services running on Azure such as Outlook and Office 365 went offline during the blackout. Pearl of wisdom – if your data center is going to crash, do it before a major release!

4.  New Jersey – The DMV just got worse…

New Jersey’s datacenter is no better off than your average SMB. Three times in Q3 2013, New Jersey’s IT systems shutdown due to minor issues such as power outage or fire alarms (without the ensuing fire), two issues which a separate, offsite datacenter can readily to go-live should the primary datacenter shutdown. 39 Motor Vehicle Commission offices across the state suffered a blackout, further distressing New Jersey citizens eager to renew their driver’s license or file tax information.

5.  Sears – $2.2M loss in profits, $2.8M to fix

This is an especially unfortunate case as third-party vendors typically promise a 99.9% uptime. With 4 power supplies infected at separate intervals, Sears suffered extensive downtime before resorting to generators to bring their servers back online. As if it couldn’t get any worse, the generator failed and Sears had to rent another for $13,500 per week – talk about insult to injury…

6.  Sony – Location, Location, Location

Just when we think we have all the technical ramifications considered, Mother Nature reasserts her dominance. The 2011 earthquake and tsunami devastated Sony’s 2011/2012 fiscal years, netting a total $3.18B loss. The earthquake measured a magnitude of 8.9, striking just off the eastern shore of Honshu Island; six Sony manufacturing facilities housing a variety of Sony technology were stationed nearby. Despite our best efforts to ensure the IT infrastructure’s technical standards, identifying the least hostile environment to house your organization’s IT backbone is essential to ensuring its structural integrity.

7.  Target – Pay attention to your security warnings…

And of course, the infamous Target debacle. Apparently various security systems including Symantec End Point Protection and FireEye – the software used by the CIA – caught the initial stages of this malware. Lesson learned – if your anti-malware system warns you of a security threat, don’t sing your own death knell by ignoring it.

Although we like to think IT systems are becoming increasingly infallible with each month’s technological innovations, the fact remains that datacenters are no more perfect than their creators. Monitor your datacenter’s health and implement a Disaster Recovery solution to prevent yourself from becoming the next datacenter meltdown headline.

Leave a Reply