Over 100 million people worldwide tuned in to watch Super Bowl XLVII. Therefore, it could be argued that was the most viewed and infamous power outage to wreak havoc on the grandest of scales.
It just goes to show, downtime happens.
We can’t really say for sure how or what occurred, although early speculation placed blame on Beyoncé’s lights-out performance, a manager at the Superdome, site of the game, said it was not the halftime show, but that a local energy company is claiming it had trouble with one of the two main lines that deliver power to the stadium from a local substation.
It could have been a software glitch, or a hardware problem that sacked power to the stadium for 33 minutes and left the NFL with a black eye. But the downtime incident powered a social media surge, as hundreds of thousands of people began Tweeting about the #poweroutage.
Which brings us to Twitter itself? Having suffered its own downtime nightmare back on January 31, Twitter was able to handle the blitz of people tweeting about the Super Bowl’s misfortune. Twitter announced it processed just over 24 million tweets during the game, with the mini missives coming in at a rate of 231,500 a minute during the power outage.
Downtime appears in many different forms and at many different times, across all industries and business landscapes. The Twitter downtime occurrence was much different from that the NFL witnessed, but both incidents took their tolls financially and in terms of a hit to brand reputation.
Within the enterprise there is an acceptable level of downtime that occurs each year. On average, businesses suffer between three and five hours of downtime per year, far too much in our humble opinion, at an average cost of $138,888 per hour. While that’s a staggering figure, the damage to the brand can be even more catastrophic.
Let’s get back to the Super Bowl and the power outage. The City of New Orleans, which hosted the game, is already worried it’ll lose out on hosting future games because of what happened. That’s a city known for its ability to show its visitors a good time, but those businesses that depend on major events like the Super Bowl to draw in tourism dollars could suffer from that 33-minute absence of electricity.
Again, downtime comes in many forms depending on the industry and the ramifications have the potential to throw their victims for a significant loss. It’s like that old saying that you need to expect the unexpected. When the unexpected does arrive you have to be prepared to come back from that downtime swiftly and with as little disruption to your business as possible. With the right technology and the right best practices in place, you can minimize the damage and decrease the chance of downtime seriously hampering your ability to do business.
In today’s always on, increasingly complex IT world, CIOs are faced with an unprecedented set of challenges — Big Data, virtualization and cloud computing, the consumerization of IT applications and devices, mobility, overburdened IT staffs and social networking – the list goes on and on. Never before has this confluence of factors created a digital demand mashup of this complexity. Layered on top of this complexity are the demands for information from employees, customers and partners to do their jobs.
These developments are fundamentally changing the scope and nature of what is considered mission critical for IT. In the past, the definition of what constituted mission critical was relatively well understood. Financial, supply chain and telecommunications applications were considered the ones that couldn’t go down. Today, that mission critical definition is expanding quickly. Areas such as analytics, sales force automation, CRM, web content, social applications and logistics, which all impact the customer experience, have pushed their way into the mission-critical category.
Broadening the mission-critical category brings many more users into the fold, which also expands the demand for access to information. This, in turn, increases the expectations on application availability. Applications that are unresponsive or altogether not available translate into lost revenue at that given moment in time, as well as take a toll on brand reputation in the future. Here are a few recent high-profile examples of how application downtime has negatively impacted organizations:
According to Stratus, CIOs on average put up with three to five hours of downtime per year, with most servers experiencing 44 hours of downtime over the life of that server. But, the aforementioned factors are redefining how much downtime is acceptable especially as costs rise. Consider this: the cost of downtime per hour, which has recently been estimated to cost on average $138,888 per hour, as reported by the Aberdeen Group, is 38% higher from the cost of downtime in 2010. The cost of downtime ranges by industry from $181,770 for average companies to $101,600 for best-in-class companies.
So, how come businesses essentially “whistle while walking past the graveyard” when it comes to downtime? Why do they think downtime is acceptable, when it’s not, especially as the data center goes through this transition? This notion should really be rejected as a business norm. Technology exists to prevent downtime, boasting six 9s or 99.9999% of availability. With this technology, proper procedures and monitoring in place, downtime can be practically eliminated.
To understand downtime and how costly it can be, you should first understand the three classifications of downtime:
Now, multiply the various amounts of downtime to see how much your company is losing in revenue alone from downtime.
As mentioned before, downtime doesn’t have to occur, and much of the reason it does is because everyone just assumes that at some point servers will go down and applications will follow suit. To combat downtime, you have to approach the challenge from the point of view of what needs to be done to keep applications available all the time. This incorporates the use of smart technology, fault-tolerant, high-availability servers that ensure 99.9999% uptime, and smart management practices, including proactive monitoring of all mission-critical systems that detect outages before they occur. This is true across the data center and cloud infrastructure.
As people continue to rely on applications for a myriad of purposes, businesses will need to make sure these applications are responsive and always available, accepting any downtime is simply not an adequate expectation.
Have you ever thought what a minute of your time is worth?
Let’s say you get paid $60 an hour – then one minute is worth $1. If you are reading this, then, my bet is you are probably willing to spend $1 waiting for an answer. Chances are you will wait much longer, especially if it’s on someone else’s dime. But, how long is too long to wait?
If you run a 911 response center (emergency phone service in the USA) then one minute of downtime is not measured in dollars but lives. Maybe you are the IT manager of a financial company, how many credit card transactions could you lose in one minute? One hundred? One thousand? Maybe many more.
In both these and many other commercial examples, the cost of downtime is both known and quantifiable. Businesses not only perform risk assessments on downtime but also, they make business decisions to avoid it. In 2011, eWeek reported a business could lose an average of about $5,000 per minute in an outage. As they say, “at that rate, $300,000 per hour is not something to dismiss lightly.” Given that, I think we can all agree for critical business applications – uptime is pretty important to many business and now, to me too.
This week, I start a new job as chief marketing officer at Stratus Technologies, one of the world’s leaders in ensuring up-time for your applications. You will find our software and servers behind many things you use day-to-day and you would be pretty upset if they didn’t work. Examples would be supporting credit card transactions and 911 services. What makes this role interesting is not just these types of services, but also, how our solutions apply to others. Let me give you an example.
I was sitting in the Austin airport waiting to board the first of two flights that would take me to Boston, my new home. I wanted to let my friends on Facebook know that I had started my journey so I thought I would check-in on Foursquare – which automatically updates Facebook. Foursquare is down.
I wait until Dallas (I am changing through DFW – one of the downsides to Austin) and Foursquare is still down. When I arrive in Boston, hours later, Foursquare is up, so I check in. Of course, I could have given up on Foursquare and just checked-in on Facebook. In the cloud, there are often alternative ways of doing things.
This may seem like a trivial example, especially compared to a 911 service, but if you are Foursquare and in search of a business model, I suspect this is not good news. As that social site looks to monetize its platform, my guess is it will use ads. I need to be on the Foursquare service to see the Ads. Another outage like this and I will not be on the service. The reality is it may not have been the site’s fault, it maybe the service provider’s fault, but as a user, I don’t care.
Just a few weeks ago, in the CIO section of the Wall Street Journal, they reported “Netflix Amazon Outage Shows ‘Any Company Can Fail’.” Forrester Research analyst Rachel Dines is quoted as saying, “It’s all about timing. This was a big deal because it was one of the worst possible times it could happen as families gathered during Christmas to watch movies.” OK, so families could have talked to each other, but you get the point and there are plenty of other alternatives to Netflix.
What excites me about Stratus Technologies was not just how our technologies applies to established commercial businesses but to these new cloud-based services. I have no doubt that as cloud applications become more important in our lives, Stratus Technologies will have a critical role to play in making them available all the time.
For now, I have a lot to learn about the business and I look forward to blogging about it as I go.
Everybody is texting these days. Teenagers, soccer moms, business people, and even grandparents have jumped on the bandwagon, sending more text messages, photos and videos than ever before. In fact, according to a 2011 Pew Internet survey, “Americans and Text Messaging,” 73 percent of cell phone users text, and nearly one-third of them would rather text than talk. With texting on the rise, it’s inevitable that 9-1-1- technology must evolve to meet the needs of today’s mobile citizens. That’s what Next Generation 9-1-1 (NG9-1-1) is all about.
NG9-1-1 is a national initiative that aims to update and improve emergency communications services. The end goal is to upgrade the country’s 9-1-1 infrastructure so that the public can not only call, but also transmit text, video, photos, and more to a Public Safety Answering Point (PSAP). In turn, the PSAP will be able to process the data, transmit it as necessary, and get it out to first responders. Unlike today’s system, the new infrastructure will also support the transmission of calls and information across county and state lines. These enhanced capabilities will be instrumental in increasing public safety by helping law enforcement, firefighters, EMTs, and other first responders get better information about the situations they face in the field.
While migrating your PSAP to NG9-1-1 may seem overwhelming at first, proper planning can help ensure a smooth and manageable transition. Give careful upfront consideration to all your technology needs — ESInet, CTI software, CAD systems, mobile data networks, TDD software, and more. Think about how you will fund your NG9-1-1 system. Explore potential liability issues. Create a public education plan. And figure out the best way to protect your NG9-1-1 solution against downtime that could lead to tragic consequences. Looking for practical advice on how to successfully move your PSAP to NG9-1-1? Download our informative white paper, “What You Need to Know About Migrating to Next Generation 9-1-1 Technology,” to learn more.
Just because a crashed system is back up and running doesn’t mean downtime cost stop mounting. Customer dissatisfaction and tarnished reputations can exact a toll long after an outage. In the case of cloud computing the impact may well spill over to your customers’ customers. A recent Tumblr outage lasting four hours meant its customers lost almost 100 million views.
Over the past year we’ve seen notable outages followed by profuse apologies and offers of compensation. It really doesn’t help the customer much, but the service provider’s total bill for the reparations can be steep and, in the end, customer may not be too inclined to forgive and forget.
Here’s a look at what some companies offered following bouts of downtime:
On October 26, Google App Engine experienced tough times for roughly six hours. To make up for the inconvenience, Google issued credits to all paid applications for ten percent of their usage for the month of October to cover any SLA violations.
Wouldn’t it more efficient and cost-effective to prevent outages rather than recovering and apologizing after the fact? Keeping operations and business applications online 24/7/365 equals happy customers, regardless of business size or industry. To learn more about what solution is the right fit for your business, visit our homepage and check out the uptime meter.