Customers could be hit by data centre cutbacks

IT staff cutbacks could be causing supply problems say industry observers, citing several recent examples of downtime where customers have been left with no service following techncal problems.

Data centres have been hard hit by staff cutbacks.  According to a survey of 300 IT centre managers last year by the Association for Computer Operations Management, half of all data centres were planning to cut 2009 budgets by an average of 15 percent. Respondents at 14 percent of those companies said the cuts would include layoffs of IT staffers.

Now customers are counting the cost. When eBay's PayPal unit suffered a worldwide outage early this month, Sailrite Enterprises, a sailing supply company lost its critical customer payment services for six hours. The next day, PayPal's services failed Sailrite again - this time for about an hour, according to Matt Grant, a vice president at Sailrite. Grant was not amused. He posted a blunt message on PayPal's blog site: "This is not acceptable."

PayPal blamed the outage on a problem with a "back-end router" that was complicated by a failure in the company's redundancy measures.

The PayPal electronic payment system is one of many Internet-based services that have been hit with outages. And based on news reports, the number of such incidents appears to have been increasing in recent months, analysts said. They cited shutdowns of the Google Apps software hosted by Google Inc., outages at data centers run by Rackspace Hosting Inc. and a distributed denial-of-service attack on Twitter.

Observers pointed to several possible reasons for the apparent uptick in online outages, including IT budget and personnel cutbacks, increasing corporate dependence on hosted applications - and bad luck.

Kurt Roemer, chief security strategist at Citrix Systems said he wondered whether a two-hour shutdown of Cisco Systems website this month "would [have] happened a few years ago... when they had multiple people checking every single change." Cisco blamed the outage on human error.

Ken Brill, executive director of Uptime Institute, a data centre engineering and consulting firm, said such budget and personnel cutbacks could prove disastrous to IT. "We're not doing the maintenance we should be doing, and when you don't do maintenance, you increase the probability of catastrophic failure," he said.

Brill added that energy-efficiency efforts may be prompting data centres to cut back on redundant equipment and run their systems harder, exposing equipment flaws.

Leslie Daigle, chief technology officer of the Internet Society standards group, said that the online service business will probably survive despite periodic outages. "The Internet is in a state of constant evolution, and that really does provide its overall resilience," she said.

And some IT managers say that recent outages aren't changing their minds about using hosted services.

For example, Ted Maulucci, CIO at real estate developer Tridel, defended the durability of hosted data centre systems, despite the fact that his provider had suffered an outage. "Five years ago, it was not uncommon to experience the odd interruption, whereas today it has been pretty rock-solid, other than the major failure that happened," he said.

Neal Puff, CIO for Yuma County, Arizona., agreed, even though the county moved its ERP system from a cloud-based provider back in-house last year. Puff said he still believes that "a well-run hosted solution with reliable connections will perform well and be as reliable as any well-run in-house system."