Like the Eveready battery, data centers in the New York metropolitan area just kept on running during and after Hurricane Sandy struck in late October, winning plaudits from industry participants.
“If there was a single takeaway from Sandy for anyone in the IT infrastructure space it was that the data center providers our industry have grown to depend on actually know their stuff,” said Ken Barnes, senior vice-president of corporate development at Options, a provider of private cloud services platform for the exchange, banking, trading and investment communities.
While motorists faced fuel scarcities around the tri-state area of New York, New Jersey and Connecticut, the generators powering these facilities hummed on, the servers running the industry blissfully unaware of the mayhem outside, Barnes said. “Despite the fact that a number of these data centers are located in flood plains around the low lying New Jersey wetlands, the investments in power and cooling resiliency made by the Equinixes, the Savvises, the Telx's of this world proved themselves of withstanding not just the natural disaster of Sandy but, equally critically, the logistical storm that followed,” he said.
Now over two months since the storm wreaked devastation over a large portion of the area’s transportation and utilities, as well as residential areas, lessons are still being distilled about business continuity planning and disaster recovery.
“One of the key elements of any disaster recovery plan is a review of lessons learned,” said Walter Dearing, senior director of resource management at trading and technology firm SunGard’s Availability Services unit. “In many cases, risk mitigation is viewed in terms of certain core assumptions; for example, the assumption that earthquakes are more prevalent in one geography over another. As a result of Sandy, I think that the view of what might happen, especially in terms of natural disasters, will grow exponentially with more varied possibilities taken into account.”
One of the major lessons learned from the Superstorm is the need for customers to pre-position their own key recovery personnel close to their projected recovery sites, Dearing added. This would minimize the transportation challenges which arise when municipalities close key arteries.
Many companies in New York City were caught off-guard by the failure of their protected and redundant power.
“Companies will evaluate just how hard their infrastructure really is,” Dearing said. “A number of companies had their protected and redundant power sources below street level. These failed as the basements of buildings filled with sea water. Similar failures occurred with communications lines located underground.”
A case in point is Allied Building Products, a buildings material distributor.
“At approximately 10pm ET on October 29, Hurricane Sandy flood water started gushing in through the front door at our Allied Corporate office in Jersey City, New Jersey,” said Scott Fischer, director of IT operations at Allied Building Products. “By 10.30pm it was determined there was nothing we could do to stop the flood of water. It was estimated the water level outside the building was approximately five feet. The data center was determined to be a total loss or at the least unusable for the foreseeable future.”
Fischer called SunGard Availability Services, and it was determined that Philadelphia was the next best location for the company.
“SunGard Availability Services’ calm cool approach to my disaster declaration put many of my fears to rest,” said Fischer. “They worked day after day, sometimes up to 20 hours a day, to restore our servers and recover various functions. Whatever the issue, they thought of a creative way to solve it.”