Data centers, and the firms that rely on them, can be hardened against natural disasters or acts of sabotage, but they are part of a larger infrastructure—such as transportation, telecommunications and power—the vulnerabilities of which were exposed to the financial community by Hurricane Sandy.
“Hurricane Sandy taught us that disaster impacts can be regional and not merely local,” said Terry Rennaker, vice-president at data center construction firm Skanska USA’s mission critical center of excellence.
Terry Rennaker, VP, Skanska USA’s mission critical center of excellence
As people and communities begin putting the pieces back together following the Superstorm last month, investment managers are evaluating how their businesses fared throughout and following the storm. In many cases, new lessons have been learned over the past few weeks and, by extension, firms are making changes to their business continuity and disaster recovery planning.
“A number of investment managers either did not have true disaster recovery plans or the plans were notably insufficient,” said Phil Niles, director and client executive at fund administrator Butterfield Fulcrum. “Given that we are just 11 short years removed from 9/11, it is nearly inexcusable for any business based in New York City to lack a robust business continuity plan, including asset management firms.”
Skanska’s mission critical center of excellence specializes in “next-generation” data center infrastructure. It has constructed new data centers and renovated existing ones for eBay, Royal Bank of Canada, TELUS Communications, IBM, CSC, Goldman Sachs, AT&T and Verizon, among others.
Skanska’s approach to the construction of data center facilities is not simply a contractor responsible for design and construction services, the company says. Instead, it views its role as a provider of expertise along the entire data center infrastructure supply chain.
The need for such expertise was made only too apparent by Sandy.
“The wholesale destruction and disabling of infrastructure and the crippling of the support systems of repairing that infrastructure created an enduring disaster that stretched the limits of the definition of disaster recovery,” said Rennaker at Skanska. “Therefore, we learned that separation of sites with more robust natural disaster plans that are routinely drilled and tested is necessary to survive such a disaster.”
A major area that needs to be revised surrounds the assumptions and logic driving many disaster recovery plans.
“The majority of such contingency strategies revolve around what happens when a disaster strikes the offices of the investment manager, knocking out power or rendering the site unusable for business operations,” said Niles at Butterfield Fulcrum. “In these cases, it is typically assumed that, in our electronically connected world, staff can work from home. For many business continuity plans, this was the extent of the strategy.”
The Human Element
However, less thought was given to what happens when the office of the investment manager is relatively untouched, but staff are unable to commute or are without power at their own homes.
“During Hurricane Sandy, travel was incredibly difficult, especially to and from Manhattan, meaning that employees could not readily access their offices,” said Niles. “At the same time, many staff experienced widespread and long-lasting power outages at their residences, making work from home impossible. This left many investment managers short on internal resources with no foreseeable way to bring these employees back online.”
“[Hurricane Sandy] demonstrated the fragility of many firms’ primary data centers where primary equipment is still being housed in mini data centers in their offices,” said Shawn Kaplan, general manager of financial services at Telx, a data center co-location and interconnection services provider. “The qualities of good office space (nice views of the water) are unfortunately not the same as a fit for purpose data center.”
Even corporate office systems such as phone systems, accounting/payroll and file shares need to be incorporated into off-site plans.
“We heard a report of at least one firm who lost their global phone system across dozens of cities because it was housed in their corporate headquarters,” said Kaplan.
Paladyne Systems, a provider of investment management software, reports that its response to the disaster was a net positive experience for clients.
With the advantage of having built everything from the ground up when it was founded in 2005 and a chief technology officer, Christian Robertson, who stays awake at night thinking about “what if” scenarios, Paladyne was able to deliver a seamless response to clients during the storm.
“Paladyne’s data centers and systems remained uninterrupted and staff were all available, despite an office closure in Teaneck, New Jersey, due to relentless contingency planning for both IT staff and the systems they support,” said Robertson.
In fact, Paladyne had performed a business continuity planning/disaster recovery simulation the weekend before the storm, despite having completed a full disaster recovery test just two weeks prior as part of the regularly scheduled quarterly exercise.
No Plan, No Gain
While the act of planning can be as important as the plan itself.
One of the major lessons of Hurricane Sandy is “to have a plan, because even if the plan does not speak directly to the emergency at hand, the act of creating a plan is a powerful management tool for unexpected events”, said Alexander Tabb practice leader and managing director for crisis and continuity services at Tabb Group, a capital markets consultancy.. “It allows you to understand how things are interconnected.”
Tabb’s expertise in crisis management runs deep. Prior to joining Tabb Group in 2004, Tabb worked at Kroll Inc., an international risk consulting company, where he was associate managing director of services.
Another lesson learned was that business continuity plans shouldn’t preclude the option of shutting down services for a time, as did the major exchanges.
“After Sandy, exchanges and major sell-side institutions closed down for a few days,” said Tabb. “What leadership did was weigh the pros and cons of implementing failover systems versus closing for two days and then resuming at a full level of service. Business continuity by its nature means providing less service than during normal times.”
