Businesses reconsider what it takes to get up and running when disaster strikes.
While disaster planning and business continuity planning are not inexpensive, but compared with the costs of having a business shut down, they are bargains.
“One lesson from Hurricane Sandy is that hard work and pre-planning exercises and tests were well-worth it because there was minimal impact, if any, and clients experienced zero degradation due to the storm,” said Justin Llewellyn-Jones, chief operating officer at Fidessa U.S.
Sandy suggests that greater physical separation is a sound option. This would demand good, low latency connectivity between data center locations and separate power infrastructures.
Mark Akass, chief technology officer, BT Global Banking & Financial Services
“However, if just one participant does this — for example by building a geographically separate facility in Ohio — what's the point?” said Mark Akass, chief technology officer at BT Global Banking & Financial Services. “There'd be no-one to trade with. The market needs to behave as a whole to get true value. Planning for disasters needs to take place across the industry in major hubs as a whole. In some way, this is lacking.”
Another lesson taken from Hurricane Sandy is one of industry coordination.
“Sifma tried to bring the financial services industry together and determine if markets should be open or closed, and then managed the response after the Hurricane to ensure that there was an orderly recovery process,” said Llewellyn-Jones. “In the future, a unified response will also need to include coordinated communications through the different utilities servicing a large percentage of the marketplace.
Industry Leaders
Other industries can learn from what the financial services industry has done in regard to getting the markets up and running quickly.
“It's always been the case with financial services that robust infrastructure is a fundamental requirement,” said Akass. “In the aftermath of 9-11, the industry took a hard look at its underlying fabric to see where the vulnerabilities lay.”
The high concentration of data centers in the New York metropolitan area argues for having some physical separation between primary and backup locations.
“One of the things that the tragedy of Hurricane Sandy highlighted for us was how data centers no longer need to be tethered to the population centers that they serve,” said Tate Cantrell, chief technology officer at Verne Global, a data center operator with a 100% renewably powered data center campus in Iceland.
Companies should segment their business applications and choose platforms that match technical, financial, and sustainability goals that are set forth for each application.
“Hurricane Sandy reinforces the point that a location like New York, while close to a population center, is not an ideal location for the critical equipment that data centers relay on for delivery of service,” Cantrell said.
To Reopen or Not to Reopen?
Deciding when to reopen following an outage can be a tricky one. The New York Stock Exchange, for example, chose to shut down for three days, until it could be certain that it could return to 100% service levels.
“Keeping the market open on a purely electronic basis, with the market having never operated this way even under perfect conditions, would only increase the chance of any minor malfunction to a high frequency trading algorithm, causing potentially great disruption,” said Michael Versace, director of worldwide risk and Big Data industry lead
at International Data Corp.
Without a full complement of engineers to recognize and rectify any algorithmic anomaly, erroneous trading strategies could cause widespread panic in an already stressed climate, exacerbated by a lack of experienced traders that can communicate to dampen volatility.
“Any decision, therefore, to re-commence trading when the markets open should be made against a thorough risk mitigation and business continuity strategy to ensure that sufficient numbers of technical and operational personnel to power the trading systems and traders are available to execute trade orders,” Versace said.
The aftermath of Sandy has proven “how critical a holistic emergency solution
Is.” said Sev Onyshkevych, chief marketing officer for FieldView Solutions, a provider of data center infrastructure management (DCIM) Services. “Not just a solution that monitors servers and switches, but the entire power supply chain - from back-up generator, fuel for the generator, UPS, cooling units, etc.”
Due to the storm, staff at certain institutions were unable to reach various locations - proving the critical need for a remote monitoring solution without compromising connectivity.
“It also caused data center operators to rethink some design aspects that link to outside of these self-contained fortresses,” said Onyshkevych. “For example, if the roads approaching your data center are impassable or washed out, having arranged diesel fuel for your standby generators is not adequate.”
Ralph Wynn, senior product marketing manager at FalconStor Software, provides the following recommendations.
"Identify what you cannot live or do without when it comes applications and resources,” he said. “Without knowing which applications and servers need to be available during a disaster how can you plan for a disaster?”
Next, create a recovery plan. “Backup solutions tend to lack the level of recovery that allows you to recover multiple servers simultaneously, so this prolongs recovery of data center services,” Wynn said.
Finally, testing is all-important. “Use a solution that allows you to test in the same manner as the type of intended recovery,” said Wynn. “Make sure this solution can account for changes in hardware and other environmental variables as well as allow for scripting for offering advanced recovery options.”
