Shelter From the Storm: Disaster Prep Lessons Learned

While offsite data backup and years of experience managing power systems during storms has improved disaster recovery efforts in healthcare, no amount of preparation is too much. From coastal hurricanes to inland storms and flooding, recent events have healthcare providers scrambling to re-visit their disaster plans, while those who’ve already weathered a storm share their lessons learned.

Before the storm

In the middle of the Atlantic Ocean, nine days before Halloween 2012, Hurricane Sandy began to swirl toward the U.S. By Oct. 25, the storm had danced across the Caribbean, already responsible for dozens of deaths in Haiti, the Dominican Republic, Cuba and more.

Sandy was projected to hit the U.S. somewhere in the Northeast, sending healthcare providers in the storm’s path into preparation mode. For Tobias Gilk, MArch, senior vice president at RAD-Planning in Kansas City, Mo., however, disaster planning happens well before storm clouds appear on the horizon.
“The more thought that can be paid in advance of a capital project—whether buying a new piece of equipment or building a new wing or renovating a suite—the better prepared you’ll be to respond to emergencies and mitigate the damage that comes from emergencies,” says Gilk.

Standing behind the notion that an ounce of prevention is worth a pound of cure, Gilk says providers can use facility assessments to detect weaknesses before a problem strikes. Even if there are issues identified that can’t be immediately addressed, considering possible scenarios can help in the development of contingency plans. For example, a rural hospital with limited access to power should identify its critical needs and design a backup generator strategy to maintain those services in the event that additional power supplies may not be accessible during a major storm. Equipment that isn’t an essential need should be removed from automatically receiving emergency power.

Natural disaster profiles available through the Federal Emergency Management Agency (FEMA) and state-level FEMA offices can help facilities with these priorities. The upside of contingency planning is that the first plan will have cascading benefits and responses translate over various scenarios, says Gilk. A blizzard or a hurricane could knock out power, but some aspects of the response will be similar for either event.

On preserving data, airtight fail-safes and backups are obvious elements of a disaster recovery plan, but a lot of care and thought must go into the decision of what to replicate and how a network is constructed, says Lynn Witherspoon, MD, system vice president and CMIO of Ochsner Health System in New Orleans. Witherspoon says that when Ochsner had to deal with Hurricane Katrina, they had two data centers—one on the hospital’s main campus and one at a secondary site nearby—that backed up EMR and PACS data, along with a traditional mainframe hot-site in New Jersey.

“Replication is one thing, and secondary hot-sites may be helpful, but you still have to have a functioning wide area network,” says Witherspoon. “Or else you have to have some sort of wireless topology that will enable you to make the connectivity.” Ochsner was fortunate in that they were able to maintain connectivity, and were able to access patient medication lists, imaging studies and more shortly after the storm had passed.

Plans also may help with unforeseen events. Gilk described a situation at one hospital that was built on high ground nowhere near a body of water. A natural flood would have been nearly impossible, but a pipefitter who broke a water pipe servicing a wing of the hospital managed to flood the ground level of the facility, demonstrating why all organizations should have a flood plan to protect vital equipment.

In the eye of the storm

Once in the path of a major storm, the next phase of disaster planning begins. At NYU Langone Medical Center in New York City, the radiology department asks its vendors to assess equipment as standard practice when there is potential for normal power to be lost, says Donal Teahan, director of radiology practice development. He says many systems are shut down, along with the power breakers to make sure there isn’t a power surge, which usually causes the biggest problems.

Beyond these steps, there aren’t many technical issues that need immediate preparation before a hurricane, says Teahan. “There’s little you can do until after the storm, because you don’t know what you’re going to be responding to.”

On the personnel side, establishing a line of contact with staff is key. At Beth Israel Medical Center in New York City, personnel were alerted prior to the storm, says Marc Katz, corporate director of radiology. Once it was announced that mass transit service would be shut down, essential personnel were told to report to the hospital where quarters were arranged for sleeping.

With staff communication accounted for, communication with patients is the next priority. NYU Langone set up toll-free numbers where calls could be forwarded if a normal hospital line was not connected. Patients could call the same number and get information even if normal phone service was cut.
With all reasonable preparations in place, all that was left to do was wait.

Landfall

Hurricane Sandy made landfall on the New Jersey shore the evening of Oct. 29, 2012. Buildings all along the Jersey Shore were leveled and the boardwalk was lifted off its supports. In New York City, the storm devastated outlying areas and sent a record 14-foot storm surge into the harbor. The massive amounts of water pushing against the city flooded subway tunnels and the East River spilled over onto 1st Avenue.

Power was cut across the city. In some areas, the electric company purposely cut power ahead of the surge to limit damage. Other areas lost electricity after an explosion that destroyed a transformer. Around 9 p.m., normal power at Beth Israel went out and emergency power kicked in. The entire main campus of NYU Langone Medical Center, however, completely lost power.

With some hospitals unable to function, patients were redirected to other facilities. Beth Israel took on patients from NYU Langone and elsewhere, and most medical records were transferred on CD and hard copy. Beth Israel’s data were secured with offsite data storage in New Jersey. “From that perspective, there was little impact to us,” says Katz.

Digital communication between hospitals allows for easier transfer of patient data, but requires pre-planning. “It has to be done beforehand,” says Michael McBiles, MD, chief of radiology at Nathan Littauer Hospital in Gloversville, N.Y. “If another hurricane came through and you didn’t have those agreements, you couldn’t set them up during the storm, because it does take some expertise to negotiate the firewalls and internet issues.”

In terms of preserving and backing up data, smaller, standalone facilities likely have technology in place through their vendors, says Gilk. “In some cases, the smaller guys might have it a little bit better in that they should be set up with an external, automated back-up system, typically through a third-party provider.” Large healthcare systems are in the same game as Amazon or other commercial data hosts, he says. They typically provide their own servers and in-house data management, and must ensure an appropriate level of redundancy and backup.

Despite the best planning efforts, some systems could be lost in a disaster scenario. The solution is often creative problem-solving. For example, PACS can be installed in a cart-mounted, mobile server if the main server room is compromised. Open source CDs of programs should be handy as backups to reinstall needed software. Smaller sites that don’t have a robust IT infrastructure can revert to old-fashioned processes—printed schedules and patient and staff contact information on paper.

Looking ahead

Every disaster is different and brings its own set of hard-learned lessons. Following Katrina, hospital building codes were revised after high winds and flooding hampered medical centers in and around the city of New Orleans, says Gilk. New York City Mayor Michael Bloomberg has stated that the city will craft future hurricane and flood protection infrastructure with climate change, and the stronger storms that are produced, in mind.

“Success depends on power. If you don’t have power, game’s over,” warns Witherspoon. In the wake of Katrina, Ochsner augmented their emergency generation system and added more fuel bladders. They also developed plans on where they’d send patients and where they’d get food and fuel if they had another extended outage. Every site has an immediate disaster response plan, but Witherspoon suggests sites put some thought into what would happen if they were in that state for a matter of weeks. Before Katrina, nobody had imagined how to deal with such widespread outages for an extended length of time. “I ran out of disaster plan on about day three,” says Witherspoon.

“If it hasn’t already happened to you, people have a hard time believing that it could, therefore the preparatory activities seem like a waste of effort,” warns Gilk. He acknowledges that healthcare providers have a lot of projects competing for time and effort, and that it’s easy for them to fall into the trap of not seeing disaster preparedness as the high priority it should be.

Each catastrophic event, however, is a reminder that disaster recovery should be a priority. Hurricanes, floods, power outages—all these and more can hamper normal operations and require contingency plans. Those unaffected by recent calamities should heed the lessons learned by those who were.

For those who were in the path of Sandy, the recovery progressing well. Teahan says the key is not to look at it as a disaster, but to “ask ‘what is the first small step I can take tomorrow?’”