How a Major Computer Crash Showed the Vulnerabilities of EHRs

Marcia Frellick

June 14, 2018

The recent communications outage at Sutter Health, the largest health system in northern California, which cut off access to electronic health records (EHRs), highlighted the frequency of such outages and the need for backup plans and drills nationwide.

The outage started at about 10:30 PM May 14 and lasted more than 24 hours, until Sutter announced systems were up and running at 2 AM May 16. During the shutdown, some elective surgeries were rescheduled, some procedures were delayed, and some patients were discharged. The 24-hospital health system executed its downtime plan and reverted to paper-based charts.

In a video message, Sarah Krevans, Sutter Health president and chief executive officer, told employees and physicians, "I am disappointed that this event meant we were not able to meet all the needs of all those we are privileged to care for."

She said that a multidisciplinary team, including outside independent advisers, would investigate the outage, which began when a fire-suppression system was activated at one of Sutter's data centers. There was no property damage to the data center.

Sutter spokesperson Dean Fryer told the San Francisco Chronicle that the cause for activation of the system was not a fire or data breach but did not disclose details.

Fryer told Medscape Medical News he would have no further comment beyond the May 16 update.

Patient Grief, Disappointment

Newspapers reported some of the fallout from the outage. The San Jose Mercury News reported the experience of Santa Clara resident Susan Harkema, who said, "The situation was horrifying."

According to the newspaper, Harkema's father died in the early morning of May 15 and she called Sutter Health's Hospice of the Valley to arrange for the removal of his body to a crematorium.

She reportedly tried the hotline and backup numbers but couldn't connect because of the outage. A hospice nurse arrived 8 hours later to collect the body.

"When you call expecting someone to pick up when you need them most and it just rings and rings, it was really a scramble and we were panicking," Harkema was quoted as saying.

Other patients took to Twitter to express their disappointment.

A patient who lives in Carmel, California, tweeted: "Had ultrasound scheduled; fasted with nothing to drink for 6 hours before scheduled appointment. Sent home due to outage. No good explanation provided for why ultrasound machine won't work when [information technology] is offline."

Another tweeted: "We had to go outside the Sutter Health system for critical care today, without the benefit of care history. It was stressful and scary, and we still aren't sure we have a successful outcome. We weren't able to consult with any of our specialists due to phone system issues."

All Systems Have Risk

Andrew Gettinger, MD, chief clinical officer for the Office of the National Coordinator for Health Information Technology, part of the US Department of Health and Human Services, said all systems need backup plans and pointed to the recommendation from the Joint Commission for annual disaster drills.

"It's not a question of IS your system going to be unavailable, because I think almost every computer system in every context is at some time or another not available," he told Medscape Medical News. "The question is then — what's the institutional contingency plan?"

Gettinger said that downtime for computer systems is not unlike other disasters health systems plan for regularly.

"It's no different from what happens when the power in the building goes out or the water supply goes out or you're no longer able to get compressed oxygen or nitrous oxide," he said.

"I don't think patients or doctors really need to be worried about it unnecessarily," Gettinger said.

SAFER Guidelines

All health systems should know about the SAFER guides (Safety Assurance Factors for EHR Resilience), put in place to address EHR safety nationally, Gettinger said. The guides were updated last year.

Dean Sittig, PhD, a professor at the University of Texas Health's School of Biomedical Informatics, helped write those guidelines and also was lead author on a study in 2014 that surveyed US-based healthcare institutions that were part of a professional collaborative on their exposure to downtime.

In that study, researchers found that nearly all (96%) of the 50 large, integrated institutions who responded had at least one unplanned downtime in the past 3 years and 70% had at least one unplanned downtime greater than 8 hours in the past 3 years.

The shutdowns have a wide variety of causes.

In another paper, Sittig wrote that, in April 2010, one third of the hospitals in Rhode Island had to delay elective surgeries and divert some patients when an automatic antivirus update crashed the system.

"You depend on the computer for everything — registration, scheduling, past visit notes, results of laboratory tests. The healthcare system is now dependent on the electronic health record to care for patients," Sittig told Medscape Medical News.

In the Sutter case, a fire-suppression system was activated. Sittig explained that the suppressions systems in data centers typically involve an alarm going off to alert people to get out of the room, then doors lock and all the oxygen is sucked out of the room and replaced with fire-retardant gas.

Because the gas has to be flushed out, then the oxygen levels restored, then the computers restarted, "you're talking probably a minimum of 4-6 hours," Sittig says. "That's when everything works perfectly."

He said systems should expect accidents to happen and that they will be costly.

"A big hospital probably loses at least $1 million per hour when they're down," Sittig said.

But investments in data protection can be a hard sell. A chief financial officer, Sittig said, may say a $3 million backup data center is too expensive, for example.

"You have to ask them, 'Can you afford to be down 5 hours? That will cost us $5 million. So we should spend the $3 million as an insurance policy,' " Sittig said.

Adding to the problem, he said, is that in the modern healthcare system, with an institution that's been using an EHR 5 or more years, many young providers have never worked in a place that has a paper system and aren't familiar with those operations.

Sittig added that paper systems are subject to their own dangers — fire, water, and wind, for example.

But electronic records that make it easy to spread information instantly across hospitals, sometimes in many states, also can mean instant, massive failures.

The first thing hospital systems do when a disaster strikes, Sittig says, is decide what can be cut, and the first thing to go is usually the elective surgeries. Then ambulances may be instructed to take patients elsewhere.

"Then you try to discharge the people who aren't very sick. Then they start sending people home early," he said.

"We've created a system where we're relying on an electromechanical device that we know is going to break. There's no question computers are going to break," he said.

For more news, join us on Facebook and Twitter


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.