Mandatory adoption of surgical safety checklists at hospitals in Ontario, Canada, did not produce significantly better outcomes soon after the hospitals implemented the checklist procedure, according to an article published in the March 13 issue of the New England Journal of Medicine.
Checklist advocates, however, say that short-term evaluation of the effectiveness of checklists falls short and that it takes a longer term to see results of what amounts to a cultural change in operating rooms.
David R. Urbach, MD, from the Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada, and colleagues analyzed survey responses from 101 Ontario hospitals that introduced a surgical safety checklist between June 2008 and September 2010. Researchers also analyzed surgical outcomes data using population-based administrative health data.
The authors studied 3-month intervals before and after implementation of checklists. The first period (109,341 procedures) ended 3 months before implementation, and the second period (106,370) started 3 months after implementation.
The researchers found that the adjusted risk for death during hospitalization or 30 days after surgery decreased from 0.71% (95% confidence interval [CI], 0.66% - 0.76%) before implementation to 0.65% (95% CI, 0.60% - 0.70%) after surgery but did not reach statistical significance. The adjusted risk for surgical complications decreased from 3.86% (95% CI, 3.76% - 3.96%) before implementation to 3.82% (95% CI, 3.71% - 3.92%) after surgery, still without statistical significance.
What did reach statistical significance were decreases in length of stay and rate of emergency department (ED) visits within 30 days of surgery. The adjusted length of stay decreased from 5.11 days (95% CI, 5.08 - 5.14 days) before implementation to 5.07 days (95% CI, 5.04 - 5.10 days; P = .003) after surgery, which the researchers characterized as "significant but small and clinically unimportant."
"The only complication for which the risk significantly decreased was an unplanned return to the operating room (from 1.94% [95% CI, 1.87 to 2.00] to 1.78% [95% CI, 1.72 to 1.85], P = 0.001)," the researchers write. They actually found increases in the adjusted risks for deep venous thrombosis and ventilator use.
The researchers adjusted for age, sex, whether a procedure was an emergency or elective and inpatient or ambulatory, and other variables.
Of the 101 hospitals, 92 provided the checklist they used, which broke down to 79 using Canadian Patient Safety Institute versions, 4 using World Health Organization versions, and 9 using customized checklists. The researchers write that almost all hospitals reported 99% or 100% compliance, with the lowest compliance rate reported at 91.6%, and that the effect of the checklist "did not vary substantially according to the type of checklist used."
Mandated Reporting, but Not Mandated Training
The Ministry of Health and Long-Term Care mandated in July 2010 that hospitals report adherence to surgical safety checklists in Canada but did not require formal team training before reporting. The researchers report in the current article that "materials were available to assist in the implementation," but no formal training was required, and implementation was not standardized.
In an accompanying editorial, Lucian L. Leape, MD, from the Harvard School of Public Health, Boston, Massachusetts, writes, "What are we to make of this? First, it is important to state the obvious: it is not the act of ticking off a checklist that reduces complications, but performance of the actions it calls for."
He writes that fully implementing a checklist is difficult, should be adapted to local needs, requires resources or expertise that many hospitals may lack, and requires buy-in from all staffers involved; he also adds that full implementation takes time.
As for mandating the use of checklists, Dr. Leape writes, "Regulation works best when a practice of unquestioned value has become the norm. We are not there yet."
Full Compliance?
He suspects that lack of positive results in the study could be because the checklist "was not actually used. Compliance was undoubtedly much lower than the reported 98%."
Atul Gawande, MD, MPH, from Harvard Medical School and Harvard School of Public Health, who was senior author on the 2009 study that prompted hospitals around the world to consider adopting surgical safety checklists, agrees. "We know virtually no hospital achieves that level of compliance," he told Medscape Medical News.
He points out, however, that, "They did show a reduction. They dropped 6 points per thousand in the 3-month period that they measured, and that's in line with what the [Veterans Administration] achieved when they implemented the surgery checklist, when they measured only over a short time period. When [the VA] then followed it out for a year, they had an 18% reduction in mortality across 74 hospitals."
He continued, "The idea that you could measure just 3 months after you've done an implementation, and expect that everywhere has adopted and adopted well, is pretty unlikely. It's a bit of a premature study."
He said the VA achieved success after they did comprehensive training in the beginning, got everyone including operating room personal and administrators involved in implementation, and followed-up with coaching and getting feedback.
He said Scotland, too, has shown improvements, "with active clinical engagement at every hospital and ongoing coaching in how to do it," but, "It took time." Three years after implementation, Scotland's death rate has dropped each year by the same amount shown in the current study, he said.
"With that steady drop, they've documented more than 9000 lives saved. They've dropped their death rate below 0.5% for the first time ever," Dr. Gawande said.
After 5 years, "We have a reduction in death following surgery of 23%," Jason Leitch, BDS, clinical director of the Quality Unit of the Scottish Government, Edinburgh, wrote to Medscape Medical News. The checklist is part of a larger program called the Scottish Patient Safety Program, he added.
Getting results "takes significant time, at least a year or 2. It is a big culture change."
This research was supported by the Canadian Institutes of Health Research and the Institute for Clinical Evaluative Services. The authors, editorialist, Dr. Gawande, and Dr. Leitch have reported no relevant financial interests.
N Engl J Med. 2014;370:1029-1038. Article abstract, Editorial extract
Medscape Medical News © 2014 WebMD, LLC
Send comments and news tips to news@medscape.net.
Cite this: Surgical Safety Checklist Use Shows Slow Progress in Ontario - Medscape - Mar 13, 2014.
Comments