Risk Calculator Does Not Alter Surgeons' Choice to Operate

Kathleen Louden

October 15, 2015

CHICAGO — An online calculator that assesses patient-specific risk and guides surgical decision making changes the perceptions that surgeons have about operative risk, but not their decisions to recommend surgery, according to a randomized controlled trial.

The surgical risk calculator, developed by the American College of Surgeons (ACS) National Surgical Quality Improvement Program (NSQIP), has a "high degree of predictive ability," according to study investigator Greg Sacks, MD, a surgery resident from the University of California, Los Angeles, and a research fellow at the Robert Wood Johnson Clinical Scholars Program.

"We were surprised that surgeons were equally likely to recommend surgery," whether they used the calculator or not, Dr Sacks said during his presentation here at the ACS Clinical Congress 2015.

This is the first independent study to test the effect of the NSQIP risk calculator on physician behavior in clinical practice, he told Medscape Medical News.

"Our result shows how difficult it is for surgeons to predict postoperative risk for a specific patient," he said. "The decision to operate often involves a tradeoff between the risks and benefits of operating and not operating."

The calculator, which has been available since 2013, uses NSQIP data from more than 1.4 million cases and 393 hospitals to generate an estimated risk score for a patient with specific characteristics, preoperative risk factors, and type of surgery. It evaluates a patient's risk for an unfavorable outcome in the first month after surgery, but does not weigh in on outcome if surgery is not performed.

We were surprised that surgeons were equally likely to recommend surgery, whether they used the calculator or not.

Dr Sacks and his colleagues used an online survey of ACS members to determine whether the risk calculator would have any effect on a surgeon's perception of patient risk, and on the decision to operate.

A sample of nearly 800 respondents reviewed four clinical vignettes in which the role of surgery was unclear. In 395 cases, the information provided was supplemented with data from the risk calculator, and in the 384 cases that served as the control group, no additional information was provided.

Most of the surgeons reported routinely performing surgical repairs for the conditions presented in the vignettes, which were mesenteric ischemia, gastrointestinal bleed, small-bowel obstruction, and appendicitis, Dr Sacks explained.

For each scenario, surgeons predicted the probability of serious complications or full recovery in the 30 days after surgical or medical treatment, and noted whether they would recommend operating.

Risk projections varied widely, but the variations were less pronounced in the supplemented group than in the control group, Dr Sacks reported.

In fact, for all scenarios, risk predicted by surgeons in the supplemented group more closely resembled risk predicted by the calculator than risk predicted by surgeons in the control group.

For appendicitis, for example, the calculator predicted an average risk of 5%, the supplemented group predicted an average risk of 13%, and the control group predicted an average risk of 25%.

For recommendations to perform surgery, there were no significant differences between the supplemented and control groups (P = .34).

Table. Recommendation to Perform Surgery

Vignette Supplemented Group, % Control Group, %
Mesenteric ischemia 69 66
Gastrointestinal bleed 59 63
Small bowel obstruction 87 86
Appendicitis 54 50


Dr Sacks offered some theories on why the risk calculator does not appear to change clinical decisions.

"Perhaps the changes in risk prediction caused by the risk calculator weren't clinically meaningful enough to influence the decision to operate," he said. "Or when the surgeons found out that the risks of operating were lower than they thought, they appeared to think that the risks of not operating were also slightly lower — a spillover effect that resulted in no net change in the risks of operating relative to the risks of not operating."

This survey "gets to an area that is rarely studied — surgeon decision making," said study discussant Michael Englesbe, MD, from the University of Michigan at Ann Arbor.

"The risk tool is very good. I think developing a data display that is well suited to show to patients would improve the tool," he told Medscape Medical News.

Dr Sacks said he agrees that there needs to be a more efficient way to present complex probabilities data. He pointed out that future versions of the calculator should include the risks of not operating, as well as the benefits of surgery.

Suggested Improvements

Currently, the calculator has a version of the risk estimate that the surgeon can give to the patient, said Karl Bilimoria, MD, from the Northwestern University Feinberg School of Medicine in Chicago, who is an ACS faculty scholar who led development of the NSQIP surgical risk calculator.

"This tool is meant for the patient to use in discussion with the surgeon," Dr Bilimoria told Medscape Medical News.

Besides showing the patient his or her estimated operative risk, the risk calculator estimates whether the patient's chance of a poor outcome is below average, average, or above average for a specific surgical procedure, he explained.

He said the ACS has tested the risk calculator, and generally found that surgeons are very similar in their risk projections.

"Surgeons are pretty good at deciding where in the risk categories — high, low, medium — their patient falls," he explained.

Surgeons are encouraged to use the NSQIP surgical risk calculator when informing patients about the risks and benefits of surgery, Dr Sacks said.

"It's not perfect, but it's the best objective tool we have at this point to predict surgical risk," he pointed out. "Surgeons should use this tool to augment their own clinical judgment when providing risk information to their patients."

However, of the 1880 ACS members who responded to the survey, 45% were not aware of the NSQIP risk calculator. Of those who were aware of the calculator, only 6% reported using it routinely, Dr Sacks told Medscape Medical News.

Dr Bilimoria said that use of the risk calculator is increasing.

"The calculator is probably best advertised to surgeons in NSQIP-participating hospitals," he said. "But the ACS felt so strongly about this important tool that it opened it to the public, not just NSQIP members."

Dr Englesbe is a founder and stakeholder of Prenovo, a healthcare company that develops risk assessment tools. Dr Sacks reports receiving funding from the Robert Wood Johnson Clinical Scholars Program at UCLA. Dr Bilimoria has disclosed no relevant financial relationships.

American College of Surgeons (ACS) Clinical Congress 2015. Presented October 5, 2015.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.