Surgical Risk Is Not Linear

Derivation and Validation of a Novel, User-Friendly, and Machine-Learning-Based Predictive OpTimal Trees in Emergency Surgery Risk (POTTER) Calculator

Dimitris Bertsimas, PhD; Jack Dunn, PhD; George C. Velmahos, MD, PhD; Haytham M. A. Kaafarani, MD, MPH, FACS

Disclosures

Annals of Surgery. 2018;268(4):574-583. 

In This Article

Abstract and Introduction

Abstract

Introduction: Most risk assessment tools assume that the impact of risk factors is linear and cumulative. Using novel machine-learning techniques, we sought to design an interactive, nonlinear risk calculator for Emergency Surgery (ES).

Methods: All ES patients in the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) 2007 to 2013 database were included (derivation cohort). Optimal Classification Trees (OCT) were leveraged to train machine-learning algorithms to predict postoperative mortality, morbidity, and 18 specific complications (eg, sepsis, surgical site infection). Unlike classic heuristics (eg, logistic regression), OCT is adaptive and reboots itself with each variable, thus accounting for nonlinear interactions among variables. An application [Predictive OpTimal Trees in Emergency Surgery Risk (POTTER)] was then designed as the algorithms' interactive and user-friendly interface. POTTER performance was measured (c-statistic) using the 2014 ACS-NSQIP database (validation cohort) and compared with the American Society of Anesthesiologists (ASA), Emergency Surgery Score (ESS), and ACS-NSQIP calculators' performance.

Results: Based on 382,960 ES patients, comprehensive decision-making algorithms were derived, and POTTER was created where the provider's answer to a question interactively dictates the subsequent question. For any specific patient, the number of questions needed to predict mortality ranged from 4 to 11. The mortality c-statistic was 0.9162, higher than ASA (0.8743), ESS (0.8910), and ACS (0.8975). The morbidity c-statistics was similarly the highest (0.8414).

Conclusion: POTTER is a highly accurate and user-friendly ES risk calculator with the potential to continuously improve accuracy with ongoing machine-learning. POTTER might prove useful as a tool for bedside preoperative counseling of ES patients and families.

Introduction

The burden of emergency surgical disease has continuously increased over the last 2 decades. Between 2001 and 2010, the United States alone reported >27 million Emergency Surgery (ES) admissions accounting for 7.1% of all hospitalizations.[1] The correlation between ES and adverse outcome has been studied extensively: when compared with similar elective surgery, ES carries a much higher risk of postoperative morbidity and mortality.[2–4] The ability to reliably predict postoperative risk is critical for surgical decision-making, counseling of patients and families, resource allocation, and quality benchmarking. The existent risk stratification models range from the simple and subjective, like the American Society of Anesthesiologists (ASA) classification,[5] to the comprehensive, like the Elixhauser[6] and Charlson[7] Comorbidity Indices. The American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) has also produced its own Surgical Risk Calculator (ACS-SRC).[8] Given that most of these models have been created with the elective surgical patient in mind, many studies have questioned their performance in ES.[9–10] Because of that concern, the Emergency Surgery Score (ESS) was recently suggested as a better predictive model of mortality and morbidity after ES.[11–13] All these aforementioned risk calculators (including ESS), although useful, assume that the variables in their models interact in a linear and additive fashion. The mathematical and medical realities, however, suggest that the interaction of comorbidities and markers of disease acuity are far from linear, and that some variables gain or lose significance due to the absence or presence of other variables.[14] Take, for example, 3 variables which have been repeatedly found to be independent predictors of postoperative mortality: age >70 years, cirrhosis, and use of steroids. In existing, linear, and predictive models, each of these variables is treated as "present" or "absent," and often assigned the same weight irrespective of the presence or absence of the other 2 risk factors. However, it is theoretically possible that, for patients >70 years, cirrhosis plays a role but the use of steroids does not; whereas in patients <70 years, cirrhosis does not play a role but use of steroids does. Therefore, in a nonlinear risk model, the age of the patient would determine whether cirrhosis or steroid use would be included in the prediction of outcomes. The inclusion of 1 of these 2 would then determine the next variable to be included, and this variable could be different for each of the 2 choices. For example, if cirrhosis was chosen, then temperature >100.4 could be the next variable added; if steroid use was chosen, then heart failure could be added. Therefore, in a linear model the surgical risk of these 2 ES patients would be established based on the presence or absence of the same set of variables. In a nonlinear model, the risk could be determined by 2 very different sets of variables. The latter arguably better represents the complexity, interactivity, and nonlinearity of real life.

In this paper, we sought to combine big data from a well-validated, national, surgical database with artificial intelligence (AI) to design and test a novel, interactive, and nonlinear risk calculator for ES. These machine-learning methods, namely, Optimal Classification Trees (OCT) and Optimal Imputation, promise a higher degree of accuracy, interpretability, and automatic integration into electronic health records (EHRs). If translated to user-friendly applications, they may be of real-time assistance to surgeons by the bedside.

Comments

3090D553-9492-4563-8681-AD288FA52ACE

processing....