Does the ACS NSQIP Surgical Risk Calculator Accurately Predict Complications Rates After Anterior Lumbar Interbody Fusion Procedures?

Ankur S. Narain, MD; Alexander Z. Kitto, MD; Benjamin Braun, MD; Matthew J. Poorman, MD; Patrick Curtin, MD; Justin Slavin, MD; Giles Whalen, MD; Christian P. DiPaola, MD; Patrick J. Connolly, MD; Michael P. Stauff, MD


Spine. 2021;46(12):E655-E662. 

In This Article


Health care value, defined as quality of care per unit cost, has become an important measure of physician and hospital system performance.[1] The value of surgical care is directly linked to the rates of complications associated with surgical procedures, hence making the ability to predict complications very powerful in surgical care. The ACS NSQIP surgical risk calculator, derived from the larger ACS NSQIP database, was devised as a risk-stratification tool to allow for delivery of greater healthcare value.[2] Within the realm of spine surgery, ALIF procedures are commonly performed and have multiple indications. The purpose of this study was to determine the ability of the ACS NSQIP surgical risk calculator to predict complication rates after ALIF procedures performed at our institution.

Our data confirm that the ACS NSQIP surgical risk calculator did not predict the incidence of total complications that occurred based on our retrospective chart review of patients who underwent ALIF procedures. The overall observed complication rate was 22.9% (n = 58) compared to a predicted complication rate of 6.7% (n = 17). The ACS NSQIP surgical risk calculator performed best in predicting acute renal failure/progressive renal insufficiency within the 30-day postoperative period. The risk calculator had fair predictive ability for adverse discharge and surgical site infections. All other complication types were poorly predicted using the ACS NSQIP surgical risk calculator.

The reliability of the ACS NSQIP surgical risk calculator has been evaluated across multiple surgical subspecialties, with results that are concordant with the present study.[7–10] In the urology literature, Winoker et al[7] determined that the ACS NSQIP surgical risk calculator was a poor predictor of complication incidence in 300 patients undergoing robot-assisted partial nephrectomy. Wang et al, in a subset of 242 geriatric patients undergoing lumbar spine surgery, demonstrated that the ACS NSQIP surgical risk calculator was a poor predictor for "all complications" and "serious complications."[11] The deficiency of the ACS NSQIP surgical risk calculator in predicting the incidence of complications is likely associated with many factors. Specifically, there were methodological limitations in developing the calculator that limits its applicability to subspecialty-specific procedures.[2] These limitations are primarily associated with the sample used to design the calculator, which was heterogeneous in both procedure and specialty types.[2] Additionally, institution and practice-associated factors such as case complexity may influence complication rates in ways that the calculator does not capture.

Adverse discharge, or discharge to non-home facility, had a higher occurrence in our cohort than was predicted by the ACS NSQIP surgical risk calculator. Variation between the incidence and prediction of adverse discharge has also been demonstrated in other studies. Edelstein et al,[9] in a large cohort of patients undergoing hip and knee arthroplasty, found that the ability of the risk calculator to predict discharge destination was only fair (c = 0.72). The disparity between the observed and predicted rates of adverse discharge may stem from the fact that postoperative discharge decisions are complex and often involve patient socioeconomic factors. These factors, such as insurance type and marital status,[12] are not used as inputs within the surgical risk calculator. The effects of these important socioeconomic factors are thus omitted and this may be associated with the under-prediction of adverse discharge that is evident within the surgical risk calculator.

Overall, the ACS NSQIP surgical risk calculator underestimated the incidence of complications compared to our observed institutional occurrences. Specifically, the 22.9% incidence of any complication in our cohort was much greater than the 6.7% predicted incidence from the risk calculator. Consistent under-prediction of complication rates by the risk calculator has been demonstrated in other studies,[7,13] including the previously mentioned investigation by Winoker et al.[7] Prasad et al[13] further demonstrated that the risk calculator underestimated risks of surgical site infection by 28.6% and reoperation rates by 44.3% in patients undergoing major head and neck surgery.

Our observed complication rate (22.9%) is slightly lower than other reported complication rates, which range from 24% to 38%.[14–17] It is unclear whether higher complication rates would allow the risk calculator to better predict complications. We believe that our cohort is generalizable given the large number of subjects, multiple different approach surgeons (general surgery and vascular surgery) and multiple different spine surgeons (orthopedic spine and neurosurgery spine).

When analyzing the variables that are used as part of the ACS NSQIP surgical risk calculator, it is clear that there are significant differences when compared to the risk factors associated with complications after ALIF procedures as noted in recent studies.[18–21] Some of these identified risk factors are similar to ACS NSQIP surgical risk calculator variables, including: age, history of COPD, diabetes, corticosteroid use, and ASA classification.[18,19,21] However, there are important risk factors for complications after ALIF procedures that are not part of the ACS NSQIP surgical risk calculator, including: preoperative nutritional status, frailty index, preoperative anemia, preoperative transfusion requirement, and preoperative weight loss.[18,20] Reasons for this variability likely are associated with the difficulties inherent in attempting to produce a "universal" surgical risk calculator, as was the purpose in the design of the ACS NSQIP surgical risk calculator.[2] Specifically, there will clearly be variability when a tool that was designed for broad use within hundreds of different types of procedures is analyzed for a single procedure. Additionally, the requirement to keep these tools relatively brief may necessitate the removal of some predictors that are too cumbersome or difficult for the majority of practitioners to utilize.

This study has limitations. First, this study was a retrospective review from a single institution. Despite the high number of subjects and multiple different surgeon providers, the analysis is still subject to confounding. Second, our cohort consisted of both stand-alone ALIFs and ALIFs as part of staged anterior–posterior procedures. Although this did add to the heterogeneity in the data set, we believe our cohort is more representative of actual usage patterns of ALIF procedures within the scope of surgical practice. Third, the presence of a large number of staged procedures complicated length of stay analysis. To mitigate this, we performed length of stay analysis on stand-alone ALIFs only, and determined that the ACS NSQIP surgical risk calculator was a poor predictor of postoperative length of stay. Fourth, only complications up to 30 days postoperatively were analyzed. Although this is in accordance with the timeline used by the ACS NSQIP surgical risk calculator and the ACS NSQIP administrative database, it does not account for those complications that occurred after 30 days postoperatively. Lastly, this study only utilized data from a single institution rather than data from the entirety of the NSQIP database. With the larger sample size associated with the use of the database sample, the discrepancy between calculator predictions and the observed incidence of complications may be minimized. However, we believe that using an institutional sample allowed for a more thorough determination of operative characteristics and a more accurate determination of clinical complication incidence.