Development and Evaluation of a Novel Instrument to Measure Severity of Intraoperative Events Using Video Data

James J. Jung, MD, PhD; Peter Jüni, MD; Denise W. Gee, MD; Yulia Zak, MD; Joslin Cheverie, MD; Jin S. Yoo, MD; John M. Morton, MD; Teodor Grantcharov, MD, PhD


Annals of Surgery. 2020;272(2):220-226. 

In This Article

Abstract and Introduction


Objective: To develop and evaluate a novel instrument to measure SEVERE processes using video data.

Background: Surgical video data can serve an important role in understanding the relationship between intraoperative events and postoperative outcomes. However, a standard tool to measure severity of intraoperative events is not yet available.

Methods: Items to be included in the instrument were identified through literature and video reviews. A committee of experts guided item reduction, including pilot tests and revisions, and determined weighted scores. Content validity was evaluated using a validated sensibility questionnaire. Inter-rater reliability was assessed by calculating intraclass correlation coefficient. Construct validity was evaluated on a sample of 120 patients who underwent laparoscopic Roux-en-Y gastric bypass procedure, in which comprehensive video data was obtained.

Results: SEVERE index measures severity of 5 event types using ordinal scales. Each intraoperative event is given a weighted score out of 10. Inter-rater reliability was excellent [0.87 (95%-confidence interval, 0.77–0.92)]. In a sample of consecutive 120 patients undergoing gastric bypass procedures, a median of 12 events [interquartile range (IQR) 9–18] occurred per patient and bleeding was the most frequent type (median 10, IQR 7–14). The median SEVERE score per case was 11.3 (IQR 8.3–16.9). In risk-adjusted multivariable regression models, history of previous abdominal surgery (P = 0.02) and body mass index (P = 0.005) were associated with SEVERE scores, demonstrating construct validity evidence.

Conclusion: The SEVERE index may prove to be a useful instrument in identifying patients with high risk of developing postoperative complications.


One of the key objectives of surgical research is to identify predictors of postoperative clinical outcomes. Understanding surgical risks is important for both patients and clinicians in their shared decision-making processes related to the types of operations to undergo and whether or not surgery should be performed at all.[1] So far, much of the research has been devoted to identifying patient-level factors associated with outcomes.[2–4] For instance, the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator uses patient factors, such as age, sex, functional status, body mass index (BMI), and comorbidity profile to predict postoperative complication risks.[5] There is emerging evidence that intraoperative factors, such as technical performance[6,7] and adverse events[8] may predict postoperative outcomes. It is intuitive to hypothesize that unintended tissue injuries caused by surgical teams during procedures will play an important role in determining postoperative outcomes. However, empiric evidence on the patterns and nature of intraoperative events has largely been limited.[9]

Innovative technology to synchronously record various intraoperative data streams, such as the OR Black Box (Surgical Safety Technologies Inc., Toronto, Canada) allows structured observation and reporting of intraoperative events.[10] These events may pose a wide range of risk of harm to patients. For instance, a self-resolved minor bleed will likely pose a much lower risk of harm than a laceration of a major vessel that results in massive hemorrhage. The variation in risk of harm to patients may be interpreted as severity of intraoperative events. Further, when intraoperative events occur, surgeons may provide corrective measures to rectify the events during the same procedure, thereby mitigating potential harm. Conversely, some intraoperative events may be inadequately dealt with or altogether missed by surgeons. Thus, severity of intraoperative events is closely intertwined with the quality of rectification processes. Currently, there is a lack of a standard instrument to measure severity of intraoperative events. This type of instrument may allow a deeper understanding of patterns, nature, and impact of intraoperative events on patient outcomes. The objectives of this study are to develop a novel instrument to measure severity of intraoperative events and quality of rectification processes and to evaluate reliability and validity of the measurement tool.