A Comparative Analysis of Sepsis Identification Methods in an Electronic Database

Alistair E. W. Johnson, DPhil; Jerome Aboab, MD, PhD; Jesse D. Raffa, PhD; Tom J. Pollard, PhD; Rodrigo O. Deliberato, MD, PhD; Leo A. Celi, MD, MPH; David J. Stone, MD


Crit Care Med. 2018;46(4):494-499. 

In This Article

Abstract and Introduction


Objectives: To evaluate the relative validity of criteria for the identification of sepsis in an ICU database.

Design: Retrospective cohort study of adult ICU admissions from 2008 to 2012.

Setting: Tertiary teaching hospital in Boston, MA.

Patients: Initial admission of all adult patients to noncardiac surgical ICUs.

Interventions: Comparison of five different algorithms for retrospectively identifying sepsis, including the Sepsis-3 criteria.

Measurements and Main Results: 11,791 of 23,620 ICU admissions (49.9%) met criteria for the study. Within this subgroup, 59.9% were suspected of infection on ICU admission, 75.2% of admissions had Sequential Organ Failure Assessment greater than or equal to 2, and 49.1% had both suspicion of infection and Sequential Organ Failure Assessment greater than or equal to 2 thereby meeting the Sepsis-3 criteria. The area under the receiver operator characteristic of Sequential Organ Failure Assessment (0.74) for hospital mortality was consistent with previous studies of the Sepsis-3 criteria. The Centers for Disease Control and Prevention, Angus, Martin, Centers for Medicare & Medicaid Services, and explicit coding methods for identifying sepsis revealed respective sepsis incidences of 31.9%, 28.6%, 14.7%, 11.0%, and 9.0%. In-hospital mortality increased with decreasing cohort size, ranging from 30.1% (explicit codes) to 14.5% (Sepsis-3 criteria). Agreement among the criteria was acceptable (Cronbach's alpha, 0.40–0.62).

Conclusions: The new organ dysfunction-based Sepsis-3 criteria have been proposed as a clinical method for identifying sepsis. These criteria identified a larger, less severely ill cohort than that identified by previously used administrative definitions. The Sepsis-3 criteria have several advantages over prior methods, including less susceptibility to coding practices changes, provision of temporal context, and possession of high construct validity. However, the Sepsis-3 criteria also present new challenges, especially when calculated retrospectively. Future studies on sepsis should recognize the differences in outcome incidence among identification methods and contextualize their findings according to the different cohorts identified.


Sepsis is a major and economically significant disease in the ICU, costing over $20 billion in the United States in 2011 (5.2% of all U.S. hospital costs),[1] with costs growing to over $23 billion in 2013 (6.2% of all U.S. hospital costs).[2] The European Society of Intensive Care Medicine/Society of Critical Care Medicine Third International Consensus Definitions for Sepsis and Septic Shock task force (the Sepsis-3 task force) recently defined sepsis as a "life-threatening organ dysfunction caused by a dysregulated host response to infection".[3] Analyzing retrospective databases, the authors proposed and evaluated new clinical criteria for detection of sepsis: an increase of greater than or equal to 2 Sequential Organ Failure Assessment (SOFA) score points in a defined temporal context of suspected infection.[3,4] These new criteria were further validated in a dataset of 184,875 adults in ICUs across Australia and New Zealand.[5] Although the utility of the Sepsis-3 criteria for clinical care is still being deliberated upon,[6] to date, little research has focused on application of the new criteria to identify septic patients in electronic health records (EHRs).

The penetration of EHRs has dramatically increased in the United States, from 9.4% in 2008 to 83.8% in 2015, a nine-fold increase.[7] Research using EHRs is becoming progressively more important and has the potential for making decision-making more precise, more robust, and more personal. Past criteria for sepsis using EHRs mainly used administratively assigned billing codes, with the research focused on the epidemiology of sepsis.[8,9] The criteria as proposed by the Sepsis-3 task force offer an attractive operational definition of sepsis in retrospective observational research because they are objectively quantifiable, incorporate an approximation for the start time of clinical concern as opposed to classifying entire hospitalizations, and are based directly on the physiologic data rather than captured indirectly via administrative codes. Our study aims to examine previously employed administrative criteria for retrospective identification of patients with sepsis and compare these with the new Sepsis-3 criteria.