Validation of a Crisis Standards of Care Model for Prioritization of Limited Resources During the Coronavirus Disease 2019 Crisis in an Urban, Safety-Net, Academic Medical Center

Albert Nadjarian, MD, MPH; Jessica LeClair, BS; Taylor F. Mahoney, MA; Eric H. Awtry, MD, FACC; Jasvinder S. Bhatia, MD; Lisa B. Caruso, MD, MPH; Alexis Clay, MD; David Greer, MD, MA; Karan S. Hingorani, MD, PhD; L. F. B. Horta, MD; Michel Ibrahim, MD; Michael H. Ieong, MD; Thea James, MD; Matthew H. Kulke, MD; Remington Lim, BA; Robert C. Lowe, MD; James M. Moses, MD; Jaime Murphy, MD; Ala Nozari, MD; Anuj D. Patel, MD; Brent Silver, MD; Arthur C. Theodore, MD; Ryan Shufei Wang, MD; Ellen Weinstein, JD; Stephen A. Wilson, MD, MPH, FAAFP; Anna M. Cervantes-Arslanian, MD


Crit Care Med. 2021;49(10):1739-1748. 

In This Article


To our knowledge, this is the first study of its kind to evaluate the applied CSC framework that considers underlying conditions. Doing so specifically within a hospitalized critically ill patient population with racial and socioeconomic diversity may provide additional support for the refinement of evidence-based resource allocation frameworks, especially during a pandemic that has disproportionately affected racial minorities.

In our population, we found that patients with +2 (major) and +4 (severe) medical conditions as defined by the BMC's scoring criteria in aggregate had 46.55% and 50.00% mortality at 1 and 5 years, respectively. However, mortality varied between conditions, with ESRD, metastatic cancer, CHF, and neurodegenerative disease criteria more consistently predicting greater than 50% mortality at 1 or 5 years than CLD or cirrhosis. Our scoring criteria for cirrhosis and CLD did not consistently identify patients with observed mortality greater than 50% at 5 years, emphasizing the importance of only including underlying conditions reliably associated with greater than 50% mortality at 1 year in CSC triage algorithms.

Only a minority of patients (15.89%) within our population scored for +2 (major) or +4 (severe) underlying conditions. For this reason, we opted to calculate mortalities both for +4 and +2 conditions in aggregate as well as separated out (with 1- and 5-yr mortalities for +4 and +2 scored conditions, respectively). The same scoring criteria were applied for data collection purposes by our hospital nurse administrators at the height of the COVID-19 pandemic (April to May 2020). At the time, 3,042 patients were reviewed, and only 338 (11.11%) scored for major or severe conditions. There are notable differences between the historical and COVID-era 2020 cohorts, as the 2020 cohort was almost exclusively hospitalized for COVID-19, whereas the historical cohort had variable illnesses. Nevertheless, the small percentage of scored patients likely reflects what would be encountered in prospective CSC scenarios and underscores the need to heavily consider acute survival metrics such as the SOFA score (which have demonstrated good association with short-term survival[8]) if crisis standards were to be implemented. It is worth noting that SOFA score criteria may need to be adjusted to reflect the context of critical illness in the COVID-19 era.[9]

We noted significant variability in the complexity of scoring criteria. For example, ESRD required only two variables to fulfill the scoring benchmark, but CLD, cirrhosis, and neurodegenerative criteria were more complex, labor intensive, and required subspecialist physician review. Future iterations of scoring criteria should seek to simplify data without significantly affecting their positive predictive value. Criteria simplification can allow scoring systems to be more easily and consistently applied across reviewers, whereas nuances in prognostication can be leveraged by specialists within CSC committees.

We suspect stringent adherence to CSC triage scoring will underreport the number of patients with chronic underlying conditions. In our study, several patients had documented histories of a chronic disease without supporting data, and therefore, we did not score them for the condition. This was often due to scarce outpatient records prior to admission. In addition, lack of specific information such as transplant eligibility, description of baseline cognitive function, outpatient laboratory values, or imaging criteria made it challenging to accurately score patients, especially for the pulmonary, cirrhosis, and neurodegenerative categories. We decided to keep include these patients in the unscored group rather than eliminating them from data analysis. Although this limited the reliability of ascertaining mortality among the specific conditions for our study, it reflects the real-life scenario that hospitals will likely face when using similar triage criteria prospectively. It also raises the possibility that patients who have received medical care and have more complete medical records with thorough documentation may be penalized by being scored for a life-limiting chronic medical condition, whereas others with the same conditions but no prior contact with our health system are not. This highlights the importance of electronic medical record intercompatibility for a more comprehensive understanding of admitted patients and their medical histories.

Most patients with a chronic lung condition had chronic obstructive pulmonary disease (COPD), with a small minority of patients suffering from other conditions such as interstitial lung disease, pulmonary arterial hypertension, or chronic bronchiectasis. Of those with COPD who did score, the vast majority did so on the basis of multiple hospitalizations. This criterion did not correlate well with mortality. Future studies can perhaps incorporate multiple hospitalizations as a supplemental rather than standalone criterion to help increase its predictive value. The pulmonary criteria that most strongly correlated with mortality included World Health Organization Class IV symptoms, hospice eligibility, and baseline blood gases suggestive of decompensation.

Chart reviewers often had limited available outpatient data to score patients with cirrhosis. For those patients, laboratory values were taken from their time of ICU admission, which may have overestimated the true severity of their underlying disease and consequently why greater than 50% mortality was not seen at 5 years. Increasing the MELD score requirement of the +4 scored population can potentially offset this limitation and strengthen the positive predictive value of the cirrhosis criteria. Furthermore, the committee made a conscious effort to include hepatic transplant ineligibility as a required scoring criterion. The rationale was that patients who are transplant eligible should not be adversely affected with a score, since if they were to survive, they could eventually receive life-prolonging treatment. However, only seven of the 31 patients with documented severe cirrhosis were also deemed transplant ineligible. Although lack of transplant eligibility evaluation may limit the application of this criterion, future scoring criteria might consider a "rapid evaluation" upon admission to identify major transplant exclusion criteria and guide decision-making for appropriate crisis scoring classification.

The neurodegenerative criteria used in our model met the greater than 50% predicted mortality benchmark at 1 and 5 years, but we found a small number of patients who met these criteria. The criteria were quite complex, requiring prior cognitive evaluations and key phrases documented in the chart (such as "bed-bound," "multiple stage 3 or 4 ulcers," or "insufficient oral intake" with documented > 10% weight loss in 6 months or albumin < 2.5 g/dL which are commonly used as hospice criteria). It is essential to have objective criteria for predicting mortality in neurodegenerative conditions as many patients have a diagnosis of dementia, but this is not a uniformly life-limiting condition. Furthermore, subjective descriptors of patients may unintentionally reflect values about quality of life which should not be considered in CSC scoring criteria.

Although the cardiac criteria were relatively straightforward, subjective data such as "frailty" could not be consistently determined from chart reviews and should be removed in favor of more objective data. Similarly, the NYHC Class IV heart failure patients could score either +2 (major) or +4 (severe) depending upon the quality of documentation in our records or our ability to access outside records to determine recent hospitalization, tolerance of medications, and implantable cardioverter defibrillator shocks. This may explain the discrepancy seen between 1- and 5-year mortality being greater than 50%, but discordance between expected mortality in the subdivided +2 and +4 groups.

Our study had several limitations. Most notably, the small number of scored patients limited the ability to assess the predictive accuracy for any one condition. Assessment of mortality was another major limitation in this study due to the need to make assumptions regarding patient status as alive or deceased. Over a third of patients (130/365) were lost to follow-up and considered alive which, given their significant comorbidities, may have overestimated survival. To a lesser degree, we may have overestimated mortality for 53.33% of patients (8/15) discharged to hospice without confirmation of death and were assumed to be deceased. To add, patients with scored conditions were significantly older (p < 0.01), meaning that age could be a driver of mortality. Although it would have been ideal to adjust for potential confounding by age, the sample size of scored patients was too small. This could be better studied in a larger population. Finally, the cause of death was quite variable in 2015 and 2019 and did not reflect the almost uniform COVID-19 diagnosis that was encountered in April 2020 during the height of the pandemic, which affects the generalizability of our results. It is clear from emerging data that other risk factors outside of our current established comorbidities such as diabetes, obesity, and smoking are likely to affect both COVID-19 illness severity and survival.[10–12] Future studies may benefit from incorporating these data into COVID-19 specific triage scoring criteria.

Overall, we believe that the experience of establishing CSC triage guidelines at our institution has taught us valuable lessons which may be generalizable to other healthcare systems facing the potential need to allocate limited resources. Notably, the majority of our population was comprised of racial minorities (Black and Hispanic). It is crucial to note that people of color are historically and presently disadvantaged by structural racism as well as unequal access to affordable healthcare, stable housing, education, and employment.[13] No scoring criteria which includes chronic underlying medical conditions can rectify the fact that lifelong disparities in healthcare will bias a model toward penalizing minority patients.[3] Given our marginalized patient population, special attention is needed to ensure that factors such as race, psychosocial issues, and socioeconomic status do not bias the risk calculation or negatively affect equitable resource allocation. These considerations have been reflected in the revised CSC issued by Massachusetts in October 2020[14] (Supplemental Table 2, For example, the new guidelines safeguard against inadvertent race-based discrimination in SOFA scoring by limiting the number of points that can be added for elevated creatinine if a patient has chronic renal insufficiency at baseline, due to the disproportionately high prevalence of chronic kidney disease among Black Americans. Thus, we would also favor removing ESRD from the chronic conditions criteria despite its high predictive value for mortality in our study.

Our study supports the recent changes to the Massachusetts criteria[14] to eliminate the inclusion of major underlying conditions and use only acute illness severity and severe underlying conditions associated with greater than 50% mortality at 1 year in a scoring algorithm. The low number of patients scoring for chronic conditions appears to support counting severe underlying conditions as +2 rather than +4 to prioritize acute illness scoring. We believe this adjustment will more equitably score patients in our vulnerable populations as well as patients who have received the majority of their medical care within our network. Further research is needed to develop improved iterations of this CSC resource allocation triage model.