Early Warning Scores Should Not Be Relied Upon, Warn UK Docs

Liam Davenport

May 21, 2020

Studies examining early warning scores (EWSs) - that aim to predict which hospital patients will deteriorate - have used poor methods, have been inadequately reported, and have missing data or details that leave them open to bias, UK scientists have discovered in a systematic review.

The result is that "clinicians can have little knowledge of how such scores will perform in their clinical setting" say the researchers, led by Stephen Gerry, MSc, Centre for Statistics in Medicine, University of Oxford.

"Therefore, clinicians should be cautious about relying on these scores to identify clinical deterioration in patients."

The research was published by The BMJ on May 20th.

Vital Signs

EWSs typically rely on vital signs such as heart rate, oxygen levels and blood pressure to determine whether hospitalised patients are likely to deteriorate, with the aim of tackling adverse events and reducing unnecessary deaths.

Such scores are "very widely used and relied upon", Stephen Gerry told Medscape News UK.

He also pointed out that the National Institute for Health and Care Excellence guideline for Acutely ill adults in hospital recommends their use on admission and then every 12 hours.

Consequently, they "are probably used more than any other clinical prediction model", and there is "increasing pressure" to use them in "primary care, ambulances and, potentially, care homes".

Study Details

To investigate further, the researchers searched the Medline, CINAHL, PsycInfo and Embase databases for studies of EWSs developed for adult hospital inpatients and published up to June 2019.

They included 95 studies, of which 11 were focused solely on the development of an EWS, 23 included both development and external validation of an EWS, and 61 were solely external validations.

Of 34 unique EWSs, the majority were developed for use in either the UK (29%) or USA (38%).

The most common predictors were respiratory rate (88%), heart rate (83%), and oxygen saturation, temperature and systolic blood pressure, each used in 71% of EWSs.

In contrast, age and sex were less commonly included, in 38% and 9% of scores, respectively.

The most common prediction outcomes were death, in 44% of development studies and 79% of validation studies. While there were a wide variety of predicted time horizons for outcomes, 24 hours was the most commonly used, in 35%.

Small Samples

Turning to the quality of the studies, the team found that key details of the analysis populations were often not reported in either developmental or validation studies, and many had small samples and too few events in both model development and external validation.

They determined that only nine of the EWSs were presented in enough detail to allow individual risk prediction and, while internal validation was commonly performed, recommended techniques such as bootstrapping and cross validation were rarely used.

Model performance was based on assessment discrimination, or which patients will or will not develop the outcome of interest, in 82% of studies, but only 15% used calibration to determine the agreement between predicted risks and observed event rates.

Finally, the team examined the risk of bias using the PROBAST prediction model risk of bias assessment tool, which covers participant selection, predictors, outcome, and analysis.

This showed that there was a high risk of bias for participant selection in 55% of studies, for predictors in 5%, for outcomes in 66%, and for analysis methods in 98%.

Inadequate Methods

They write that scores "developed using inadequate methods will probably result in poorly performing scoring", while "poor methods in external validation studies could lead to implementation of inferior scoring systems, with false reassurances about their predictive ability and generalisability".

This, they say, "could explain why recent systematic reviews have found little evidence of any clinical effectiveness of EWSs".

The researchers add that, with electronic health records increasingly being used to record vital signs and EWSs and so the development of more sophisticated scores, it is "important" that future research "is of the highest quality".

They write: "The move towards electronic implementation of EWSs presents an opportunity to introduce better scoring systems, particularly with the increasing interest in modern model building approaches, such as machine learning and artificial intelligence."

"However, if methodological and reporting standards are not improved, this potential might never be achieved."

The study was funded by the National Institute for Health Research (NIHR) and Cancer Research UK.

Peter J. Watkinson is chief medical officer for Sensyne Health and holds shares in the company; Timothy Bonnici receives royalties from Sensyne Health. No other potential conflicts of interest declared.

BMJ 2020;369:m1501 doi: 10.1136/bmj.m1501


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: