Little Agreement Among Four Hospital Rating Systems

Susan London

March 17, 2015

There is scant agreement among the four most prominent national systems used to rate the quality and safety of care provided by US hospitals, concludes a comparative study published in the March issue of Health Affairs.

"The differences across rating systems may reflect true differences in quality and safety across different indicators. However, the divergent ratings of hospitals may also reflect poorly defined concepts as well as idiosyncratic issues of measure selection or data quality (for example, measurement error, missing data)," write the investigators, led by J. Matthew Austin, PhD, an assistant professor at the Armstrong Institute for Patient Safety and Quality, Johns Hopkins Medicine, Baltimore, Maryland.

"It is possible that greater agreement might exist between other national ratings or state- or region-level ratings. However, we chose these national ratings because they are both most visible and readily available," they note.

The investigators determined ratings of hospitals during 2012 to 2013 on each of the four systems: U.S. News & World Report's Best Hospitals (1928 hospitals eligible for rating), Leapfrog's Hospital Safety Score (2514 hospitals eligible for rating), Consumer Reports' Health Safety Score (2040 hospitals eligible for rating), and HealthGrades' America's 100 Best Hospitals (262 hospitals eligible for rating).

Results showed that the large majority of hospitals, roughly 60% to 89%, depending on the system, fell into medium-performance categories for the three rating systems having such categories.

Among the 83 hospitals rated by all four systems, none was rated as a high performer by all of them. Merely 10% of the 844 hospitals rated as a high performer by one rating system were rated as a high performer by any other system.

Similarly, none of the hospitals was rated as a low performer on all three systems capturing such performance.

Among the hospitals rated as high performers on more than one rating system, agreement was greatest for Leapfrog and HealthGrades, at 55%. Among the hospitals rated as low performers on more than one rating system, agreement was greatest for Leapfrog and Consumer Reports, at 47%.

Some cases of extreme disagreement were identified. For example, 14 hospitals were high performers according to the Leapfrog system, but low performers according to the U.S. News system, and seven hospitals were high performers according to the Leapfrog system but low performers according to the Consumer Reports system.

Final analyses showed that for each of the three graduated rating systems, hospitals with certain characteristics were disproportionately represented in the low- and high-performance categories. However, the specific characteristics varied across systems.

"Although the variety and differences among hospital rating systems may be beneficial to patients, providers, and purchasers, these stakeholders would benefit if rating organizations could agree on standards for transparently reporting key features of their ratings. Full transparency in how ratings are constructed is especially important to patients, hospitals, and researchers," the investigators maintain.

"Future research should examine how the preferences of different health care stakeholder groups can be used to validate, refine, or weight the measures comprising the ratings. It would also be useful for future research to explore how hospitals and their administrators make use of the ratings," they recommend.

Dr Austin received funding support from the Leapfrog Group. Five coauthors all voluntarily serve on the Leapfrog Group's Hospital Safety Score Expert Panel and receive no compensation for their service. One coauthor serves as the immediate past chair of the American Board of Internal Medicine and a current member of the American Board of Internal Medicine Foundation board, he received a stipend and stock options for serving on the board of directors of IPC-The Hospitalist Company, and he serves on the scientific advisory boards for Amino.com, PatientSafe Solutions, CRISI, QPID, and EarlySense (for which he receives stock options).

Health Affairs. 2015;34:423-430. Abstract

Comments

3090D553-9492-4563-8681-AD288FA52ACE
Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as:

processing....