A group of methodology experts have turned the tables on four organizations that rate hospitals by grading their work. Three only achieved grades of C and D. The highest grade received was a B.
The analysis, "Rating the Raters," was published August 14 in the New England Journal of Medicine (NEJM) Catalyst . It looked at rating systems from Healthgrades, Leapfrog, US News & World Report, and the Centers for Medicare and Medicaid Services' (CMS) Star Ratings.
"Current hospital quality rating systems often offer conflicting results — a hospital might rate best on one rating system and worst on another," lead author Karl Bilimoria, MD, director of the Northwestern Medicine Surgical Outcomes and Quality Improvement Center in Chicago, Illinois, said in a news release.
"We found that many of the current hospital quality rating systems should be used cautiously as they likely often misclassify hospital performance and mislead the public," Bilimoria said.
In response, Healthgrades objected to the analysis as "inaccurate" and "flawed," while Leapfrog characterized it as an "opinion piece."
"The authors are entitled to their own opinions and it is valuable to hear their perspectives," said Leah Binder, president and CEO of The Leapfrog Group. "However, they are not entitled to their own facts."
US News & World Report and CMS were more appreciative of the analysis.
No Gold Standard
Given the lack of a gold standard for how a rating system should be constructed and no objective way to compare the rating systems, Bilimoria and colleagues evaluated the strengths and weaknesses of four major public hospital quality rating systems based on their experience as physician scientists with methodological expertise in healthcare quality measurement.
No rating system received a grade of A or F. The highest grade was a B, achieved by US News & World Report. CMS's Star Ratings received a grade of C. The lowest grades were for Leapfrog (C-) and Healthgrades (D+).
The investigators identified several limitations of each individual rating system and the field as a whole.
"Each rating system had unique weaknesses that led to potential misclassification of hospital performance, ranging from inclusion of flawed measures, use of proprietary data that are not validated, and methodological decisions," the authors write.
"Opportunities to advance the field of hospital quality measurement include the need for better data subject to robust audits, more meaningful measures, and development of standards and robust peer review to evaluate rating system methodology," Bilimoria and colleagues say.
Raters Respond
Mallorie Hatch, PhD, director of data science at Healthgrades, told Medscape Medical News that the analysis paints a "highly inaccurate portrayal of Healthgrades' hospital ratings."
"The opinions provided regarding Healthgrades," Hatch said, "are flawed as there are numerous inaccuracies."
First, she said, "our methodologies are fully transparent and publicly available." Second, "the authors only assessed our overall hospital award, misrepresented that methodology, and conveniently did not include an analysis of our other service line ratings and awards, which would have addressed many of the criticisms in the article."
And third, "while Healthgrades was invited to explain our methodologies and we corrected the above inaccuracies, our feedback was not incorporated."
Binder, of The Leapfrog Group, also takes issue with the analysis.
The article is an "opinion piece. It is not designed to offer the evidence and replicability that a traditional study would offer, nor do the authors detail what standards they applied to reach their conclusions about the four ratings programs," Binder said in a statement sent to Medscape Medical News.
"That said, the ratings organizations in the piece would not allow themselves the luxury of issuing hospital ratings as random opinions, without basic rigor and transparency," she added.
"Rudimentary fact-checking would have uncovered serious errors in the description of Leapfrog's ratings programs in the piece."
For example, Binder said, "the assertion that Leapfrog audits only a handful of hospitals reporting to the Leapfrog Hospital Survey is demonstrably false."
In addition to "basic fact-checking," Binder said the study would have "greater credibility if the majority of authors were not employed at health systems with a history of feuding with one or more of the ratings organizations they analyze."
Ben Harder, managing editor and chief of health analysis at US News & World Report, whose company received the "B" rating, told Medscape Medical News that the study takes "an important look at hospital ratings [and] we're gratified that the study recognized how responsive we have been to advances in measurement science and feedback from patients, doctors, and other stakeholders.
"The methodology changes we made this year reflect our commitment to ongoing enhancement of our rankings. The researchers also tipped their hats to our decision to make statistical adjustments for socioeconomic status and other factors to ensure fair comparisons among hospitals," Harder said.
A CMS spokesperson told Medscape Medical News, "The agency is confident the Overall Hospital Quality Star Ratings drive systematic improvements in care and safety and are critical in getting hospitals to compete on the basis of quality." The agency also works with patients, hospitals, and the healthcare community to improve its ratings, she said.
"After receiving feedback from hospitals and other stakeholders through a series of listening sessions and input from a technical expert panel, CMS developed potential changes to the Star Ratings methodology, which were released for public comment this past spring," she notes.
The agency is appreciative of the feedback the agency has received so far "from a variety of stakeholders on the Star Ratings methodology, including the work of the researchers in the study you reference, and looks forward to sharing improvements to the Star Ratings in the future," the CMS spokesperson said.
Bilimoria reports that he has been an unpaid advisor to US News & World Report and CMS' Star Ratings.
NEJM Catalyst. Published online August 14, 2019.
For more news, follow Medscape on Facebook, Twitter, Instagram, and YouTube.
Medscape Medical News © 2019
Cite this: Rating the Raters: Hospital Rating Systems Get Low Grades - Medscape - Aug 23, 2019.
Comments