Physicians Launch Call for BMJ to Retract Medical Error Study

Marcia Frellick

July 13, 2016

In a recent blog post, two physicians say a study recently published in BMJ that concluded medical error is the third leading cause of death in the United States is "shoddy" science.

The paper was widely reported May 3 in print, broadcast, and online media, including by Medscape Medical News.

Medical blogger Shyam Sabat, MD, an associate professor of neuroradiology at Penn State College of Medicine in Hershey, Pennsylvania, and Virginia Hall, MD, associate professor of obstetrics and gynecology at the college, say the article has unfairly condemned the US healthcare system and call for BMJ to retract the article and issue an apology to the US medical community. They have started a campaign on the online platform to support retraction. By July 11, the campaign had 176 supporters.

They write in their post: "[T]he paper is a shoddy piece of scientific and statistical work which cannot stand the close scrutiny of peer physician researchers and professional statisticians."

The journal, however, told Medscape Medical News a retraction is not warranted.

One Study Dominates Other Three

Dr Sabat and Dr Hall write, "We reviewed the paper with expert statistician Dr. Vernon Chinchilli (Prof. and Chair Department of Public Health Sciences, Penn State College of Medicine), who found it shoddy science that the paper which calls itself a meta-analysis of four studies, is actually just a borrowed summary of a single study (by Healthgrades published in 2004). The other three studies just have 795, 838 and 2341 patients respectively versus 37 million in the Healthgrades study."

The bloggers say the three other studies included in the analysis do not have the statistical power to be lumped in with the first study, and the conclusions actually belong to the Healthgrades 2004 study, which included only Medicare patients (who are aged 65 years and older).

"Moreover, the authors borrowed the mortality rates from this Medicare population study and applied it to all U.S. inpatient admissions (all ages from 0-100+ years) without any correction of any form. It is common knowledge that the Medicare inpatient population is older, sicker and more vulnerable, and hence will have a higher morbidity and mortality for a given medical error than the general U.S. population," Dr Sabat and Dr Hall write.

BMJ Responds

Navjoyt Ladher, MBBS, BMJ analysis and scholarly comment editor and a general practitioner, told Medscape Medical News, "We do not believe there are any grounds to retract the paper, and are not considering this course of action."

"The BMJ's channel for debate and post-publication peer review of articles published in the journal is our rapid response section. We invite Dr Sabat and Dr Hall to make their points in a rapid response to the Analysis article, thus giving the authors of the article and other readers an opportunity to read and reply to their comments. We will continue to watch closely as the discussion develops."

Biostatistician Agrees No Retraction Necessary

E. John Orav, PhD, associate professor of medicine in biostatistics at Harvard Medical School and Brigham & Women's Hospital in Boston, Massachusetts, agrees there is no need for a retraction.

He told Medscape Medical News that part of the problem in the debate is the blog authors' repeated use of the term "meta-analysis," a term study authors Martin Makary, MD, from the Department of Surgery at Johns Hopkins University School of Medicine in Baltimore, Maryland, and Michael Daniel, a research fellow in the same department, did not use in the original paper.

Dr Orav noted: "A meta-analysis is a rigorous, formal method for addressing a scientific question by combining results from previously published studies, rather than by collecting new primary data. The process involves a comprehensive process for identifying and rating the published studies, sometimes giving quality scores to the studies, and then choosing from a number of possible statistical methods for combining the results of the studies.

"If the authors of the BMJ article had claimed their study to be a meta-analysis, they would indeed have done a poor job."

Instead, he said, and the authors state, it is a less formal literature review, which, Dr Orav says, "consequently promises the reader less in terms of statistical rigor."

But even excluding the large study, the others would rank medical error either third or fourth, he points out.

A true meta-analysis "would also have weighted the combined estimate to emphasize the largest study. And, if we ignore the Healthgrades study and focus on only the other three studies, we would rank medical errors as third, using the Office of the Inspector General, third using Classen et al, and fourth using Landrigan et al. I don't think that the conclusion of the study hinges on this criticism," Dr Orav said.

Authors Should Have Been Clearer

As to the point that the conclusion borrowed the summary of a single study, he agrees the authors do not explicitly point out the BMJ paper final estimate reflects the Healthgrades result almost exclusively.

"This point is important enough that it should have been emphasized in the article," he said.

He also agrees with the blogger's criticism that "the authors borrowed the mortality rates from this Medicare population study and applied it to all U.S. inpatient admissions without any correction of any form."

"This statement is true and the criticism is warranted," he said. "Since Medicare patients account for only one third of all deaths, extrapolating to the other two thirds of deaths is risky."

He added that the BMJ authors acknowledge that shortcoming with this stated limitation: "Although the assumptions made in extrapolating study data to the broader US population may limit the accuracy of our figure, the absence of national data highlights the need for systematic measurement of the problem."

Whether that warning is enough, he said, or whether it excuses a potentially poor calculation is up for debate.

The authors and Dr Orav have disclosed no relevant financial relationships.

BMJ. 2016;353:i2139. Full text

For more news, join us on Facebook and Twitter


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: