Scientists Question Data, Ethics, Findings of Lancet HCQ Study

Marcus A. Banks

June 01, 2020

Editor's note: Find the latest COVID-19 news and guidance in Medscape's  Coronavirus Resource Center.

More than 140 scientists and physicians are challenging the validity of an influential study that found an association between prescribing the antimalarial drugs hydroxychloroquine and chloroquine for COVID-19 patients and increased in-hospital mortality. After the observational study's results were published in The Lancet on May 22, the World Health Organization (WHO) temporarily suspended enrollment into an ongoing randomized clinical trial testing hydroxychloroquine for COVID-19.

An open letter to the observational study's authors and The Lancet's editor-in-chief posted May 28 lists 10 concerns. The signatories, who identify themselves as "clinicians, medical researchers, statisticians, and ethicists from across the world," say the researchers failed to account sufficiently for factors that may have influenced their results, including disease severity, and raise concerns about a lack of ethics review and errors in the underlying database.

They also charge that the study's authors are being unduly secretive about their data sources and methods, despite the fact that The Lancet signed a pledge to support data sharing during the coronavirus pandemic.

"This paper has had a really negative impact on clinical trials," said James Watson, DPhil, a statistician at the Mahidol-Oxford Tropical Medicine Research Unit in Thailand and the lead signatory on the open letter. "A lot of decisions [about hydroxychloroquine] have been made on the basis of very poor evidence. This drug could be harmful, it could be beneficial, it could do absolutely nothing at all, but we need a randomized trial," Watson said.

The Lancet study is based on data from the medical services company Surgisphere about 96,032 hospital patients diagnosed with COVID-19 from December 20 to April 14 from every continent except Antarctica. Every patient was discharged by April 21, unless they had died by then. The majority of patients, 81,144, did not receive antimalarial drugs. The remaining 14,888 patients began to receive the antimalarial drugs chloroquine or hydroxychloroquine within 48 hours of their positive diagnosis, either alone or with an antibiotic.

After controlling for numerous factors including age, race, sex, and comorbidities such as cardiovascular and lung disease, the authors found that patients on antimalarials were twice as likely to die in the hospital as patients who did not receive them (18% mortality for patients who received hydroxychloroquine, 16.4% for chloroquine, 9.3% for those who did not receive an antimalarial). Patients who also received antibiotics experienced even higher mortality rates.

"We were fairly reassured that, although the study was observational, the signals were robust and consistent across all regions of the world in diverse populations, and we did not see any muting of that signal, depending on region," lead author Mandeep R. Mehra, MD, MSc, the William Harvey Distinguished Chair in Advanced Cardiovascular Medicine at Brigham and Women's Hospital in Boston, Massachusetts, previously told | Medscape Cardiology about the results.

Severity-of-Disease Stratification Needed

Watson's central critique is that The Lancet authors did not properly stratify patients by severity of disease. The parameters the study authors used to determine disease severity, the level of blood oxygen saturation and rates of suspected infection, did not identify significant differences between people who received antimalarials and those who did not. Watson feels that another parameter, the ratio of arterial oxygen pressure to fractional inspired oxygen (the PaO2/FiO2 ratio), would have revealed differences between the two groups.

Antimalarials are generally provided to the sickest COVID-19 patients under the rubric of "compassionate use," Watson says, after other options have failed. The difference in mortality, he suggests, could appear because patients who received hydroxychloroquine or chloroquine were sicker than others to begin with.

The study authors say that they adjusted for 35 potential confounders and stand by their work while recognizing its limitations. "The authors leveraged the data through Surgisphere to provide observational guidance...the authors have underscored the importance and value of randomized clinical trials and articulated that such trials will be necessary before any conclusions can be reached," Mehra said in a statement. The study's second author, Sapan S. Desai, MD, is CEO of Surgisphere.

Previous critiques posted online to the scientific forum PubPeer noted discrepancies between the official number of deaths reported in Australia and the observational study's count. This past weekend, The Lancet published corrections to the paper, changing one hospital's location from Australasia to Asia and including some raw data that was omitted in the initial publication. The correction notice states: "There have been no changes to the findings of the paper."

Susan Ellenberg, PhD, a biostatistician at the University of Pennsylvania's Perelman School of Medicine in Philadelphia, said she sees no reason to halt clinical trials of the effectiveness of antimalarials for COVID-19 treatment on the basis of The Lancet study. "When you're looking for treatment effects, whether it's effectiveness or safety, your best answer is going to come from randomized trials," Ellenberg said.

Ellenberg notes that WHO's decision to pause enrollment in its hydroxychloroquine study is temporary, and says that enrollment could resume based on WHO's review of its own safety data. She is hopeful that the medical community will have a more solid understanding of effective treatments for COVID-19, based on the results of randomized trials, in the next 2 or 3 months.

Another concern highlighted by the open letter is that mean daily doses of hydroxychloroquine in the observational study were 100 mg higher than FDA recommendations, although 66% of the study's data were from North American hospitals. In a tweet thread predating the open letter, Watson also said the wide range of antimalarial doses — the mean dose for chloroquine was 765 mg per day, for example, with a standard deviation of 308 mg — make the results hard to interpret.

The study data is reported at a global level, with some details provided by continent but not for individual hospitals. The 671 hospitals that gave data to Surgisphere are not named, and the authors did not make the statistical code underlying the study publicly available for others to use and critique. "The authors have not adhered to standard practices in the machine learning and statistics community," the open letter states.

The open letter asks for aggregated patient hospital-level data to be made public, and for an independent analysis of it to be overseen by WHO. In his statement, Mehra said that he and his coauthors will initiate "an independent academic review of the data."

Marcus Banks is a health journalist whose work has appeared in Spectrum, TCTMD, and Nature Medicine.

For more news, follow Medscape on Facebook, Twitter, Instagram, and YouTube.  Here's how to send Medscape a story tip.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.