Leading Journals Sound Alarm on COVID-19 Studies

Batya Swift Yasgur, MA, LSW

June 03, 2020

Editor's note: Find the latest COVID-19 news and guidance in Medscape's Coronavirus Resource Center.

Major journals are raising concern about the database underlying recent studies of patients with COVID-19.

The editors at The Lancet issued an "Expression of Concern" today regarding data reported in a large observational study that showed an association between the antimalarial drugs hydroxychloroquine and chloroquine — with or without the antibiotics azithromycin and clarithromycin — and dramatically increased in-hospital mortality and cardiac arrhythmias in patients with COVID-19 treated vs those not treated with these agents.

The Lancet editors state that "important scientific questions have been raised" about the data reported in that paper. The expression of concern followed an open letter, penned by more than 200 scientists, ethicists, and clinicians and posted on May 28, questioning the data and ethics of the study, as reported by Medscape Medical News.

A similar "Expression of Concern" was issued by the New England Journal of Medicine June 2, noting "substantive concerns" regarding the "quality of information" used in a study published May 1 that analyzed cardiovascular disease and mortality in patients with COVID-19, noting that there was no increase in in-hospital mortality with ACE inhibitors and angiotensin receptor blockers in the setting of COVID-19.

The statement was likewise a response to a June 2 open letter carrying 174 signatures.

The May 28 and June 2 open letter signatures were spearheaded by James Watson, DPhil, a statistician at the Mahidol-Oxford Tropical Medicine Research in Thailand.

The New England Journal of Medicine has "asked the authors to provide evidence that the data are reliable," stated Eric J. Rubin MD, PhD, editor of the New England Journal and author of the comment.

Far-Reaching Implications

On the basis of the findings reported by Mehra et al in The Lancet, the World Health Organization (WHO) halted enrollment in the hydroxychloroquine component of the SOLIDARITY trial, an international randomized trial of medications being used to treat COVID-19.

However, in light of the new critique of the data that appeared in The Lancet, a safety monitoring committee evaluated the mortality rates in the WHO's SOLIDARITY trial as well as the RECOVERY trial — a United Kingdom-based trial also studying hydroxychloroquine — and "recommend that there are no reasons to modify the trial protocol," stated WHO Director-General Tedros Adhanom Ghebreyesus today in a press conference.

Soumya Swaminathan, MBBS, MD, chief scientist at the WHO, added that it is an "urgent priority for all of us to do the needed studies, do the randomized controlled trials, in order to get that evidence as quickly as possible," as there are many limitations and "potential biases" in observational studies.

Unreliable Database?

The database in question, which was used in both studies, is a multinational registry (the Surgical Outcomes Collaborative, Surgisphere Corporation, Chicago, Illinois), the reliability and transparency of which have come under scrutiny.

As reported by theheart.org | Medscape Cardiology, the database used in The Lancet study included 96,032 patients from 671 hospitals on six continents who had been hospitalized between December 20, 2019, and April 14, 2020, with confirmed COVID-19 infection.

The open letter to The Lancet included concerns that the patients were not sufficiently stratified according to disease severity. In addition, serious concerns were raised regarding "methodological and data concerns," including lack of transparency regarding the hospitals contributing to the data source, and discrepancies in the Australian and African data.

On May 30, The Lancet issued corrections of some of the data that were called into question.

In response to the "skepticism from the scientific community and global institutions," Surgisphere, whose CEO and founder is study coauthor Sapan S. Desai, MD, posted a statement on the company's website.

"It is also vitally important that our scientific colleagues around the world understand the validity of our database, particularly regarding data acquisition, warehousing, analytics, and related reporting processes," the statement said.

Th statement noted that the independent academic audit that is being conducted will "validate" the "core principles" of Surgisphere — ie, data integrity, data security, data acquisition, warehousing, analytics, and reporting processes, and "bring further transparency to our work" and "further highlight the quality of our work."

Independent Review

Responding to both statements, Mandeep R. Mehra, MD, MSc, William Harvey Distinguished Chair in Advanced Cardiovascular Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, who was the lead author of both studies, said in a release that the "remaining coauthors of the recent studies…have initiated independent reviews of the data used in both papers after learning of the concerns that have been raised about the reliability of the database."

The goal of this independent auditor is to "verify the source data and assess the accuracy of the database and the authors' findings."

Mehra's statement emphasized that the review is "independent of Surgisphere" and its finding will be provided simultaneously to the editors of the journals and the coauthors of the studies.

"I have routinely underscored the importance and value of randomized, clinical trials and articulated that such trials will be necessary before any conclusions can be reached," he said, noting that use of the dataset was an "intermediary step undertaken because of the urgency of the situation."

Mehra added, "I eagerly await word from the independent audits, the results of which will inform any further action."

Postpublication Dialogue

Commenting on these developments for theheart.org | Medscape Cardiology, Harlan Krumholz MD, SM, professor of medicine (cardiology) and professor in the Institute for Social and Policy Studies, of Investigative Medicine, and of Public Health (Health Policy), Yale-New Haven Hospital, Connecticut, said that the expressions of concern "were needed and signal an urgent need to clarify the provenance and quality of the data used in these studies. No one should depend on these articles until the issues surrounding them are clarified."

Krumholz, who is also the director of the Center for Outcome Research and Evaluation, Yale-New Haven Hospital, said he has "concerns" about both studies "because they cannot be better than the data they used, and if there are questions about the data — the source, processing, curation, and analysis — then they should not be considered further until the audits are done."

He encouraged being "careful readers of the literature and engaging in post-publication dialogue to get to an understanding of what is strong enough to change practice or policy," as articles, "even in the best journals may have issues that undermine their impact."

In this case, "there may be even bigger issues, but we should wait for more information before judging." Meanwhile, "these studies need to be placed on hold," Krumholz concluded.

The Lancet. Published online June 3, 2020. Expression of Concern

N Engl J Med. Published online June 2, 2020. Expression of Concern

For more news, follow Medscape on Facebook, Twitter, Instagram, and YouTube.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.