Some Data May Be Worse Than No Data in the COVID Era

Editor's note: Find the latest COVID-19 news and guidance in Medscape's Coronavirus Resource Center.

I will critique a research letter on anticoagulation in patients with COVID-19 recently published in the Journal of the American College of Cardiology (JACC). Do not mistake this critique for statistical minutia. There are larger implications.

Researchers from Mount Sinai Hospital in New York City reported a retrospective observational study of nearly 2800 patients with COVID-19 hospitalized in March and April. They studied the association between use of anticoagulation (AC) and in-hospital mortality.

This has become an important question because numerous studies have suggested a high rate of thrombotic complications in patients with COVID-19.

Key Findings

In-hospital mortality for patients treated with AC was 22.5%, with a median survival of 21 days, compared with 22.8% and median survival of 14 days in patients who did not receive AC.
In patients who required mechanical ventilation (n = 395), in-hospital mortality was 29%, with a median survival of 21 days for those treated with AC vs 63% with a median survival of 9 days in patients who did not receive AC.
Bleeding events occurred in 3% of patients who received AC vs 2% in those who did not receive AC.

Study Authors' Conclusions

The authors write that their observations are limited by unobserved confounding, unknown indication for AC, lack of metrics to further classify illness severity in the mechanically ventilated subgroup, and indication bias.

But then they conclude that "Our findings suggest that systemic AC may be associated with improved outcomes among patients hospitalized with COVID-19." And this: "Interestingly, there was an association with AC and improved survival after adjusting for mechanical ventilation."

Comments to the Media

The second author of the paper, Valentin Fuster, MD, who is also the editor-in-chief of JACC, spoke to the media.

To theheart.org | Medscape Cardiology , Fuster said, "I can tell you any family of mine who will have this disease absolutely will be on antithrombotic therapy and, actually, so are all of the patients at Mount Sinai now."

To the Washington Post , Fuster said, "My opinion is cautious, but I must tell you I think this is going to help." He also said the Mount Sinai hospital system changed its treatment protocols several days ago to begin giving patients with COVID-19 higher doses of blood thinners.

Critique

Let’s start with the (slightly) positive: Academic intensivist Josh Farkas, MD, from the University of Vermont, tweeted that he found the low rate of serious bleeding compelling. I find that overly optimistic. This paper provides little details of the drugs used other than noting that oral, subcutaneous, and intravenous formulations were included. Comparing only AC vs no AC belies the complexity of the differing drug classes, routes of administration, and dosing.

The most important and likely fatal flaw in this study is immortal time bias, which the authors do not acknowledge. Immortal time bias occurs in observational studies when the groups are defined by exposure (AC) or lack of exposure (no AC) after follow-up has begun.

Patients who received an anticoagulant had to be alive in order to get it; they are said to be immortal before receiving the drug. In contrast, patients who did not receive an anticoagulant could have died any time during their hospitalization and, therefore, were mortal throughout the hospitalization.

To demonstrate this flaw in this paper, look at the immediate separation in the AC and no-AC groups on the Kaplan-Meier survival curve (Figure 2 in the paper). Within a handful of days of being admitted, about 20% of the patients in the no-AC group died but nearly all the AC group are still alive at 5 days. A group of clinicians from Massachusetts General Hospital discussed this issue in their FLARE newsletter.

A specific scenario from this study: A patient with COVID is admitted, then gets put on AC on hospital day 5; here, the AC exposure gets credit for that 5-day survival. That’s the immortal time. Conversely, the no-AC arm is assigned all the patients who did not live long enough to receive AC.

In other words, you could have given the patient who survived to 5 days anything and it would appear to have caused the longer survival. In 2018, cardiologist Robert Yeh, MD, authored a now famous Twitter thread of immortal time bias from an observational study of revascularization after myocardial infarction.

In addition to the lack of randomization and selection bias are implausible findings: The median duration of AC therapy is 3 days. It’s hard to believe that that little exposure to AC led to a dramatic increase in survival in patients on mechanical ventilation for respiratory failure. Implausible effect sizes further strengthen the presumption of bias.

The Larger Problem With Clinical Science

Medical journals are supposed to have vetting processes. It defies logic to think that Fuster’s role as the editor-in-chief of JACC had no effect on the rigor of peer review. I’m not an academic, and I am nervous about writing this column.

In a perfect world, referees would call it as they see it, but we don’t live in a perfect world. We live in a world where publication is the currency in academics. Why did multiple experts expose the flaws of this paper on Twitter within hours of publication, yet none of this surfaced before publication?

There are also ethical implications of publishing flawed studies with over-reaching conclusions. First, observational studies are prone to medical reversals. And even when observational studies show a true association, effect sizes are often inflated. Both outcomes could erode public trust.

More important, though, is that when influential researchers publish and promote over-reaching conclusions, it impairs our ability to get to the truth. If a system of hospitals such as Mount Sinai thinks enough of the practice to codify it in COVID-19 protocols, the equipoise needed to enroll patients in proper trials can be destroyed.

And if you falsely reduce equipoise, which is easy in a pandemic (Exhibit A: hydroxychloroquine), you could actually impede scientific knowledge. (We still don’t have a definitive answer with hydroxychloroquine.)

I have heard smart people counter by saying that the desperation of trying to help patients with COVID-19 is a reason to use observational data in clinical decisions. "It’s all we have," goes the argument.

Yet this paper and its downstream results actually argue that flawed data can be worse than no data. Whether by protocol or simply an understood standard of care, the effect is to reduce a clinician’s choice to individualize therapy in the absence of high-level evidence.

Randomized controlled trials are far from perfect. But in the case of using potentially dangerous interventions in COVID-19, blinding and random allocation are the best answer.

Observations are not useless; they can help us tweak and refine decision-making and influence the design of randomized controlled trials. But observational studies such as this should not be used to establish a new normal.

Even in a pandemic, slow science has its virtues.

John Mandrola practices cardiac electrophysiology in Louisville, Kentucky, and is a writer and podcaster for Medscape. He espouses a conservative approach to medical practice. He participates in clinical research and writes often about the state of medical evidence.

Follow John Mandrola on Twitter

Follow theheart.org | Medscape Cardiology on Twitter

Follow Medscape on Facebook, Twitter, Instagram, and YouTube

Comments

Commenting is limited to medical professionals. To comment please Log-in.

Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.