How Medicine Did Harm With Anecdotal Data

Benjamin Mazer, MD, MBA


August 11, 2020

I have often felt like COVID-19 was destined to be yet another "anecdotal epidemic." The news media, medical journals, and political leaders have all been amplifying anecdotal case reports and observational studies to try to make sense of this illness. In a few situations in particular — hydroxychloroquine and convalescent plasma, for example — this instinct has been especially seductive and posed the greatest risk of misleading us.

It would seem that we are finally turning a corner, with reason for authentic hope: Results are now available for a series of powerful clinical trials testing therapeutics, and phase 3 studies are already underway for promising vaccines. Yet these moments of true progress should not placate us; rather, they should inspire us to contribute to these efforts by encouraging enrollment in trials. Sadly, despite an overwhelming number of COVID-19 patients in the United States, some trials are having difficulty enrolling patients.

Interventions Don't Always Produce an Effect

To be the best advocates for rigorous evidence, we should understand why medical anecdotes are so seductive in the first place. It is often debated whether interventions by doctors do good or harm, but the underlying assumption we nearly always make is that they must do something. It is easy to incorrectly assume that when an outcome — good or bad — follows a treatment, then the two must be related.

Unwarranted faith in the power of medical intervention is hard to break. In its most destructive form, it can lead parents to misattribute their child's autism to the vaccines he or she happened to receive around the same time symptoms began.

We caution the public against this poor reasoning, but researchers and journalists are not immune to the same fallacy. In medical error research, for example, adverse outcomes are often misattributed to medical intervention simply because they occurred around the same time.

Let's look at a subtle but instructive example drawn from cancer screening. A study was recently published in Annals of Internal Medicine showing that patients who received a "negative" colonoscopy — meaning no cancer or polyps were found — had a lower than average chance of developing colon cancer. The appropriate conclusion to draw from this study is that a negative colonoscopy is reassuring. But there is no reason to believe that the screening colonoscopy actually altered the patient's risk for malignancy.

This message got lost in translation when shared with the public. One report of the study implied that a negative colonoscopy provided "durable efficacy" and "effective protection" against cancer. In fact, the patients with negative colonoscopies were the ones least likely to benefit from medical intervention: Their outcomes were good despite receiving no treatment.

Many of medicine's interventions may be "negative colonoscopies" in disguise — treatments that, rather than making patients healthier, are simply what healthier patients are more likely to receive. These subtle selection biases are easy to miss but might invalidate observational findings. That hasn't stopped some physicians from telling the media that patients in their hospital who received convalescent plasma treatments "appear to have recovered more quickly."

Large Effects Are Not Always Real

It is commonly taught that observational findings associating a treatment with a good outcome can be the false result of unmeasured confounding factors or chance alone. But when the observed effect is large, we downplay these caveats.

This misplaced faith in large effects has contributed to a feeling of whiplash over use of the drug hydroxychloroquine to treat COVID-19. The public has become confused by a barrage of observational studies, some reporting beneficial effects, others reporting no effect, and still others reporting harm. This confusion has even led some people to doubt the results of more reliable randomized trials.

Statisticians Andrew Gelman and John Carlin have described a possible explanation for the large effects seen in some of these studies: the "exaggeration ratio" — the difference in size between a published finding and the real effect. Gelman and Carlin observed that because so many scientific studies are underpowered and our journals are biased toward publishing only exciting "statistically significant" results, weak studies with exaggerated findings are surprisingly common. It is easy for "researchers to drastically overestimate the magnitude of an effect."

Common sense tells us that most interventions have only modest effects. It's likely that whether hydroxychloroquine ultimately helps or hurts COVID-19 patients, it probably doesn't make much of a difference overall.

The Case for Optimism and Action

Fortunately, the anecdotes and observational studies saturating the public's attention may soon be replaced by clinical trials run by universities and public health agencies around the world. A randomized trial for the antiviral drug remdesivir was published in May in The New England Journal of Medicine, and while it may not have shown the drug to be a miracle cure, it was an achievement in its own right. The trial began enrolling patients around the world as early as February, and the results were peer-reviewed and published just a month after the trial finished. That's warp speed for high-quality medical research. The World Health Organization is currently spearheading a global clinical trial that has the potential to test multiple possible therapies for COVID-19 quickly. The RECOVERY trial in the UK recently published results of a study testing multiple interventions simultaneously, demonstrating a benefit for dexamethasone in the sickest COVID-19 patients.

Convalescent plasma is also being studied rigorously for the first time, with multiple randomized trials being organized. The problem now is that researchers are having difficulty recruiting patients as they instead opt to receive the treatment through uncontrolled observational studies.

The fact is that medical science only advances because patients make a sacrifice by consenting to be experimented upon. This is a scary prospect. There has been an outpouring of gratitude toward healthcare providers during this crisis, but we should also be publicly thanking patients who volunteer for clinical trials.

A full-throated advocacy for clinical trials — in the media and in the clinic — is more important than ever as we enter the vaccine era of the pandemic. In this politically polarized environment, we can't let partisan antics discourage research and eventually adoption of potential COVID-19 vaccines. There is still value in presenting ourselves as thoughtful scientific practitioners.

When taking care of patients, we shouldn't just dismiss individual observations. I have seen experienced physicians pick up rare diagnoses and hidden concerns. Early case reports have alerted the public to emerging diseases and complications. But doctors are only as good as the evidence they draw on. Thanks to the rapid development of clinical trials for COVID-19, this will be one of the first pandemics where doctors won't have to rely on anecdotes alone — but only if we all lend our support.

Benjamin Mazer, MD, MBA, is a pathologist at Johns Hopkins Hospital with interests in diagnostic surgical pathology, laboratory management, and evidence-based medicine.

Follow Medscape on Facebook, Twitter, Instagram, and YouTube


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: