It's a Meta-Analysis, Not a Magic Wand

Natalia Pasternak, PhD


July 07, 2021

The meta-analysis rose through the ranks of the medical evidence pyramid during the COVID-19 pandemic. As doctors urgently searched for answers to pressing questions, they leaned harder than ever on preliminary science to help guide patient care. They felt like they couldn't afford to wait for high-caliber clinical trial data in the crisis, so they turned to alternatives.

But not all science is created equally.

Meta-analyses and systematic reviews are tools that aim to gather the best trials available on a drug or intervention and organize that evidence according to its quality, the type of trial, number of people involved, and other factors. The meta-analysis is a statistical procedure that pools together the data of many trials, creating a virtual larger one. A benefit of collecting information in this way is it can lead to higher statistical power and a more robust estimate than is possible from any individual study.

However, any meta-analysis is only as good as the trials it includes.

When a meta-analysis combines many poorly conducted trials, observational studies, and case studies, the results can be dubious. In the book The Conscious Universe, Dean Radin, PhD, claims that his meta-analysis of psychic data "proves" that it was backed by science. But the work hasn't held up under statistical review and the book, published in 1997, is refuted. (The case was documented by The Skeptical Inquirer magazine.) Statisticians call out the file-drawer effect in this work, pointing out that only positive results tend to get published and negative ones are left in the proverbial cabinet, biasing any analysis that combines the results.

During the COVID-19 crisis, the meta-analysis was brandished again to support the use of controversial medications and treatments. Some stepped into the limelight claiming the mere existence of a single meta-analysis favoring a drug was enough to establish proof of benefit beyond reasonable a doubt. But that's not how it works.

And even when presented in a neat package, the meta-analysis doesn’t have the power to turn lead into gold. That's why the caveat "garbage in, garbage out" is so important. What goes into the meta-analysis determines what comes back out again, and the conclusions will be compelling or highly questionable.

The meta-analysis has been broadly applied during the pandemic to encourage the use of unproven drugs such as hydroxychloroquine and ivermectin. However, taking a closer look at these meta-analyses shows that there is more lead than gold within. Some studies included preprints together with peer-reviewed articles, while others included a small number of articles and added low-quality observational data or deeply flawed clinical trials. And almost all of the meta-analyses excluded well-designed, randomized clinical trials.

Having worked in science communication for many years, I know the most difficult themes to explain to the public are correlation vs causation and risk assessment. People tend to want to deal in absolutes with right or wrong answers to yes or no questions that are easy to follow. They want simple instructions about to what to do, what medication to take, and what to expect.

The pandemic brought about a new challenge for scientists and health teams, and many struggled to explain what we don't know and why we don't know it yet. However, it is crucial that we explain uncertainty without making people feel insecure or disregarded.

In Brazil, patients are demanding to be prescribed drugs with no evidence that they work, and some doctors are offering them under the guise of meta-analyses backing their decision making. Invigorated by claims on social media, many ill-informed patients are demanding to be treated with so-called "miracle cures."

I believe that the doctor's office is as good a place as any -- maybe even better than most -- to start educating people about how science works and how we test drugs and vaccines.

And there is no magic wand.

Regardless of how we feel personally about psychic experiences and whether they exist, scientific confirmation is built brick by brick on a pyramid of quality evidence. Not all science is created equally, and we must never embrace it all as though it were.

I'd love to hear about the faulty meta-analyses you've been reading as we work to debunk the junk and hold science to the highest standards.

Follow Medscape on Facebook, Twitter, Instagram, and YouTube

About Dr Natalia Pasternak
Natalia Pasternak, PhD, has a bachelor's degree in biology from the University of Sao Paulo, and a PhD and postdoctoral fellowship in microbiology (bacterial genetics) at the Institute of Biomedical Sciences. In 2018, she founded the Instituto Questão de Ciência in Brazil, the first Brazilian institute for the promotion of skepticism and critical thinking. She is the publisher of Questão de Ciencia magazine; a columnist or contributor for O Globo newspaper, CBN radio, and The Skeptic magazine (UK); and a fellow of the Committee for Skeptical Inquiry (US). Currently, she works as a research fellow in the Vaccine Development Laboratory at the University of Sao Paulo and is a visiting professor at the Public Administration School at Fundação Getulio Vargas, São Paulo.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.