Proceed cautiously when you come across a clinical study declaring a large treatment effect. Such findings most often emerge from small studies, and if replicated, the strength of the finding generally drops significantly. In addition, only very rarely do these reports show a significant survival benefit for patients, according to a study published in the October 24/31 issue of JAMA.
"Nominally significant very large effects arose mostly from small trials with few events," write the authors, Tiago V. Pereira, PhD, from the Health Technology Assessment Unit, Institute of Education and Sciences, German Hospital Oswaldo Cruz, Sao Paulo, Brazil; Ralph I. Horwitz, MD, from GlaxoSmithKline and the Yale University School of Medicine, New Haven, Connecticut; and John P.A. Ioannidis, MD, DSc, from the Stanford University School of Medicine in California. Dr. Ioannidis is known for a 2005 study published in PLOS Medicine entitled "Why Most Published Research Findings Are False."
The investigators searched the Cochrane Database of Systematic Reviews and identified 3082 systematic reviews with treatment comparisons and outcomes expressed in a total 85,002 binary-outcome forest plots.
Of the 228,220 trials in these systematic reviews, 9% reported a very large treatment effect, defined as an odds ratio of 5.0 or greater (or an odds ratio ≤ 0.20). This percentage also included studies that achieved nominal statistical significance (P < .05).
A total of 16% of the 85,002 forest plots came from this group. Often these trials are the initial and only report in the literature related to the treatment or outcome of interest, the authors note, and should be viewed cautiously.
Interestingly, very large effects that emerged from subsequent (not the initial) studies were more likely to be evaluated in follow-up research. In addition, when additional data were reported, treatment effect size generally shrank, with many findings losing their nominal statistical significance.
Only 3% of the forest plots involved very large effects with strong statistical support (P < .001). These very large effects pertained almost exclusively to nonfatal outcomes. In fact, across the 85,002 forest plots, only 1 intervention emerged with such a significant, positive effect on mortality from a study with no major validity concerns. In this case, researchers demonstrated that extracorporeal membrane oxygenation afforded a greater than 5-fold reduction in the likelihood of death compared with conventional ventilator support for newborns with severe respiratory failure.
"Based on this picture, most large treatment effect estimates should be considered with caution: many are spurious findings, while the vast majority may represent substantial overestimations," the authors note.
"The implications of all this for clinicians, patients, policy makers, and researchers is a reminder to be humble, uncertain, and collaborative," Andrew D. Oxman, MD, from the Global Health Unit, Norwegian Knowledge Centre for the Health Services in Oslo, writes in an accompanying editorial. "[A]lthough a large range of effective interventions are available, some with large effects, most have modest (albeit important) effects and the effects of many are uncertain."
Dr. Oxman notes this report may not represent the entire research literature. "[T]hat number of reviews is likely to be, at most, one-third of the number of reviews needed to reliably summarize what is known about the effects of health care interventions."
Some intervention effects may be under-represented in the study, the authors note. In general, researchers do not conduct randomized trials for interventions widely accepted as extremely effective. In addition, very large effects do not always guarantee an intervention is clinically useful.
Dr Pereira was supported in part by grants from Fundação de Amparo à Pesquisa do Estado de São Paulo. Dr. Horwitz is an employee of GlaxoSmithKline. Dr. Ioannidis and Dr. Oxman have disclosed no relevant financial relationships.
JAMA. 2012;308:1676-1684, 1691-1692.
Medscape Medical News © 2012 WebMD, LLC
Send comments and news tips to news@medscape.net.
Cite this: Clinical Research: Small Studies Often Yield Large Outcomes - Medscape - Oct 23, 2012.
