Many widely cited clinical-research studies are later refuted

July 12, 2005

Jul 12, 2005

Chicago, IL - Just because a trial has been cited widely does not mean that its findings are accurate, new research has found [ 1 ].

In a paper in the July 13, 2005 issue of the Journal of the American Medical Association, Dr John Ioannidis (University of Ioannina School of Medicine, Greece) concludes that "contradicted and potentially exaggerated findings are not uncommon in the most visible and most influential original clinical research."

Ioannidis examined all original clinical-research studies that were published in the three major journals ( New England Journal of Medicine, JAMA, and the Lancet) and "high-impact-factor" specialty journals between 1990 and 2003 that were cited more than 1000 times in the literature. He compared the results of these highly cited studies with those from subsequent studies of comparable or larger sample sizes and similar or better designs. The same analysis was also performed for matched studies that were not as highly cited.

He found 49 highly cited original clinical-research studies, 45 of which claimed that the intervention was effective. However, approximately one third of these studies were subsequently believed to have shown inaccurate results. Specifically, seven studies (16%) were contradicted by subsequent clinical studies and another seven were later shown to have found an exaggerated benefit. Of the remaining studies, 20 were replicated and 11 remain largely unchallenged.

Ioannidis's paper serves as a reminder of the perils of small trials, nonrandomized trials, and those using surrogate markers. Of the six nonrandomized trials identified as highly cited, five were found to be inaccurate. And more than one third of the top-cited randomized trials published between 1990 and 1995 have since been shown to be inaccurate; small trials are more likely to have their findings refuted or questioned later. There were only two studies with surrogate markers included in this paper, but both were subsequently contradicted in their clinical extrapolations.

Ioannidis concludes: "Evidence from recent trials, no matter how impressive, should be interpreted with caution when only one trial is available. It is important to know whether other similar or larger trials are still ongoing or being planned. Therefore, transparent and thorough trial registration is of paramount importance to limit premature claims [of] efficacy."

Heart disease—the most highly cited field

Of the 49 highly cited studies identified by Ioannidis, more than half were in the cardiovascular field.

Examples of highly cited heart-disease studies that were later contradicted include:

  • The Nurses' Health Study, showing a 44% relative risk reduction in coronary-artery-disease events in women receiving hormone therapy. This was later refuted by the much larger Women ' s Health Initiative, which found that hormone treatment significantly increases the risk of coronary events.

  • Two large cohort studies, the Health Professionals Follow-Up Study and the Nurses' Health Study, and a randomized trial of 2002 patients all found that vitamin E was associated with a significantly reduced risk of coronary artery disease. But larger randomized trials subsequently showed no benefit of vitamin E on coronary artery disease.


  1. Ioannidis JPA. Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294:218-228.