Many widely cited cardiovascular studies are later refuted

Chicago, IL - Just because a trial has been cited widely does not mean that its findings are accurate, new research has found.

In a paper in the July 13, 2005 issue of the Journal of the American Medical Association^[1], Dr John Ioannidis (University of Ioannina School of Medicine, Greece) concludes that "contradicted and potentially exaggerated findings are not uncommon in the most visible and most influential original clinical research."

Ioannidis examined all original clinical-research studies that were published in the three major journals (New England Journal of Medicine, JAMA, and the Lancet) and "high-impact-factor" specialty journals between 1990 and 2003 that were cited more than 1000 times in the literature. He compared the results of these highly cited studies with those from subsequent studies of comparable or larger sample sizes and similar or better designs. The same analysis was also performed for matched studies that were not as highly cited.

He found 49 highly cited original clinical-research studies, 45 of which claimed that the intervention was effective. However, approximately one third of these studies were subsequently believed to have shown inaccurate results. Specifically, seven studies (16%) were contradicted by subsequent clinical studies and another seven were later shown to have found an exaggerated benefit. Of the remaining studies, 20 were replicated and 11 remain largely unchallenged.

Ioannidis's paper serves as a reminder of the perils of small trials, nonrandomized trials, and those using surrogate markers. Of the six nonrandomized trials identified as highly cited, five were found to be inaccurate. And more than one third of the top-cited randomized trials published between 1990 and 1995 have since been shown to be inaccurate; small trials are more likely to have their findings refuted or questioned later. There were only two studies with surrogate markers included in this paper, but both were subsequently contradicted in their clinical extrapolations.

Ioannidis concludes: "Evidence from recent trials, no matter how impressive, should be interpreted with caution when only one trial is available. It is important to know whether other similar or larger trials are still ongoing or being planned. Therefore, transparent and thorough trial registration is of paramount importance to limit premature claims [of] efficacy."

Of the 49 highly cited studies identified by Ioannidis, more than half were in the cardiovascular field.

Examples of highly cited heart-disease studies that were later contradicted:

• The Nurses' Health Study, showing a 44% relative risk reduction in coronary-artery-disease events in women receiving hormone therapy. This was later refuted by the much larger Women's Health Initiative, which found that hormone treatment significantly increases the risk of coronary events.

• Two large cohort studies, the Health Professionals Follow-Up Study and the Nurses' Health Study, and a randomized trial of 2002 patients all found that vitamin E was associated with a significantly reduced risk of coronary artery disease. But larger randomized trials subsequently showed no benefit of vitamin E on coronary artery disease.

Examples of cardiovascular trials showing exaggeratedly strong results:

• The Primary Angioplasty in Myocardial Infarction (PAMI) randomized trial of 395 patients showing immediate angioplasty to be better than tissue plasminogen activator (tPA) in MI patients, with a 58% reduction in relative risk of death/MI. A subsequent meta-analysis with more than 2500 patients suggested the benefit of angioplasty is probably much smaller.

• The Stent Restenosis Study and the BENESTENT studytwo randomized trials of 410 and 520 patients, respectively, showing that stents were superior to angioplasty in the management of coronary artery disease, with 31% and 42% relative risk reductions, respectively, in the need for revascularization. Later, a meta-analysis of almost 10 000 patients suggested that the benefit is probably much smaller (about a 10% relative risk reduction).

Heartwire from Medscape © 2005 

Cite this: Many widely cited cardiovascular studies are later refuted - Medscape - Jul 12, 2005.