CV Research Spin Abundant in the Era of Alternative Facts

Patrice Wendling

May 03, 2019

In the court of the Internet, how facts are assembled is king. In the straightforward world of scientific analyses, facts are facts, right?

A new systemic review has a different take, suggesting that cardiovascular (CV) researchers put some kind of spin on their results in two-thirds of randomized controlled trial (RCT) reports published in six high-impact medical journals.

"The high prevalence of manipulation of language in the literature has great importance for consumers of medical evidence, which, given the freedom of information in the digital age, affects both patients as well as clinicians," lead author Muhammad Shahzeb Khan, MD, John H. Stroger Jr. Hospital of Cook County, Chicago, and colleagues write in their study, published today in JAMA Network Open.

The investigators identified 93 CV RCTs with statistically nonsignificant results for their primary outcomes published from 2015 through 2017 in the New England Journal of Medicine, the Lancet, JAMA, the European Heart Journal, Circulation, and the Journal of the American College of Cardiology. Two reviewers extracted all relevant data onto a standardized data collection form, with a third arbitrating any disagreements.

Following methodology recently set forth by Boutron and Ravaud, the team defined three spin strategies:

  • Pivoting to statistically significant secondary results, such as within-group comparison, secondary outcomes, and subgroup or per protocol analyses.

  • Interpreting statistically nonsignificant results of the primary outcomes as showing treatment equivalence or ruling out an adverse event.

  • Emphasizing the beneficial effect of the treatment with or without acknowledging the statistically nonsignificant primary outcome.

Results showed that spin was present in at least one section of 53 abstracts (57%; 95% CI, 47% - 67%) and 62 main texts (67%; 95% CI, 57% - 75%). Moreover, spin saturation reached all sections in 26 abstracts (28%; 95% CI, 20% - 38%) and 18 main texts (19%; 95% CI, 12% - 29%).

About half of the conclusion sections included spin, but it was mostly low-level spin, according to the authors.

Many physicians and journalists often "just grab the twisted line" in a published study without reading the full text or being aware of the statistical significance, Khan told | Medscape Cardiology.

"It's one of the major reasons we published this study," he said. "Readers should be aware, reporters should be aware, that there might be some twists in the report, despite all the peer review."

The study does not examine why some researchers resort to spin but Khan speculates it may be unconsciously done or reflect concerns that medical journals are less willing to publish studies with negative findings.

Contrary to previous studies ascribing reporting bias to financial interests, there was no association between spin and conflict-of-interest disclosures from the first or last author. Further, industry-funded research had a lower proportion of spin than nonprofit-funded research, the authors report.

Beating Bias

"Interpretation of trials is highly subjective and quantifying spin is even more subjective, but to their credit, they did correspondence analyses and there seems to have been reasonable agreement," said Sanjay Kaul, MD, cardiology division, Cedars-Sinai Medical Center, Los Angeles, who was not involved in the study.

Kaul noted that he would have liked more detail on the impact of spin on leaders, research funding, and the clinical-development program; and whether journals tried to "set the record straight" on RCT results in an accompanying editorial.

"A fair question to ask is why is the peer review not doing its job, if two-thirds of the articles published in high-impact journals have this problem," observed Kaul.

In an accompanying editorial, JAMA Network Open Deputy Editor Stephan D. Fihn, MD, MPH, University of Washington, Seattle, said they meet twice weekly to review manuscripts and "almost invariably there are 1 or more manuscripts for which the authors have placed their results in a more favorable light than we considered justified."

He details their efforts to address spin, such as having editorial staff "scour manuscripts to expunge unsupported causal language and correct overinterpretation of results," using external referees, and "on several occasions" requesting authors recast the manuscript's conclusions from weakly positive to categorically negative.

"At the same time, we make every effort to embrace well-founded, negative findings as avidly as positive ones," Fihn says. "Although negative trials typically do not attract as much attention, they are often critically important in countering erroneous inferences from earlier studies or debunking widely held misconceptions."

Kaul says the editorial begs the case that quality-control processes are in place to fish out spin, but fails to detail to what extent those processes work for JAMA Network Open or JAMA, one of the six journals in the review.

Increased publication of neutral/negative RCT findings may help reduce the level of spin, he said, noting that although he doesn't have an account, he looks to the Twittersphere to see how the medical community is accepting the results of a pivotal trial.

"I'm reassured to see that there is a lot of skepticism," Kaul said. "They don't readily accept these data and point out the methodological flaws and that there may be other degrees of interpretation. So the Twitterati may be one of minimizing the impact; open-access peer review is another."

Coauthor John Mandrola, MD, is a columnist for Medscape. Coauthor Richard Krasuski, MD, reported grants from Edwards Lifesciences and Abbott; grants and personal fees from Actelion; and serving as a nonfunded scientific advisor for Ventripoint outside the submitted work. Fihn reported no conflicts of interest. Kaul reports serving as a consultant for Boehringer Ingelheim, Novo Nordisk, Johnson & Johnson, and AbbVie.

JAMA Netw Open. Published online May 3, 2019. Full text, Editorial

Follow Patrice Wendling on Twitter: @pwendl. For more from | Medscape Cardiology, follow us on Twitter and Facebook.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: