Analysis Attacks Stem-Cell Data in CHD, Another Gives Tepid Support

April 29, 2014

LONDON, UK ( updated ) —  The promise of autologous stem-cell therapy for ischemic heart disease has either taken an existential hit, if one agrees with an analysis in this week's literature[1], or deserves subdued, provisional approval if one favors another[2].

The more provocative of the two reports claims to have found >600 "discrepancies" in data, methodology descriptions, and other crucial information in 133 separate publications covering 49 randomized, controlled trials exploring the effect of autologous bone-marrow–derived stem-cell treatment on LV function in patients with ischemic heart disease.

The analysis acknowledges that when the trials, as reported, are considered together, "there is on average a positive effect on ejection fraction," but there was wide variability in how much individual studies contributed to that overall positive effect, the report states. As it happens, "the studies with the most discrepancies seem to be contributing most to the positivity, while the studies with no discrepancies show a zero effect" of cell therapy on LV ejection fraction.

We all want regenerative medicine to work, but genuine scientific advances may arrive in small steps.

To dismiss the discrepancies as minor or unimportant would be "an easy trap to fall into," the report's senior author, Prof Darrel P Francis (National Heart and Lung Institute, Imperial College London, UK) told heartwire . "Our study is evidence that they are very important, because without them the effect size is zero. For the good of our patients, we should all be asking why," he said by email.

"We all want regenerative medicine to work, but genuine scientific advances may arrive in small steps. Recognizing them will be difficult among a literature filled with unreliable claims of large effects."

The analysis, with lead author Dr Alexandra N Nowbar (Imperial College London), is published online April 29, 2014 in BMJ.

A Tale of Two Studies

The other report is a sobering assessment of the stem-cell literature that at least found room for optimism, but over a thin evidence base, in 23 randomized controlled trials encompassing >1200 patients with ischemic heart failure who received bone-marrow–derived stem-cell therapy vs control management.

In the meta-analysis, cell therapy "had no clear effect" on the risk of death or heart-failure hospitalization during the first year but may have shown later improvement, senior author Dr Enca Martin-Rendon (University of Oxford, UK) told heartwire by email. The analysis points to a long-term 72% mortality risk reduction with cell therapy after the 12-month mark. But the signs of long-term benefit "are based on a combination of few events and discordant results from a small number of studies and their small number of participants." The thin 2.6% average improvement in LV ejection fraction seemed insufficient to explain the long-term clinical outcomes, she noted. There was "no evidence of adverse events associated with the treatment."

Despite some encouraging findings, Martin-Rendon said, "The evidence that the stem-cell treatment reduces death is of low quality at present," although that could change with the arrival of further studies.

The analysis is published in the Cochrane Database of Systematic Reviews with first author Dr Sheila A Fisher (NHS Blood and Transplant, Oxford, UK).

According to Dr Douglas W Losordo (NeoStem, New York, NY), the Cochrane review is more on the right track than the analysis in BMJ. Stem-cell therapy as a field is itself embryonic, most of the studies are investigator initiated, "a lot of them are on shoestring budgets," and yes, "some of the accounting isn't as crisp as one would like," he told heartwire .

"Everybody should aim for very high standards in conducting clinical research, but I think when there are some discrepancies—most of which, the vast majority, are unintentional—to jump from that to say it therefore invalidates the findings, I don't know how you can really justify taking that leap," he said. "There's a disproportionate skepticism being raised about the results, based on these discrepancies."

Losordo, also an adjunct professor at Northwestern University Feinberg School of Medicine, Chicago, wasn't part of either analysis but contributed to the development of the ongoing phase 3 RENEW trial looking at stem cells for treating chronic myocardial ischemia. As NeoStem chief medical officer, he is also involved in the phase 2 PRESERVE-MI trial of stem cells administered in patients with acute ST-segment-elevation MI. He said he believes he had no involvement in any study included in the BMJ analysis but the Cochrane review includes two of his trials.

"Distortion" of the Field

According to Francis, "Even the strict Cochrane protocols forced [Fisher et al] to incorporate many trials with many discrepancies. The exciting reduction in mortality is driven by one trial, with 71% of the events, which . . . had numerous discrepancies, including impossible standard deviations and incorrect arithmetic. We think it should therefore be viewed with caution."

The exciting reduction in mortality is driven by one trial, with 71% of the events.

He and his colleagues also found discrepancies in two trials in which "the evaluation of ejection fraction [was] heavily weighted." Their analysis, he said, "suggests that the Cochrane system can be even bolder and more stringent."

Martin-Rendon said, for its part, the analysis from Francis's group "highlights how the science in this field has been distorted by, one, inclusion of nonrandomized studies into the evidence base; two, discrepancies in analyses that appear to bias findings toward favorable effects; and three, focus on intermediate non–clinical-event outcomes, with likely selective reporting."

The main implication of both reports, according to Martin-Rendon, "is that larger randomized trials with clinical primary outcomes . . . are needed to establish the value of stem-cell treatment." Small trials are essential in early evaluations of new treatments, she said, but they are no substitute for large trials exploring treatment effects on clinical outcomes.

Losordo said big part of the story is often left out of discussions of these trials: What if patients were given a random, unknown dosage of a study drug but were still included in the primary analysis? "No one would think that's a useful study, but that's what is done a lot, unfortunately, in this stem-cell field."

Generally, the bone-marrow mononuclear cells given to patients are part of a heterogeneous cell population, some of which are believed to be the "active agent," others that are not, and maybe still others "that one would think, and animal studies would show, would inhibit the repair process that you're trying to engender. To look at all bone-marrow–cell studies together doesn't make a lot of sense to me," Losordo said.

"This young field is maturing. Study designs keep evolving. Every study is a little bit smarter than the previous one; each gets a bit more informative than the last."

"Mathematical Impossibilities"

In the analysis of alleged discrepancies, which covered the literature to April 2013, the percentage-unit change in LV ejection fraction associated with stem-cell therapy ranged from a decrement of 3.9 to a rise of 14, while the documented number of discrepancies ranged from zero to 89.

"For example, we found that one trial had been published twice under different names, with different study sizes, and different designs—randomized in one publication and nonrandomized in another—and both reports contained mathematical impossibilities. Despite our repeated requests, the journal refused to notify readers of these serious problems," according to Francis. "In almost half of the trials that reported means and standard deviations of NYHA [functional class], it is mathematically impossible for the values to have come from genuine [NYHA-class] measurements."

We . . . believe that journals should be held responsible for notifying readers when they learn of impossible features in papers they have published.

The number of discrepancies in studies went up with the reported cell-therapy effect on ejection fraction (p=0.005). The five trials for which no discrepancies were identified showed the lowest mean effect, a negative change of 0.4 percentage units. The mean effect on LVEF in the 24 trials with one to 10 discrepancies was an increase of 2.1 units; it averaged 3.0 units for the 12 trials with 11 to 20 discrepancies, 5.7 units for the three trials with 21 to 30 discrepancies, and 7.7 units for the five trials with >30 discrepancies.

"This discrepancy problem may not exist if authors publish online appendices fully detailing raw results and detailed methods," Francis proposed. "We also believe that journals should be held responsible for notifying readers when they learn of impossible features in papers they have published. It is in patients' best interests for journals to relay information to readers rather than burying it."

Late Benefit May Be Strongest

The Cochrane review, which included the relevant trials reported through March 2013, showed a relative risk (RR) of 0.28 (95% CI 0.14–0.53, p=0.02) for mortality, stem-cell therapy vs control; the finding was based on eight studies with a total of <500 patients. The RR for rehospitalization due to heart failure was 0.26 (95% CI 0.07–0.94, p=0.04) based on only two studies with fewer than 200 patients in all. Both findings were at follow-ups exceeding one year; neither end point was significant within one year of treatment.

Cell therapy was associated with a mean 0.63-tier reduction (p=0.005) in the four-tier NYHA functional class scale for follow-ups under a year across 11 studies. The average improvement over longer follow-ups was a 0.91-tier reduction (p=0.0002) across four studies.

There were also significant improvements from cell therapy in mean LV end-systolic volume, but not LV end-diastolic volume, at both short- (p=0.001) and long-term (p<0.00001) follow-ups. Those findings were based on 13 and three studies, respectively.

LV ejection fraction went up a mean of 4.22 points (p<0.00001) within a year, based on 18 studies, and 2.62 points (p=0.02) after a year, based on six studies.

"However, the improvement in LVEF does not adequately define the clinical efficacy of bone-marrow stem-cell treatment," observe the Cochrane reviewers. Change in global LVEF is sometimes used as a surrogate end point, especially in large trials, but "its use in cell therapy is controversial." In the current analysis, they write, many patients started the trials with normal LVEF, and the primary end points—including mortality, new heart failure, HF hospitalization, NYHA class, and symptoms—were clinically oriented.

Only if the cell-therapy benefits suggested by this analysis are realized in appropriate trials could the technique "become a real potential treatment that could improve patients' quality of life," according to Martin-Rendon. She pointed to the ongoing, international phase 3 BAMI trial, which is randomizing a projected 3000 patients with successful reperfusion therapy for acute ST-segment-elevation MI and an LVEF <45% to receive bone-marrow–derived mononuclear cells or placebo; the primary end point is death from any cause.

The Nowbar et al coauthors have "no financial relationships with any organizations that might have an interest in the submitted work in the previous three years, [nor] other relationships or activities that could appear to have influenced the submitted work." A coauthor on the Cochrane report, Prof Anthony Mathur (William Harvey Research Institute, London, UK) is lead investigator of the BAMI trial; none of the other authors had relevant disclosures. Losordo discloses "speaking, advising, consulting, or providing educational programs" for Baxter Healthcare, which sponsors the RENEW trial, and that he later consulted for Baxter; in addition to his academic position, since August 2013 he has been an employee of NeoStem, "a biotech company developing cell therapies for cardiovascular disease, diabetes, asthma and cancer" that sponsors the PRESERVE-MI trial.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.