August 8, 2007 (Los Angeles, CA) - When Dr Steven Nissen and Kathy Wolski (Cleveland Clinic, OH) published their 42-trial meta-analysis in the New England Journal of Medicine ( NEJM) examining the cardiovascular safety profile of rosiglitazone (Avandia, GlaxoSmithKline [GSK]), they found a 43% statistically significant increase in MI and a 64% increase in CV death that just missed statistical significance [ 1]. But in a new analysis based on the same trials, but using several different statistical methods, Dr George Diamond (Cedars-Sinai Medical Center, Los Angeles, CA) and colleagues found no significant increase in either end point [ 2]. They also detail what they say are critical flaws in the way Nissen and Wolski conducted their analysis, although one author agrees there is a “suggestion” of increased risk.
Nissen, in response, says the reanalysis of his work is now neither "useful" nor "relevant" because the FDA has subsequently conducted its own patient-level analysis, available through the FDA's website. Nissen's meta-analysis, as well as the analyses performed by Diamond et al, used study-level data. "We performed a study-level analysis because the company would not provide patient-level data," Nissen said, but he pointed out that the FDA's analysis turned up similar results to his own.
Dr Sanjay Kaul (Cedars-Sinai Medical Center), one of the authors of the new analysis, told heartwire that he and his coauthors were concerned about how Nissen and Wolski selected the studies they included in their analysis, how they were combined, and, in particular, about the fact that their methodological approach, known as the Peto fixed-effect model, could not account for the exclusion of studies in which no cardiovascular events occurred. Given these questions, Nissen's gloomy predicaments about rosiglitazone in the mainstream media following the publication of the original analysis were unjustified, Kaul said.
"You need to utilize every tool you have in your armamentarium to explore the robustness of the data. If different methods are giving you the same answer, you know the data are quite robust. But if different methods are giving you different answers, both qualitatively and quantitatively, you need to be more measured in your pronouncements."
Diamond and Kaul also presented their analysis during the open public hearing at the July 30 FDA advisory panel hearing on rosiglitazone. Their concerns, and the results of their reanalyses of the data, appear online August 6, 2007 in the Annals of Internal Medicine.
The trouble with zeros
Diamond et al take issue with how Nissen and Wolski selected the 42 trials for inclusion in their study, despite having first identified 48 by their predefined inclusion criteria (having a randomized comparator group, plus at least 24 weeks of drug exposure). Six trials, however, were excluded from the start because they did not report any CV events.
"Right off the bat, that is not appropriate," Kaul told heartwire . "When you define a selection strategy, you don't take a peek at the data and then make up your mind whether or not you're going to include that trial. It's an axiom in science that the hypothesis should drive the data; the data should never drive the hypotheses."
The problem of null events also complicated the analysis of trials that were included, Diamond et al point out. "Zero-total-event trials may provide relevant information by showing that event rates for both the intervention and control groups are low and relatively equal," they write. By default, the meta-analysis software program used by Nissen and Wolski excluded any trials that had no events of interest in both the treatment and comparator groups--this meantexcluding four trials from the MI analysis and 19 trials from the CV death analysis.
"When we saw the analysis in its published form, we were struck by the number of zeros in the database; it was just a very sparse database," Diamond told heartwire . "And they reported only a single meta-analytic method, the Peto method. There are other methods to use, and other things you can do, to adjust for values of zero; they simply dropped the studies out of consideration. . . . So we took it upon ourselves to do the alternative analyses and correction factors, and when we did that we got numbers that were all lower than the single numbers that Nissen and Wolski reported."
In their paper, Diamond et al report six models using continuity corrections for dealing with the zero events. With these, the odds ratio for MI ranged from 1.26 to 1.34, and for CV death from 1.17 to 1.51, with wide confidence intervals, none statistically significant. According to Diamond, he and his colleagues also used Bayesian methods to examine the data, but the journal editors decided it would be "overkill" to include these additional analyses.
"The ones we included are the most conventional," Diamond told heartwire . "Some of them aren't that appropriate. The inverse-variance method I think most people would agree is not suited to this particular purpose. But the problem is there are a lot of zeros in the data, and all the methods have limitations. It's somewhat analogous to having a cheap camera and going out at night and taking some snapshots without a flash attached. Whatever pictures you take are not going to show very much, and it doesn't matter how many pictures you take. The fundamental problem is, there aren't that many events, and the methods are insensitive to the issue at hand when there are so few events."
Diamond continued, "The Peto method, by dropping out the studies with zero values, produces the highest estimate of risk of all the possible alternatives. And I don't know if they knew that at the time, or if they did any other exploratory analyses and reported only the Peto analysis, or if they just didn't think to do them. But our point was to say that the conclusions about increased risk are far more tenuous than are implied in the article."
Asked to comment on Diamond et al's analysis, Nissen was dismissive, insisting that the FDA's analysis, presented at the July 30 hearing, was better powered statistically to address the question of cardiovascular risk and included relevant data that neither he nor Diamond et al had access to. Moreover, the FDA's meta-analysis yielded odds ratios closer to those published in the NEJM than those reported by Diamond et al.
"The analysis published in the Annals is a reanalysis of the study-level data for rosiglitazone using more conservative statistical methods; it is no longer relevant," Nissen stated. "A much more statistically powerful, patient-level analysis was presented by the FDA at the advisory board meeting. The FDA analysis reported virtually identical results compared with our NEJM manuscript: a 40% increase in risk with rosiglitazone. When you have patient-level data, a study-level analysis is not useful or relevant."
| FDA analysis That FDA analysis, presented during last week's hearing, also used data from 42 trials, but not the same trials used in Nissen's paper (14 studies were different between the two analyses). The FDA also had access to patient-level data. The FDA's meta-analysis yielded an overall odds ratio of 1.4 for total ischemic events (95% CI 1.1-1.8; p=0.02), and 1.4 for serious ischemic events (95% CI 1.0-2.1; p=0.06). But when events were analyzed by "metagroups" that combined trials with similar comparator groups, the risk associated with rosiglitazone varied considerably. Results for cardiovascular death were also challenging to interpret, ranging from an odds ratio of 1.3 for ischemic heart disease deaths to 1.7 for all deaths, neither of which were statistically significant. When continuity corrections were applied in each cell of trials with zeros in one or both arms, the odds ratios were much closer to one, the FDA review noted. Moreover, subgroup analyses pointed to a potentially higher risk of adverse events with rosiglitazone in patients who were older, had heart failure, or took nitrates or ACE inhibitors. "The inconsistencies across the subgroups (particularly without the insulin trials) suggest that the ischemic effect of rosiglitazone varies considerably and that confirmation of these effects is needed to ascertain whether the overall effect is primarily driven by effects in identifiable subgroups," FDA reviewers concluded. GSK's meta-analysis pointed to a 31% increase in myocardial ischemic events (95% CI 1.01-1.70). During the open public hearing, FDA reviewers also saw observational results from the Department of Defense, using its TRICARE PRIME database, and a second study conducted by WellPoint, a major American health insurance company. In both observational studies, there was no increased risk of cardiovascular events with rosiglitazone or with pioglitazone (Actos, Takeda Pharmaceuticals), the only other thiazolidinedione on the market, as compared with other oral antidiabetic drugs. |
Clinical heterogeneity problematic
Diamond et al also take issue with the heterogeneity in study designs and in the populations combined for the analysis, particularly since Nissen and Wolski justified combining the data on the grounds that the studies lacked statistical heterogeneity by the Cochran Q test.
"We believe that the decision to pool all studies despite design and population heterogeneity probably led to artificial inflation and precision of the risk estimate," Diamond, Bax, and Kaul write.
According to Kaul, the Cochran Q test "has low statistical power when the event rates are low. It's usually designed for large sample sizes, large event rates."
Kaul also argues that sufficient clinical heterogeneity may be enough to preclude pooling the trials. "We believe that both clinical as well as statistical heterogeneity should be considered when you're pooling data; otherwise, as somebody once said, when you combine apples and oranges, sometimes you come up with no more than a hill of beans."
Diamond et al point to the fact that some of the trials in the meta-analysis were of diabetics, while others were in nondiabetics, including one study of people with Alzheimer's disease and two with psoriasis patients. One trial included patients with congestive heart failure--a group already identified as a contraindication. Forty of the 42 trials were small and of shorter duration, while the remaining two enrolled more than 4000 patients with a follow-up of three to five years. Then there were differences in treatment groups: some trials were placebo-controlled against rosiglitazone monotherapy, some were studying rosiglitazone monotherapy against active controls, and others were add-on rosiglitazone on a background of standard antidiabetic therapy.
Indeed, during the FDA hearing, panel members saw evidence to suggest that rosiglitazone's cardiovascular risk profile is likely greater in studies that were placebo controlled but that the drug shows little or no suggestion of increased risk in trials that compared rosiglitazone with metformin or with sulfonylurea.
"These meta-analyses work best when they are used as tools, not as weapons, so we have to be very circumspect about these," Kaul said. "I will be the first person to acknowledge there is no single best or correct approach to meta-analyses, especially with sparse data. So when that is the situation, we need to explore alternative methods to examine the sensitivity of the data to the operative assumptions. And the results of the sensitivity analyses should be reported in a transparent manner."
Asked by the FDA advisory panel member Dr Nancy Geller (National Heart, Lung, and Blood Institute, Bethesda, MD) during the hearing whether he had conducted any sort of continuity corrections to account for the zero events, Nissen said that he and Wolski had indeed done so and obtained "essentially identical results."
It's a response that sits ill with Diamond and Kaul, both of whom raised the issue with heartwire . Kaul called it "a mathematical impossibility," while Diamond deemed it "inconsistent with our analyses."
"If you include the zeros, the numbers will change," Diamond said. "If he did do that, he would have had to do that with some other statistical package than the one he reports in the paper, because that package doesn't permit that."
In response, Nissen told heartwire that he and Wolski are providing alternative analyses in an upcoming response to letters to the editor in the NEJM. Moreover, he added, "The question of studies with zero events is not particularly important. These were small short-term studies that contribute almost nothing to the analysis. The fact that the FDA analysis showed nearly identical results validates our approach."
To heartwire , Kaul emphasized that their reanalysis was focused solely on Nissen and Wolski's paper, but he doesn't accept the idea that the FDA's analysis or a meta-analysis performed by GSK is in any way "vindication" for Nissen.
"The FDA and the GSK database analyzed all myocardial ischemic events, which includes serious as well as nonserious events," Kaul said. "When you focus your attention only on the serious events, neither the FDA nor the GSK meta-analysis showed a statistically significant increase in risk."
But one of the FDA reviewers, Dr David Graham, argued during the hearing that even in the absence of statistical significance, the "compass points" for rosiglitazone consistently point in the wrong direction.
"I think there is some validity to that statement," Kaul admitted to heartwire . "If you were to ask me, is there increased risk? I would say yes, there is a suggestion of increased risk, but the data are weak and there is some uncertainty associated with it. Is it certain enough to warrant yanking it off the market? No."
Diamond and Kaul report in the paper that they have no conflicts of interest; Nissen, in his NEJM paper, reports receiving research support to perform clinical trials through the Cleveland Clinic from Pfizer, AstraZeneca, Daiichi Sankyo, Roche, Takeda, Sanofi-Aventis, and Eli Lilly. Nissen also consults for drug companies but states that he donates any fees to charity, receiving neither income nor tax deductions for doing so.
Nissen SE and Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med 2007; 356:2457-2471. Abstract
Diamond GA, Bax L, Kaul S. Uncertain effects of rosiglitazone on the risk for myocardial infarction and cardiovascular death. Ann Intern Med 2007; 147; published online before print August 6, 2007. Available at: https://www.annals.org. Abstract
The complete contents of Heartwire , a professional news service of WebMD, can be found at www.theheart.org, a Web site for cardiovascular healthcare professionals.
Heartwire from Medscape © 2007 Medscape
Cite this: Shelley Wood. Reanalysis of Nissen/Wolski Rosiglitazone Meta-Analysis Scrutinizes Findings, Methods - Medscape - Aug 08, 2007.
Comments