ACC/AHA Risk-Calculator Flaws Not Explained by Statin Use, Coronary Revascularizations

October 07, 2014

BOSTON, MA — The discrepancy between the lower observed risk of atherosclerotic cardiovascular disease (ASCVD) vs the much higher predicted event rate using the new American College of Cardiology/American Heart Association (ACC/AHA) cardiovascular risk calculator is not explained by patients starting on statin therapy after their baseline assessment or having a coronary revascularization—two possible explanations for this gap, according to the results of a new study[1].

The analysis, which is published October 6, 2014 in JAMA: Internal Medicine, was conducted by Dr Nancy Cook and Dr Paul Ridker (Brigham and Women's Hospital, Boston, MA). Cook and Ridker are the two researchers who first shook things up when they found that the ACC/AHA pooled-cohort equations overestimated risk in three large-scale primary prevention cohorts—the Women's Health Study (WHS), the Physicians' Health Study (PHS), and the Women's Health Initiative Observational Study (WHI-OS)—by as much 150%.

The risk-assessment calculator was developed by an ACC/AHA expert panel of physicians who wrote new guidelines for the management of patient risk. It departs from the older Framingham Risk Score (FRS) and includes a patient's 10-year risk of stroke and cardiovascular disease.

In the latest analysis, the researchers turn again to an analysis of 27 542 women participating in the WHS and still find a considerable miscalibration between observed and predicted events even after accounting for statin use and coronary revascularizations.

"There were a lot of reasons that the pooled equations may not fit in our data," Cook told heartwire . "One of the big reasons was the use of statins. Not necessarily at baseline, because you can control for that easily. What happens is that people go on them over time, especially since the start of the Women's Health Study, as the drugs have become more popular. The baseline was in the 1990s, and that's when the first big statin trials were starting to come out. We wanted to see if that's what caused the discrepancy."

The Role of Statins

Within the WHS cohort, 632 women experienced an ASCVD event, translating into a 10-year event rate of 2.2%. But when the ACC/AHA risk calculator was applied to the baseline data, the 10-year predicted event rate was 3.6%. For patients at higher risk, those with a 10-year event rate >7.5%, the risk calculator overestimates risk by more than 40%, and by 90% among those with a lower risk of ASCVD.

To account for the use of statins, the researchers assumed the drugs would provide a 25% reduction in risk. As a result, there would be an expected increase in the observed 10-year risk of ASCVD from 10.8% to 11.2% among patients at the highest risk (those with an ASCVD risk >10%). Using the ACC/AHA risk algorithm, however, the 10-year risk of ASCVD in this group of patients is predicted to be 15.6%. Overall, after accounting for statins, the risk calculator overestimated risk by approximately 37% in those with a 10-year ASCVD risk >7.5% and more than 85% in the lower-risk patients.

"We had the predicted risk from the pooled equations, and that didn't change," said Cook. "For the observed risk, we said, what would happen if these people had not gone on a statin? We know from meta-analyses that statins lead to about a 25% reduction in cardiovascular disease rates. We basically increased the observed rates by that 25%. And we did this over time because at baseline statins weren't available."

At 10 years, 22% of women without a previous ASCVD event were taking a statin. In an analysis that excluded patients once they started statin therapy, there was little effect on the adjusted risk model.

The researchers also assumed that coronary revascularization would reduce the risk of ASCVD by 25%, an "optimistic" reduction that isn't borne out by clinical data comparing coronary revascularization vs medical therapy. Still, even with this generous 25% reduction in risk, there remained a discrepancy between the observed and predicted rates. For those at the highest risk of ASCVD, accounting for coronary revascularization increased the 10-year "observed" rate to 11.7% vs 15.6% using the ACC/AHA risk calculator.

Finally, in an analysis that sought to determine whether potential ascertainment biases could explain the differences between observed and predicted event rates, the researchers found there would need to be 60% more ASCVD events in the WHS cohort to match the number predicted by the risk calculator. "It's really too extreme," said Cook in reference to the number of extra events needed to match the predicted event rate.

Not a Trivial Overestimation

In an editorial[2], Dr Steven Nissen (Cleveland Clinic, OH) writes that the new analysis shows convincingly that the ACC/AHA risk calculator is inaccurate, even though this would be evident, says Nissen, to any doctor who uses it. Overestimating risk to this extent is not a trivial problem, either.

"While statins are valuable drugs, particularly in secondary prevention, they do have downsides, and prudence requires not administering drugs to patients who will likely not benefit," writes Nissen. "The implications of the overestimation of risk are profound. A 50% overestimation by the guideline risk equations would likely add millions of Americans to the rolls of patients for whom statins are recommended."

Nissen also recounts some of the controversies with the clinical guidelines, many of which were reported by heartwire . He points out that the ACC and AHA assumed responsibility for the prevention guidelines after the National Institutes of Health (NIH) withdrew as the supervising authority. The guidelines were issued in November 2013, five years in the making, and garnered a lot of media attention, especially the cholesterol guidelines, which shifted away from the traditional LDL-cholesterol targets.

Regarding the flawed risk calculator, Nissen said—as he has in the past—it should have been published prior to the release of the guidelines and subjected to serious peer review. This would have allowed researchers to test it against known cohorts to see whether it was accurate. Right now, the flaws in the risk-assessment calculator simply undermine the confidence the medical community has in the guidelines. "The ACC and AHA should promptly revise the guidelines to address the criticisms offered by independent authorities," states Nissen.

Driving a Conversation

Asked to comment on the suggested revisions, the AHA provided a statement attributed to president Dr Eliot Antman (Brigham and Women's Hospital, Boston, MA) saying that these comments are the same as those heard when the guidelines were published last year. The AHA says that the risk-assessment calculator and the cholesterol guidelines have been validated in other studies. Since they were presented last year, the AHA has maintained that the risk calculator and clinical guidelines are tools to help physicians have conversations with patients about appropriate care, something Antman says they are doing.

As pointed out by Cook and Ridker, however, the overestimation of risk with the ACC/AHA risk calculator has now been observed in seven external validation cohorts. The ACC/AHA guidelines committee also attempted to validate the risk calculator in the Multiethnic Study of Atherosclerosis (MESA) and Reasons for Geographic and Racial Differences in Stroke (REGARDS) observational cohorts and found that it significantly overestimated risk. An analysis of the Rotterdam Study yielded similar overestimations. 

To heartwire , Cook noted that the WHS includes healthier patients, which is reflected by the risk factors. Blood pressure and cholesterol levels are lower, she noted, adding that "if the risk equation works, it should work in everybody." Cook believes the risk calculator can be recalibrated, something she said is not very hard to do and wouldn't require starting from scratch. She does not think it would be completely necessary to refit the risk calculator using a more contemporary patient population.

Ridker is a coinventor on patents held by the Brigham and Women's Hospital relating to the use of inflammatory biomarkers in cardiovascular disease that have been licensed to AstraZeneca and Siemens and has received research grant support from AstraZeneca and Pfizer. Cook  has reported no relevant financial relationships. Nissen reports that the Cleveland Clinic receives funding to perform clinical trials from AstraZeneca, Amgen, Eli Lilly, Pfizer, the Medicines Company, Novartis, Takeda, Orexigen, Vivus, and Eli Lilly. He is involved in these clinical trials but receives no personal remuneration for his participation and consults for many pharmaceutical companies but asks that they donate all honoraria or consulting fees directly to charity.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.