Why Number Needed to Treat Can Be Misleading for Vaccines

Some believe that the widely reported efficacy of the various COVID-19 vaccines overstates their true benefit. These commentators propose that in addition to the vaccine efficacy, which is based on the relative risk reduction (RRR), we must consider the absolute risk reduction (ARR) and number needed to treat (NNT).

Andrew Althouse, PhD

I and others disagree, and I'll explain why the NNT concept is misleading when applied blindly to the vaccine trials.

RRR, ARR, NNT Explained

Initial reports that the mRNA COVID-19 vaccines are 95% effective and the adenovirus vaccine is 70% effective were computed on the basis of the relative risk reductions observed in phase 3 clinical trials. Consider the following hypothetical example of a vaccine trial that enrolls 40,000 participants (half receive the vaccine, half get the placebo). The trial is designed to perform a statistical analysis after 200 confirmed cases of disease (Figure).

The relative risk (RR) is computed by dividing the percentage of patients that contracted disease in the vaccine arm by the percentage of patients that contracted disease in the placebo arm, which is about 0.05 in the example provided. The RRR is computed by subtracting the RR from 1, yielding an RRR of about 0.95. This translates to a "vaccine efficacy" of 95%, meaning that 95% of cases that would occur in an unvaccinated population are prevented in a fully vaccinated population (assuming the same background risk and timeframe in which the trial occurred).

Those who argue that this overstates the vaccine's benefit say we must go further and look at the ARR, which does indeed appear less impressive — on the surface. The ARR is computed by subtracting the percentage that contracted disease in the vaccine arm from the percentage that contracted disease in the placebo arm, for an ARR of 0.9% in our hypothetical vaccine trial. The NNT — or the number of patients that need to receive the treatment to prevent one case of disease — is computed by taking the reciprocal of the ARR. In this case, an ARR of 0.9% translates to an estimated NNT of 111, meaning that 111 people would need to receive the vaccine to prevent one case of disease.

The NNT concept has been taught and promoted widely for interpreting clinical trial results, and admittedly provides a useful perspective in many cases. However, there are a few important contextual points to consider when attempting to compute the NNT for a vaccine.

The first issue is that the ARR and NNT are influenced by the baseline risk for infection and amount of time at risk, while the RRR is not affected by these. Because the vaccine trials were relatively short (a few weeks) and carried out in a time where many people were taking risk-mitigation measures such as masking and social distancing, an ARR computed from the trial data understates the effect that the vaccine would have on a person's absolute risk over a longer duration of time, especially if risk-mitigation measures such as masking and social distancing are relaxed.

Therefore, the NNT computed on the basis of the vaccine trial's data is not an accurate estimate of the NNT in the real world, because people will continue to be at risk for months or potentially years — much longer than the vaccine trial participants were followed (until the prevalence of COVID-19 declines dramatically).

The second issue (less appreciated but more important) is that vaccine trials typically use an "event-driven" analysis approach, meaning that efficacy analyses are performed after a prespecified number of observed events. The absolute risk reduction is therefore bounded at a low number, as explained below.

The Perfect Vaccine and Varying NNT

Our hypothetical trial above was designed to perform a statistical analysis after 200 confirmed cases of disease. What if our vaccine were perfect — 100% effective? Then all 200 cases would occur in the placebo arm. There will be 200 cases out of 20,000 patients in the placebo arm (1%) vs no cases out of 20,000 patients in the vaccine arm. The ARR in this trial is only 1%; this would be converted to an NNT of 100, meaning that 100 people must receive the vaccine to prevent one case of disease.

Now consider a similar trial of 40,000 participants, designed to perform its analysis after 2000 confirmed cases of disease rather than 200. Again, if the vaccine is perfectly effective, all cases will be observed in the placebo arm. But this time, with 2000 cases out of 20,000 participants, the absolute risk will be 10% for the placebo arm. The ARR is now 10%, translating to an NNT of 10 (ie, 10 people must receive the vaccine to prevent one case of disease).

The vaccine effectiveness is the same — 100% effective in both trials — yet the ARR and NNT are quite different. This is because the second trial simply waited until a much larger number of cases were observed. With event-driven trial design, the ARR is "bounded"; it literally cannot be higher than the number of events at which the analyses are scheduled, divided by the number of patients in the placebo arm.

Furthermore, this is a necessary feature of vaccine trial design for vaccines to be useful in the real world. If vaccine trials were required to demonstrate a large ARR, we wouldn't analyze the results until a large percentage of the enrolled patients had contracted the disease. For a 100% effective vaccine to demonstrate a 20% ARR, the trial would have to wait until 20% of the placebo arm had contracted the disease.

This would be counterproductive for public health, because the incidence of disease in the placebo arm of a vaccine trial will approximate the incidence in the population at large (if anything, it's likely to be lower, as vaccine trial participants tend to be health-conscious). The analysis would therefore be delayed until a large share of the general population had already contracted the disease.

It would essentially prevent vaccines from ever being approved in time to be useful.

The important take-home message is that you should not interpret an ARR or number needed to vaccinate from the COVID-19 vaccine trials without also considering these key points about the trials:

The trials occurred over a few months, whereas the disease(s) they are intended to prevent will probably (in the absence of vaccines) remain with us for far longer than that.
They occurred during a time when people are probably taking risk-mitigation measures (masking and social distancing) that we hope to one day relax, meaning the ARR is lower than it would be in a "fully open" society with no risk mitigation.
They were designed with event-driven analyses, which place a relatively low ceiling on the maximum ARR they can demonstrate; furthermore, this feature of vaccine trials is necessary to produce results in time for the vaccine to be useful in reducing spread of the disease.

I hope this clarifies why the NNT concept is quite misleading when applied to the vaccine trials. I would encourage readers to be more critical of the medical literature and to think more deeply about relevant contextual points when interpreting the results of the COVID-19 vaccine trials.

Andrew Althouse is assistant professor of medicine at the University of Pittsburgh as well as statistical editor for Circulation: Cardiovascular Interventions. His primary research interests are designing and analyzing clinical trials that hopefully may be used to inform medical practice. In his spare time, he enjoys lifting weights, cooking (and eating), and sampling craft beer.

Follow Andrew Althouse on Twitter

Follow theheart.org | Medscape Cardiology on Twitter

Follow Medscape on Facebook, Twitter, Instagram, and YouTube

Comments

Commenting is limited to medical professionals. To comment please Log-in.

Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.