The E-Cigarette Debate: What Counts as Evidence?

Amy Lauren Fairchild, PhD, MPH; Ronald Bayer, PhD; Ju Sung Lee, MHA


Am J Public Health. 2019;109(7):1000-1006. 

In This Article

Systematic Review Approaches

In February 2018, PHE issued "Evidence Review of E-Cigarettes and Heated Tobacco Products."[8] Its aim was to "to summarise evidence to underpin policy and regulation" of e-cigarettes.[8] (p25) More importantly, it was the first in a series of annual updates required by the Tobacco Control Plan. It thus represented an ongoing commitment to monitor the emerging evidence of the risks and benefits of e-cigarettes in a nation that was at the forefront of promoting these devices as part of an explicit smoking harm-reduction campaign that prioritized reducing health risks over achieving total abstinence from nicotine.[8]

The report used "systematic review methods" focused on other systematic reviews, analyses of survey data, government reports, and "high profile studies" or studies that provided "new relevant information" or that generated media interest published between January 2015 and August 2017, the period since PHE's previous systematic review.[8] (p28,29,150) The report relied on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) standard to report results. It also investigated e-cigarette–related accidents and poisonings. Because it used its earlier published systematic reviews as a starting point, the authors made a strategic decision:

"A full systematic review was not possible given the timeframe within which the report was commissioned and needed to be delivered, and the wide scope of the topics covered. However, a full systematic review was carried out for heated tobacco products. [8] (p28)"

Heated tobacco products were newly introduced into the market and have not been subjected to systematic review of potential health risks and benefits.[8]

While the report did weigh randomized controlled trials, it did not hold them to be the gold standard for its evidentiary review. The demands of randomized controlled trials, the PHE report noted, were "discordant with what happens in real life" and therefore were not generalizable at the policy level.[8] (p126) PHE authors argued that new inclusion criteria, such as those proposed by Villanti et al.[9] and a recent Cochrane review of e-cigarettes,[10] were required. For example, Villanti et al. underscored that no systematic reviews to date have addressed all of the most pressing use issues that contribute to variations in findings. Those include but are not limited to whether studies adequately measured exposure to e-cigarettes (as opposed to use on 1 or 2 occasions), whether e-cigarettes were used with the intention of cessation, and whether e-cigarette exposure actually preceded smoking cessation.[9]

The NASEM report, "Public Health Consequences of E-Cigarettes," released a month earlier, was quite different in tone and perspective. The Committee on the Health Effects of Electronic Nicotine Delivery Systems approach "incorporated major attributes of systematic reviews."[11] (p43) It included methods established by the Cochrane Collaborative, the Center for Reviews and Dissemination, the US National Toxicology Program's Office of Health Assessment and Translation, and ROBIS (a new tool for assessing Risk of Bias in Systematic Reviews).[11] The official charge was to analyze the research literature, identify the need for research to fill evidentiary gaps, and make judgments about the short- and long-term health effects of e-cigarettes. With a focus on health effects and an emphasis on determining causality, the committee underscored that it "did not treat all bodies of evidence equally, and prioritized human studies."[11] (p43) Considering their strength in determining causality, randomized controlled trials and prospective longitudinal studies provided the most robust evidence. The report took note of population-based ecological data on the changing prevalence of smoking and e-cigarette use over time and noted where it contradicted experimental data or observational data, yet it applied a precautionary standard in which proof of no harm was required.[11]

With these methodological differences as backdrop, we weighed how each report addressed the central challenges posed by e-cigarettes when it came to smokers, bystanders, and youths. Table 1 summarizes the findings that we discuss in detail in the following sections.