Comparing the Quality of Pro- and Anti-vaccination Online Information

A Content Analysis of Vaccination-Related Webpages

Gabriele Sak; Nicola Diviani; Ahmed Allam; Peter J. Schulz


BMC Public Health. 2016;16(38) 

In This Article


The main aim of this study was to compare the quality of pro- and anti-vaccination webpages. Our analyses highlighted significant differences between pro-vaccination, anti-vaccination, and neutral (or undefined) webpages along all four quality dimensions: webpage design, interactivity, health-related content, and vaccination-specific content. Generally speaking, pro-vaccination webpages resulted to be qualitatively superior to both anti-vaccination and neutral (or undefined) web sources (Total Aggregated Quality Index). However, on some quality features, neutral and/or anti-vaccination webpages showed better results than the pro-vaccination subset.

According to our analyses, higher quality of pro-vaccination websites might be explained by higher professionalism of their owners, who were more often internationally recognized medical institutions (e.g., WHO, CDC), while anti-vaccination websites were often operated by activists of the anti-vaccination movement, that is private citizens expressing their personal views on the topic. This difference makes it likely that anti-vaccination websites are less often designed professionally and may have difficulties to keep up to date with regard to quality standards. However, it has to be noted that various counterexamples were observed with regard to the quality levels of some anti-vaccination webpages. Eighty-seven anti-vaccination webpages reviewed obtained high quality scores, satisfying between 23 and 29 of the quality attributes. For instance, the following anti-vaccination webpage: obtained a total aggregated quality score of 29 attributes out of 34. The website is administered by a well-known American doctor and activist (Dr. Joseph Mercola), and it promotes a wide range of health information such as wellness, dietary, and vaccines. The website has been ranked by the 'World's first Natural Health Website'. A further well-known counterexample is represented by the National Vaccine Information Center (NVIC), which is a non-profit organization advocating mainly vaccine safety (

Past research[39] has claimed that the most reliable domain name systems to retrieve quality health information might be the following three: .gov, .org, and .edu. However, given that the .org domain was also frequently the DNS of anti-vaccination web sources in our sample, we suggest that in the context of online vaccination information this should be limited only to the .gov and .edu domain options. In light of this, international and national bodies monitoring online vaccination information have to stress the positive relationship between governmental institutions (i.e., holding the .gov domain) and the provision of quality contents related to vaccination. For end-users, the identification of the DNS might be a simple and quick way to infer the credibility of a webpage disclosing information on immunizations (governmental or public affiliation).

With respect to aspects relating to webpage design, pro-vaccination web information was disclosed through well-organized web portals with highly operative internal links (i.e., increasing application of menu bars, and search toolbars), and, as a matter of fact, have to be considered aesthetically more pleasant to navigate than anti-vaccination webpages (i.e., lack of distracting promotional messages). Remarkably, the adoption of "two-way communication" formats resulted to be high (90.6 % of the final sample), and on average webpages being pro-vaccination satisfied this aspect better than the opponent subset. This finding may highlight the difficulties of web editors administering anti-vaccination webpages in designing modern and dynamic online platforms, which force end-users to consume contents only in a static traditional fashion.

The most notable exception to the quality gap between pro- and anti-vaccination websites occurred for the vaccination-specific information section. In this dimension, pro- and anti-vaccination websites were found to be equally informative. But does this mean the information on anti-vaccination websites is as good as on pro-vaccination-sites? Not necessarily so. The indicators employed in this study used the presence of information as criterion, not its relevance, not its substance, not its truthfulness. These are the ultimate quality criteria, nevertheless they are very difficult to measure and next to impossible to assess on a large-scale quantitative basis. This is why our study and its forebears use proxies and shortcuts aiming at the presence of information. We have to keep in mind that what is present need not necessarily be helpful or truthful.

Consistently with previous research on health information quality (e.g.,[7,14,23]), the findings of the present study confirmed the average quality level of pro- and anti-vaccination information available online. This result could be taken as an indication that not much progress was made with regard to the quality of health related websites since the earlier studies were fielded. Moreover, according to the readability assessment tool applied (Flesch-Kincaid Grade Level), the majority of webpages reviewed were written in a demanding manner. Indeed, webpages' readability levels were higher than the average American reading level, which is set between 7th and 9th grade (see:[36]). This latter negative result, which is coherent with past research evaluating online health information (e.g.,[7,45]), emphasizes the potential shortcomings of the Web as a complementary source of health information, and especially as a medium to retrieve and exchange vaccination-related information. As advanced by Fagerlin, Wang and Ubel,[54] given the complex nature of health information, websites promoting vaccination have to carefully design their messages in a way that are in line not only with the target audience's needs and "prior knowledge but also with their capacity to process the information, such as numeracy and health literacy, as well as their preferences for how information is presented" (as cited in,[12] p. 3731).

Implications for Online Health Information Providers

The findings of this study, based on a large sample size (N =1'093), allow us to draw several conclusions that can be translated into advice to online health information providers on how to make vaccination-related webpages more accurate, complete, attractive, and easy-to-navigate.

First, it is evident that to be supportive, complementary, and beneficial for everyone, health information available on the Internet has to be written below the average reading level (between 7th and 9th grade level) (e.g.,[7,36]). The increasing availability of online readability assessment tools (e.g., opens the door to web editors to easily implement this first and relevant suggestion. Another way to assure understandability of the information provided might be the adoption of "Question and Answer" (or FAQ) formats.

Second, given that the frequency of inclusion of pictures related to vaccination preventable diseases' (VPDs) effects resulted to be low, future educational vaccination-related web sources, in order to stress the perceived severity of VPDs, might include more often visual representations. Another feature, which is now missing across webpages, and most likely represents a useful aid to consumers seeking online vaccination information, is the target audience's "vaccination recommendation schedule". Additionally, the large number of videos available on anti-vaccination websites might be balanced by pro-vaccination pages providing more visual recordings that enable users to understand in a vivid manner the positive effects of vaccinations, and the risk derived by contracting a vaccine-preventable disease (e.g., showing, in form of narrative, a sad mother which lost her son due to a VPD).[12] Moreover, in order to facilitate understandability of the contents provided, the application of videos/animations is especially suited for low health literate individuals.[55] Third, the so-called "post comment on this page" feature is still missing on a large portion of webpages being in favor to immunizations. This type of interactive service has the potential to increase users' involvement, and, consequently, the relative effect of messages on them[56,57] as well as attention, knowledge, perception of being socially supported, and ultimately positive health outcomes.[58,59]

Fourth, since secondary ethnic groups are present in many English-speaking countries, vaccination web sources might offer the selection of a further language (e.g., Spanish).

Limitations and Further Research

The present study has some limitations. First, as the nature of Internet information is purely dynamic (updating processes), the currency of the results gained from this content analysis may have been already challenged. Secondly, while some of the webpages were retrieved with the original Google search engine, the rest of them were obtained through two manipulated versions of the same engine. In particular, one engine was filtered (with a set of predefined keywords) to only search for vaccination-related links holding a negative tone, and the other to yield only pages from reputable sources (mixes of HONcode certified sites). Thus, some of the webpages reviewed might not have been retrieved by typical online end-users. However, the applied keyword strategy considered a set of numerous terms, which could have been easily used by consumers on a normal online information seeking session. The initial sample represented also another constraint for the results of the current study. Pro-vaccination webpages were obtained through the application of a filter manipulating the original Google search engine to yield sources of online health information certified by the Health On the Net Foundation (HON) and other highly credible online sources. This aspect has directly influenced our descriptive, and with minor impact, our comparative findings that have to be interpreted with caution. Last, it has to be noted that the final sample of 1093 web links included various webpages owned by the same website (e.g., webpages were around 100).

The findings of the present study can be extended to future comparative analyses in other fields7. In fact other qualitative categories apart from the core one ("general tone" of the webpage) can be employed in the same way as the principal construct. For instance, mean scores of the four aggregated indices can be calculated, among others, for the category labeled as "scope of information of the website", or again for the "domain name system" (DNS) construct.

Our content analysis did not code the specific type of vaccination (e.g., HPV) or related disease (e.g., mumps) presented by the different webpages reviewed. We recommend future studies evaluating vaccination-related web sources to include this data to ultimately conduct more sophisticated comparative analyses. Future studies are also necessary to build solid findings that both test the instrument's usability, and reconfirm the strong inter-rater agreement (k = .89) attributed to the present assessment tool.