Effectiveness of E-learning in Pharmacy Education

Sandra M. Salter, BPharm, PhD Candidate; Ajay Karia, BSc Hons Pharm; Frank M. Sanfilippo, PhD; Rhonda M. Clifford, PhD


Am J Pharm Educ. 2014;78(4) 

In This Article


This review is the first to comprehensively examine the effectiveness of e-learning in pharmacy education. Effectiveness is a complex, theoretical construct; here we used Kirkpatrick's hierarchy to guide the development of a detailed e-learning effectiveness map in pharmacy education. Our primary interest was effective learning. Eleven studies evaluated knowledge change. Ten studies conducted pre- and post-intervention tests only, and 1 study conducted an additional 2 follow-up tests.[39] All reported a significant improvement in knowledge after e-learning, although the magnitude of the gain varied widely (7% to 46%). This confirms that e-learning in pharmacy education is effective at increasing knowledge immediately after training. Additionally, in comparisons, e-learning was as effective as traditional learning and superior to no training. These results concur with the breadth of literature demonstrating effectiveness of e-learning in developing knowledge, in other professions.[8,9,16–22] However, long-term knowledge change as a result of e-learning remains unknown.

Attitudinal change (assessed as pre and post e-learning ratings) evaluated professional confidence in performing tasks and perceived knowledge. The evidence, while significant was realistically limited. In all cases assessment was subjective, gleaned through questionnaires with rating scales and survey instruments. Improvements in attitude were seen immediately after e-learning. However, the results need to be interpreted with caution: scale format data should not be analyzed on an item-by-item basis, and ordinal data is at risk of distortion when reported as mean scores, as occurred in 4 studies.[30,33,34,37,47,48] There was no evidence of long-term change in attitude.

We also primarily defined effectiveness as change in skills or practice. The evidence for effectiveness in these terms was limited and generally based on self-report data from small groups.[30,31,34,41] Only 1 study employed sufficient methodological rigor to objectively report a positive change in skills after e-learning.[35] To conduct objective skills or practice assessments is costly and time consuming, and requires greater dedication than objective knowledge assessments. However, the goal of quality education must be to improve skills and practice, and research should be directed to address this. There were no e-learning effectiveness studies for organizational change or patient benefit – the highest level in Kirkpatrick's hierarchy. Translational research is required to determine the benefits of e-learning at this level.

Our secondary aim was to assess effectiveness as reactions to e-learning programs. Effectiveness measures for reactions included perceived benefits of e-learning, relevance of the specific e-learning course, and e-learning functionality. Most pharmacists agree that e-learning formats stimulate interest, provide flexible alternatives to traditional methods, and are easy to use. There is limited evidence for acceptance of technology used in e-learning, although technology is central to the process. This may be because the Internet is so inherently a part of everyday life that the details of technology are overlooked in research. Poor recordings or difficult access can lead to bad learning experiences. Further, as students (as part of the millennial generation) embrace other e-learning opportunities such as social media applications or massive open online courses (MOOCs), continued evaluation of e-learning technology will be essential. Finally, courses were presented in a myriad of formats, and satisfaction with course design and educational content was generally high.[31,32,35,37,39,40,44–46]

Overall, the findings of these studies show that learners consider e-learning a highly acceptable instructional format in pharmacy education. However, we acknowledge the risk that ratings may have been subject to response bias and that respondents' impressions may have changed over time after completing the e-learning course. Opinions may be affected by external factors, especially in times of stress (eg, pharmacy students may score ratings differently after examinations compared to usual coursework); however, this is true for any instructional format. Finally, what we observed is missing from e-learning satisfaction research is the impression of the educator.

Our study has several limitations. We limited the eligibility criteria for inclusion in the study to those studies that reported evaluations of the effectiveness of e-learning in pharmacy education. Other research evaluating effectiveness alongside different constructs may have been overlooked. Although 2 reviewers independently abstracted the data, differences in study interpretation may have impacted the data obtained, as evidenced by the low to moderate agreement within some of the data extraction levels. Although overall quality was moderate, study methodological quality was generally low. Three particular flaws stood out: selection bias, lack of control groups, and lack of validated tools. Most studies were conducted within a narrow sampling frame, did not employ appropriate control groups, and used only partially validated or non-validated tools, thus limiting internal and external validity. We attempted to synthesize results for a group of studies that held only 2 commonalities: pharmacy and e-learning. Interventions, topics, duration, and setting were different for every study. However, while this may have affected combination of results, the fact that e-learning was effective in different environments may support generalizing these results. Further, we acknowledge that all included studies reported significant (and positive) effects, and that publication bias was likely to exist. Lastly, we synthesized the evidence for pharmacists and pharmacy students as one. We recognize each have distinct learning needs, motivations, and environments. As pharmacy students progress to pharmacists, learning styles may change. Future reviews should identify specific aspects of effective e-learning for each population.

In the context of the broader literature, our review adds e-learning as an effective instructional method in pharmacy education, to the evidence that it is effective for other health professions.[7,9,19,22] Individual e-learning programs should continue to be evaluated for effectiveness, not to answer the question of whether e-learning works in pharmacy, but to inform educators and decision makers that the program itself is effective. There are 2 key reasons why this matters. First, e-learning programs are often developed for large-scale distribution; thus, confidence that the program will effectively teach (often complex) pharmacy topics is essential. Second, e-learning programs may not always be subject to the same scrutiny that traditional programs undergo, especially those developed by smaller organizations specifically for a target audience.

Finally, 13 of the 17 studies reviewed evaluated more than 1 effectiveness measure, in some cases using multiple methods. Problems with reporting, methodology, and thus quality may stem from this multiple outcome approach, suggesting that effectiveness studies of e-learning in pharmacy education are trying to address too many questions at once. Now that we know e-learning is effective in the short-term, it may be more useful to see well-conducted research that reports the long-term effectiveness of e-learning in pharmacy education (defined by 1 or 2 measures only) rather than broad snapshots of immediate impact.