Comparison of Methods for Detecting Medication Errors in 36 Hospitals and Skilled-Nursing Facilities

Elizabeth A. Flynn, Kenneth N. Barker, Ginette A. Pepper, David W. Bates, and Robert L. Mikeal 


Am J Health Syst Pharm. 2002;59(5) 

In This Article


The literature was reviewed to find the best methods for detecting medication administration errors in U.S. hospitals and skilled-nursing facilities, comparing the methods' validity and cost-efficiency. Twelve methods were considered for inclusion in the present study:


  1. Directly observing medication administration,[3,9,17]

  2. Reviewing patients' charts,[18,19]

  3. Reviewing incident reports involving medication errors,[20]

  4. Attending medical rounds to listen for clues that an error has occurred,[21]

  5. Interviewing health care personnel to stimulate self-report,[18,19]

  6. Analyzing doses returned to the pharmacy,[22]

  7. Testing urine for evidence of omitted drugs and unauthorized drug administration,[23]

  8. Examining death certificates,[24]

  9. Attending nursing change-of-shift report,[25]

  10. Comparing medication administration record (MAR) with physicians' orders,[26]

  11. Performing computerized analysis to identify patients receiving target drugs that may be used to treat a medication error or to search for serum drug concentration orders that may indicate an overdose,[18] and

  12. Comparing drugs removed from an automated drug-dispensing device for patients with physicians' orders.[27]


A panel of six experts on medication errors considered these methods and recommended that incident report review, chart review, and direct observation be compared for their ability to detect medication administration errors. (The inclusion of prescribing errors was discussed and tried but found too difficult for the data collectors recruited in the time available.)

The accuracy of the three error detection methods was compared by using them to evaluate an identical sample of doses called the "comparison doses." The standard for comparison comprised the direct-observation results from all data collectors after the accuracy of each dose (in error or not in error) was confirmed independently by a research pharmacist. The efficiency and cost of each method and data collector type were determined by comparing the time required to examine each dose. The sample of doses studied was drawn from the medications administered or omitted in the stratified random sample of 36 facilities in Georgia and Colorado.

R.N.s, L.P.N.s, and pharmacy technicians were compared as data collectors. Pharmacists have been compared with R.N.s previously and were found to be capable of obtaining results that are not significantly different.[28] The effectiveness and efficiency of L.P.N.s and technicians as observers, chart reviewers, and incident report reviewers have not been compared with pharmacists' abilities.


Suspecting that differences in the medication-use process might exist between the East and West, and between hospitals accredited by the Joint Commission on Accreditation of Healthcare Organizations, nonaccredited hospitals, and skilled-nursing facilities that might influence the findings, we decided to use a stratified sample. A stratified random sample of 36 facilities was drawn from the Atlanta, Georgia, metropolitan statistical area (MSA) and the Denver-Boulder-Greeley, Colorado, consolidated MSA. Lists of Medicare- and Medicaid-certified hospitals and skilled-nursing facilities located in each MSA were provided by HCFA. Six primary and 18 back-up facilities were randomly selected for each of the 3 facility types in each state. Letters soliciting participation were mailed to the administrator. The first 18 sites in each MSA that qualified and agreed to participate were included in the sample (26 sites declined). All sites met the inclusion criteria of having an incident report system in place.

The population from which nonaccredited hospitals were randomly selected was expanded to include the entire states of Georgia and Colorado after we found that there were not six nonaccredited hospitals within each MSA that would agree to participate. A minimum bed-size requirement of 25 was established after we found that the census was unpredictable in hospitals with fewer than 25 beds (which at times did not have any inpatients).

The sampling unit comprised doses observed as given or omitted during a one- to four-day period on a nursing unit identified by nursing administration as having a high medication volume. If the facility had several high medication volume units, up to four were included in the sample. A different nursing unit was studied each day to enable the data collectors to work on a new sample of patients. Direct observation occurred during the morning medication administration session, when the greatest volume of doses were given, or until 50 doses were observed. Only staff nurses were observed.

The comparison doses were defined as the set of doses for which data were collected using all three methods. This set comprised the doses observed as given or omitted but was limited to those for which the chart review could be completed within the eight-hour workday of the data collector.


A medication error was defined as any discrepancy between the prescriber's interpretable medication order and what was administered to a patient. Definitions for the following categories of medication errors employed in this study appear in Appendix A: omission, unauthorized drug, wrong dose, extra dose, wrong route, wrong form, wrong technique, and wrong time. Doses were considered to have been given at the wrong time if they were administered more than 60 minutes before or after they were due, unless a drug was ordered to be administered with a meal, in which case a 30- minute window was allowed (based on HCFA guidelines[8]). The schedule of administration for routine doses as listed on the MAR at each site was consulted to determine whether a dose was administered within the acceptable time.

A dose was defined to be an opportunity for error (a measure long used as the basic unit of data in observational error studies).[3] A dose included any dose given plus any dose ordered but omitted. Any dose given was designated only as correct or incorrect (error or no error) when calculating the error rate. This definition prevented the error rate from exceeding 100%. Doses were defined, for the observation method, to include only those for which both the preparation and administration of the medication was witnessed by the observer.[3] For chart review, doses were defined as those recorded in the chart or scheduled to be administered in accordance with the prescriber's orders but omitted, plus as-needed medications for which there was evidence of administration on the MAR during the observation period.

In determining if an omission error had occurred, the data collectors always sought an explanation for the omission. For example, a patient's blood pressure might have been too low for the patient to receive antihypertensives, or a patient's heart rate was lower than the ordered limit to receive a certain medication. Data collectors were instructed to review a prescriber's orders twice when they detected an unauthorized-drug error to ensure that an order was not in the patient's chart and when an omission error was found to ensure that there was not an order to discontinue or temporarily stop administration of the drug.

Efficiency was defined as the time required to perform the tasks associated with each error detection method. The starting and ending times for each task were recorded by each data collector, and the total time spent on each patient was indicated for each task. The time spent on all aspects of data collection for individual patients (e.g., copying drug orders and reviewing a patient's chart) was documented. The time spent on tasks not associated with individual patients (e.g., introducing themselves to nurses, observing a medication administration session, and calculating an error rate) was divided and assigned equally for all patients. Nonproductive tasks, such as breaks and lunch, were excluded. Efficiency was measured in minutes per dose and calculated by dividing the time spent on each patient by the number of doses evaluated for that patient using the assigned method. Data collectors were paid for an eight-hour workday, regardless of how long it took them to complete the work. This removed the possible incentive to spend longer than necessary on each patient. A maximum of eight hours was spent on each nursing unit.

Data Collection and Error Detection Methods

Each data collector type was randomly assigned to use one of the three methods at each facility, so that each method was employed an equal number of times. For example, at the first site, the R.N. might have been assigned to use direct observation, the technician might have been assigned to review charts, and the L.P.N. might have been assigned to review incident reports. The data collectors worked independently of each other and were asked not to discuss their findings.

Incident Reports
Data collectors allowed two to three weeks to pass after the observation period before returning to analyze reports and classify any errors reported. Incident reports had to be filed at least 7 days before the first day of observation in hospitals, at least 30 days before the first observation day in skilled-nursing facilities, and no more than 7 days after the last observation day (to allow enough time to identify incidents and file reports for events occurring on the observation day). The time required to complete this process was recorded for each incident report. To assess the accuracy of each data collector's work, the research pharmacist reviewed photocopied incident reports, if available, or went to the facility to review the original documents.

Chart Review
Chart review was based on the research method described by Bates and colleagues[18] and modified in accordance with suggestions made by Bates for this study. A list of up to 10 patients who were directly observed during the medication administration session was provided to the chart reviewer after the observer completed her work. The charts were reviewed the day after the medication administration session to allow for errors that occurred during that time to exhibit effects on the patients. The following sections of each medical chart were evaluated by the data collectors: physician's orders, laboratory test results, physician's progress notes, nurse's notes, and the MAR. Trigger events that could result from a medication error included laboratory test results that exceeded normal values to a degree that it was likely that an error had occurred (e.g., blood glucose concentration of >400 mg/dL could indicate that an insulin dose was omitted). Forms were provided to data collectors to remind them of trigger events. A list of medication orders was extracted from each patient's chart to facilitate the comparison of each order with the MAR. Data collectors noted the use of "red flag" drugs, such as naloxone and flumazenil, as indicators that a medication error may have occurred. A complete list of these drugs appears in Appendix B.[29]

Retrospective study using chart review and incident report review was performed. The time period reviewed during chart review extended from the last day of observation to 7 days in hospitals and 30 days in skilled-nursing facilities (if the patient was there at the time). (Note that such retrospective data were not a source of comparison doses but were used to compare the efficiency and effectiveness of chart reviewers with incident report reviewers.)

Direct Observation
Direct observation required the data collector to accompany the nurse administering medications and observe the preparation and administration of each dose. The observer recorded exactly what the nurse did with the medication and witnessed the drug's administration to the patient. Data recorded included related procedures, such as measuring the patient's heart rate and giving medications with food.

Each observer attended the change-of-shift report on the nursing unit to meet the nursing staff and answer questions they had about the study. The observer then followed the first nurse to begin preparing medications and observed other nurses periodically to try to witness the administration of at least 50 doses. The observer made handwritten copies of the medication orders that were in the patient's chart and compared each dose observed with each prescriber's order. Uninterpretable orders (n = 6) were excluded from the study. Any deviations between the prescriber's order and what was observed were recorded as errors. After examining all of the doses witnessed, the observer then tallied all of the doses omitted. The medication error rate was calculated by dividing the number of errors by the sum of the number of doses given plus the number of omissions and then multiplying the result by 100.

Error Confirmation
To establish a standard set of "true errors" for comparison, the observer first finished the error determination and then gave the data to a research pharmacist. The researcher made a blinded, independent determination of errors by comparing each dose on the observer's medication administration session worksheet with the pharmacist's copy of the prescriber's orders. Each dose was then confirmed, classified as a false positive (i.e., data collector noted an error when the research pharmacist had evidence to the contrary), or classified as a false negative (i.e., data collector failed to detect an error that was subsequently found by the research pharmacist). The research pharmacist for the Colorado area also reviewed the data collected at the Georgia sites in order to address inconsistencies.

Recruiting and Training Data Collectors

To emulate procedures practical for most health care facilities, newspaper advertisements were placed to recruit R.N.s, L.P.N.s, and pharmacy technicians for training as data collectors. Applicants took a qualifying test to determine their base knowledge of medication and administration techniques. Two R.N.s, two L.P.N.s, and one pharmacy technician were hired in Denver, and two R.N.s, two L.P.N.s, and two pharmacy technicians were employed in Georgia.

A five-day training program was held in each city and included lectures about each data collection method, practice sessions using each method with sample patients, an interactive videotape about observation, practice sessions on nursing units in a hospital, and examinations. Additional practice observations and chart reviews were performed in hospitals and skilled-nursing facilities after the training was completed to develop proficiency in each method before the study began. One R.N. in Georgia withdrew from the project for personal reasons before completing the training program.

Two examinations about the observation method were given to evaluate each data collector's ability to detect errors. One exam determined intraobserver and interobserver reliability. The overall percentage of agreement on the error detection exam was 96%, with the agreement on each individual question ranging between 89% and 100%, indicating interrater reliability. Repeated measures analysis of variance found that the split-halves test scores were not significantly different within subjects, indicating intrarater reliability.

Assessment of Clinical Significance

The frequency of clinically significant errors was based on comparison doses and compared for each method. A clinically significant medication error was defined as a medication error with the potential for causing a patient discomfort or jeopardizing a patient's health and safety. A three-physician panel evaluated (by majority decision) each medication error detected for potential significance by considering the patient's condition, drug category involved, and frequency of the error. HCFA guidelines for determining significance were provided to the panel (Appendix C).[7] The research pharmacist reviewed each chart and provided information to the panel about each patient's condition, including sex, age, allergies, diseases, laboratory test results associated with a medication, "red flag" drugs ordered, and noteworthy physicians' and nurses' progress notes.

Statistical Methods

The number of errors detected by each method was compared by method and site. The rate of agreement between observation and chart review was determined, and the rate of false negatives and false positives was calculated.

Kappa values were calculated to assess the accuracy of the data collectors' information compared with that from the research pharmacist. The purpose of using the kappa statistic was to determine whether the rate of agreement with the errors confirmed by the research pharmacist (the standard) was greater than what would have been found by chance alone.

The cost of each data collector type and method was compared by using chi-square analysis after it was found that the cost per dose results were not normally distributed, based on the Shapiro-Wilk test result in SAS version 6.12 for each error detection method (SAS Inc., Cary, NC). Significance was set at p 0.05.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: