MOC: Open-Book Format Passes Muster, but Does It Matter?

Bridget M. Kuehn

August 14, 2017

Allowing physicians to use an online resource during maintenance of certification (MOC) exams does not hamper the ability of the tests to discriminate between more and less skilled physicians, a study found. In fact, more skilled physicians received a greater boost from the open-book format than lower-performing colleagues.

However, at least one expert wonders whether this addresses the most important issues surrounding MOC.

Rebecca S. Lipner, PhD, senior vice president of assessment and research, the American Board of Internal Medicine (ABIM), Philadelphia, Pennsylvania, and colleagues report their findings on the quality of open book testing in an article published online August 14 in the Annals of Internal Medicine.

The ABIM MOC program has drawn much fire from physicians for being too costly, too time-consuming, and a poor way to measure the quality of care physicians provide. Legislative efforts are underway in several states to prevent using MOC exams as a requirement to maintain insurance or hospital credentials.

"One complaint expressed by physicians is that answering questions on a closed-book examination does not reflect the way they answer questions while caring for patients, when they have access to a wide variety of informational resources," Steven Weinberger, MD, executive vice president and chief executive officer of the American College of Physicians, explains in an accompanying editorial.

The ABIM-funded study assessed whether allowing physicians access to an online resource would affect test performance. The study included 825 physicians who passed the Internal Medicine Certification exam or sat for the Internal Medicine MOC, including some who failed the MOC. The study randomly assigned participants to complete either a closed- or open-book MOC in 2 hours or to complete a closed- or open-book MOC in 2 hours and 15 minutes. Physicians in the open-book MOC group had access to the online resource UpToDate during the exam.

As expected, the open-book MOCs took physicians longer to complete. The open-book format was better than the closed-book format at discriminating more skilled physicians from less skilled ones (mean discrimination, 0.39 [95% confidence interval, 0.37 - 0.41] vs mean discrimination, 0.34 [95% confidence interval, 0.32 - 0.35] per question). More skilled physicians seemed to get more of an edge when taking the open-book versions of the test than less able physicians.

"In this study, we demonstrated that introducing electronic resources into test taking is feasible, making the examination more authentic to practice and not reducing its discriminatory power," the researchers write. "As a result, the ABIM will begin allowing access to an electronic resource for some MOC programs."

The authors caution that study participants might not have been as motivated as they would have been taking a real MOC with career consequences. The results also may not be generalizable to other specialties, they warn.

The results seem to allay fears that an open-book test would be less valid than a closed-book one, Dr Weinberger writes. But he cautions it is unclear how physicians would feel about an open-book option.

"Looking up information may decrease anxiety for some physicians, but it also might increase anxiety for others who are not used to taking a time-limited, open-book examination," Dr Weinberger explains.

A survey completed by 760 (92% response rate) of the 825 study participants found that 90% of the physicians use UpToDate as a resource in clinical practice, and 84% would be comfortable using it during an exam. More than three quarters said access to this resource would reduce their test anxiety, but only 43% say they would spend less time preparing for an open-book exam.

In addition, Dr Weinberger notes that open- vs closed-book exam is not the biggest issue facing the ABIM and MOC. "I believe that an ideal MOC process should identify knowledge gaps that are important and have some relationship to a physician's practice, allow him or her to close those knowledge gaps, and then demonstrate that the gaps have been closed," he writes. "This puts the priority for MOC reform on its relevance and value to both patients and physicians, rather than on such outcomes as the ability of a question to discriminate between high- and low-performing examinees."

He urges the ABIM to continue to work with physician organizations such as the American College of Physicians to make the MOC process more valuable and less burdensome.

"The ABIM should work collaboratively with those organizations to ensure that MOC is not just a hoop to jump through, but a process that ultimately improves the knowledge and skills of physicians and the quality of the health care they provide," he writes.

The ABIM funded the study and Dr Lipner and three coauthors report being employed by ABIM as staff or consultants. The company UpToDate is a possible vendor of online physician resources for the ABIM open-book examinations. Dr Weinberger reports being an employee of the American College of Physicians and has received royalties from UpToDate.

Ann Intern Med. Published online August 14, 2017. Article abstract, Editorial extract

For more news, join us on Facebook and Twitter


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.