Rebecca S. Lipner, PhD; Bradley G. Brossman, PhD; Kelli M. Samonte, PhD; Steven J. Durning, MD, PhD
Note: Drs. Lipner, Brossman, and Samonte had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Acknowledgment: The authors thank Bryn Herrschaft-Eckman, PhD, and Dwan King Bovell of the ABIM; Sharon Kemp of UpToDate; and Kimberly Currier Bloom of Pearson VUE for their administrative and technical work in supporting the implementation of the study. They also thank other staff at UpToDate and Pearson VUE.
Financial Support: Funds for incentives were provided by the ABIM Foundation. The ABIM provided support for the administrative and technical work and statistical analysis by allowing staff time to be used for this purpose.
Disclosures: The ABIM is pursuing use of external resources on its high-stakes examination as part of routine business for 2018 and beyond. UpToDate (which supplied the external resource for this study) is a potential vendor for that future work. Drs. Lipner, Brossman, and Samonte report grants from the ABIM Foundation during the conduct of the study and are employed by the ABIM. Dr. Durning was a research consultant for the ABIM during the conduct of the study. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M16-2843.
Editors' Disclosures: Christine Laine, MD, MPH, Editor in Chief, reports that she has no financial relationships or interests to disclose. Darren B. Taichman, MD, PhD, Executive Deputy Editor, reports that he has no financial relationships or interests to disclose. Cynthia D. Mulrow, MD, MSc, Senior Deputy Editor, reports that she has no relationships or interests to disclose. Deborah Cotton, MD, MPH, Deputy Editor, reports that she has no financial relationships or interest to disclose. Jaya K. Rao, MD, MHS, Deputy Editor, reports that she has stock holdings/options in Eli Lilly and Pfizer. Sankey V. Williams, MD, Deputy Editor, reports that he has no financial relationships or interests to disclose. Catharine B. Stack, PhD, MS, Deputy Editor for Statistics, reports that she has stock holdings in Pfizer and Johnson & Johnson.
Reproducible Research Statement:Study protocol: See the . Statistical code: Available from Dr. Lipner (e-mail, firstname.lastname@example.org). Data set: Not available.
Requests for Single Reprints: Rebecca S. Lipner, PhD, American Board of Internal Medicine, 510 Walnut Street, Suite 1700, Philadelphia, PA 19106; e-mail, email@example.com.
Current Author Addresses: Drs. Lipner, Brossman, and Samonte: American Board of Internal Medicine, 510 Walnut Street, Suite 1700, Philadelphia, PA 19106.
Dr. Durning: Uniformed Services University of the Health Sciences, Division of Health Professions Education (MED), 4301 Jones Bridge Road, Bethesda, MD 20814.
Author Contributions: Conception and design: R.S. Lipner, B.G. Brossman, K.M. Samonte, S.J. Durning.
Analysis and interpretation of the data: R.S. Lipner, B.G. Brossman, K.M. Samonte, S.J. Durning.
Drafting of the article: R.S. Lipner, B.G. Brossman, K.M. Samonte, S.J. Durning.
Critical revision for important intellectual content: R.S. Lipner, B.G. Brossman, K.M. Samonte, S.J. Durning.
Final approval of the article: R.S. Lipner, B.G. Brossman, K.M. Samonte, S.J. Durning.
Provision of study materials or patients: R.S. Lipner.
Statistical expertise: R.S. Lipner, B.G. Brossman, K.M. Samonte.
Obtaining of funding: R.S. Lipner.
Administrative, technical, or logistic support: R.S. Lipner, B.G. Brossman, K.M. Samonte.
Collection and assembly of data: R.S. Lipner, B.G. Brossman.
Electronic resources are increasingly used in medical practice. Their use during high-stakes certification examinations has been advocated by many experts, but whether doing so would affect the capacity to differentiate between high and low abilities is unknown.
To determine the effect of electronic resources on examination performance characteristics.
Randomized controlled trial.
Medical certification program.
825 physicians initially certified by the American Board of Internal Medicine (ABIM) who passed the Internal Medicine Certification examination or sat for the Internal Medicine Maintenance of Certification (IM-MOC) examination in 2012 to 2015.
Participants were randomly assigned to 1 of 4 conditions: closed book using typical or additional time, or open book (that is, UpToDate [Wolters Kluwer]) using typical or additional time. All participants took the same modified version of the IM-MOC examination.
Primary outcomes included item difficulty (how easy or difficult the question was), item discrimination (how well the question differentiated between high and low abilities), and average question response time. Secondary outcomes included examination dimensionality (that is, the number of factors measured) and test-taking strategy. Item response theory was used to calculate question characteristics. Analysis of variance compared differences among conditions.
Closed-book conditions took significantly less time than open-book conditions (mean, 79.2 seconds [95% CI, 78.5 to 79.9 seconds] vs. 110.3 seconds [CI, 109.2 to 111.4 seconds] per question). Mean discrimination was statistically significantly higher for open-book conditions (0.34 [CI, 0.32 to 0.35] vs. 0.39 [CI, 0.37 to 0.41] per question). A strong single dimension showed that the examination measured the same factor with or without the resource.
Only 1 electronic resource was evaluated.
Inclusion of an electronic resource with time constraints did not adversely affect test performance and did not change the specific skill or factor targeted by the examination. Further study on the effect of resource inclusion on other examinations is warranted.
Lipner RS, Brossman BG, Samonte KM, Durning SJ. Effect of Access to an Electronic Medical Resource on Performance Characteristics of a Certification Examination: A Randomized Controlled Trial. Ann Intern Med. ;167:302–310. doi: 10.7326/M16-2843
Download citation file:
Published: Ann Intern Med. 2017;167(5):302-310.
Published at www.annals.org on 15 August 2017
Education and Training.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use