The full content of Annals is available to subscribers

Subscribe/Learn More  >
Articles |

Identifying Adverse Events Caused by Medical Care: Degree of Physician Agreement in a Retrospective Chart Review

A. Russell Localio, JD, MPH, MS; Susan L. Weaver, MS; J. Richard Landis, PhD; Ann G. Lawthers, ScD; Troyen A. Brennan, MD, JD; Liesi Hebert, ScD; and Tonya J. Sharp, MS
[+] Article, Author, and Disclosure Information

From Pennsylvania State University College of Medicine, Hershey, Pennsylvania; Harvard School of Public Health, Boston, Massachusetts; and Rush University and Rush-Presbyterian-St. Luke's Medical Center, Chicago, Illinois. Grant Support: In part by grant R01 HS07067-01 from the Agency for Health Care Policy and Research. Requests for Reprints: A. Russell Localio, JD, MPH, MS, Center for Biostatistics and Epidemiology, Pennsylvania State University College of Medicine, PO Box 850, Hershey, PA 17033-0850. Current Author Addresses: Mr. Localio and Dr. Landis: Center for Biostatistics and Epidemiology, Pennsylvania State University College of Medicine, PO Box 850, Hershey, PA 17033-0850.

Copyright ©2004 by the American College of Physicians

Ann Intern Med. 1996;125(6):457-464. doi:10.7326/0003-4819-125-6-199609150-00005
Text Size: A A A

Objective: To 1) assess the degree of agreement among physicians on the cause of previously flagged adverse outcomes and 2) relate the findings to systems of quality assurance and performance assessment and proposals for no-fault compensation for medical injuries.

Design: Observational study of 7533 pairs of “structured implicit” reviews (subjective opinions based on guidelines) of medical records done by 127 physicians working independently.

Setting: Random sample of 51 inpatient facilities in New York State.

Patients: Random sample of inpatient medical records from the selected facilities.

Measurements: 1] Number of agreed-upon adverse events compared with the number of cases of extreme disagreement and 2) internally and indirectly standardized rates at which physician reviewers found adverse events (injuries to patients caused at least in part by medical management).

Results: In 12.9% of cases (971 of 7533), the two physicians in a pair had extreme disagreement about the occurrence of an adverse event. These cases outnumbered those in which both reviewers found an adverse event (10%; n = 757). Agreement was highest for wound infections and lowest for adverse events attributed to failure to diagnose or lack of therapy. The amount of experience the physicians had in reviewing records tended to increase the level of agreement. Even after standardization to the results of the entire sample, individual physicians' rates of finding at least slight evidence of an adverse event varied widely (range, 9.9% to 43.7%) (P < 0.001).

Conclusions: Structured implicit reviews produced disagreement on the causes of adverse patient outcomes. If systems of quality assurance, performance audits, or no-fault patient compensation are to succeed, methods for overcoming the common tendency toward disagreement among experts must be developed.


Grahic Jump Location
Figure 2.
Screening Criteria Implemented at Stage 1 Review by Nurses and Medical Records Administrators.

Judgments on adverse events by pairs of physician-reviewers and rate of agreement on occurrence of adverse events compared with extreme disagreement. If a = cases of extreme disagreement (one reviewer scored the outcome as 0 [no possible adverse event] and the other scored the case as 4, 5, or 6) and b = cases for which both reviewers found adverse events (both scored the case as 4, 5, or 6), then the reported rate of agreement = a/(a + b). Bars represent exact binomial 95% CIs. Numbers in parentheses are the population-weighted estimates of the number of cases in New York State in 1984 that are represented by the sampled cases reported in this figure.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Topic Collections
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.