The full content of Annals is available to subscribers

Subscribe/Learn More  >
Original Research |

Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine

Steven N. Goodman, MD, PhD; Jesse Berlin, ScD; Suzanne W. Fletcher, MD, MSc; and Robert H. Fletcher, MD, MSc
[+] Article, Author, and Disclosure Information

From Johns Hopkins University School of Medicine, Baltimore, Maryland; the University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania; Harvard Medical School, Boston, Massachusetts. Requests for Reprints: Steven Goodman, MD, PhD, Johns Hopkins University, Oncology Center, Division of Biostatistics, 550 North Broadway, Suite 1103, Baltimore, MD 21205. Acknowledgments: The following “expert assessors” are thanked for their contributions to this project: Wendy Adams, MD; Eric Bass, MD; Greta Bunin, PhD; Jeffrey Carson, MD; Neal Dawson, MD; John Dickenson, MD; Kay Dickersin, PhD; Jose Escarce, MD; Harold Feldman, MD; Daniel Ford, MD, MPH; Martha Gerrity, MD; Matthew Gillman, MD, PhD; Deborah Grady, MD; Alan Hillman, MD; Thomas Imperiale, MD; Sharon Inouye, MD; Jeane Grisso, MD; Walter Kernan, MD; Thomas Keyserling, MD; Michael Klag, MD; Kurt Kroenke, MD; Seth Landefeld, MD; Suzanne Landis, MD; Valerie Lawrence, MD; Richard Lofgren, MD; Peter Margolis, MD, PhD; Wayne McCormick, MD; Alfredo Morabia, MD, PhD; Ann Naftinger, MD; Roberta Ness, MD; Warren Newton, MD; Sandra Norman, PhD; Patrick O'Connor, MD; Michelle Petri, MD, MPH; John Philbrick, MD; Jennifer Pinto-Martin, PhD; Peter Pompei, MD; Bruce Psaty, MD; Jeffrey Silber, MD, PhD; Rebecca Silliman, MD; John Steiner, MD; Ramon Velez, MD; Paul Whelton, MD, MSc; Larry Wissow, MD, MPH. The authors also thank Ms. Torii Ransome and Ms. Grace Lobb for their efforts in coordinating this project and Mr. Andrew Langman for assisting with the design of the questionnaire.

Copyright ©2004 by the American College of Physicians

Ann Intern Med. 1994;121(1):11-21. doi:10.7326/0003-4819-121-1-199407010-00003
Text Size: A A A

Objective: To evaluate the effects of peer review and editing on manuscript quality.

Setting: Editorial offices of Annals of Internal Medicine.

Design: Masked before–after study.

Manuscripts: 111 consecutive original research manuscripts accepted for publication at Annals between March 1992 and March 1993.

Measurements: We used a manuscript quality assessment tool of 34 items to evaluate the quality of the research report, not the quality of the research itself. Each item was scored on a 1 to 5 scale. Forty-four expert assessors unaware of the design or aims of the study evaluated the manuscripts, with different persons evaluating the two versions of each manuscript (before and after the editorial process).

Results: 33 of the 34 items changed in the direction of improvement, with the largest improvements seen in the discussion of study limitations, generalizations, use of confidence intervals, and the tone of conclusions. Overall, the percentage of items scored three or more increased by an absolute 7.3% (95% CI, 3.3% to 11.3%) from a baseline of 75%. The average item score improved by 0.23 points (CI, 0.07 to 0.39) from a baseline mean of 3.5. Manuscripts rated in the bottom 50% showed two- to threefold larger improvements than those in the top 50%, after correction for regression to the mean.

Conclusions: Peer review and editing improve the quality of medical research reporting, particularly in those areas that readers rely on most heavily to decide on the importance and generalizability of the findings.


Grahic Jump Location
Figure 2.
The distribution of manuscript quality scores before and after revision.

The quality score displayed here is the percentage of items for each manuscript that was rated above 3 (out of 5).

Grahic Jump Location
Grahic Jump Location
Figure 1.
Item analysis of the study.nPP

The length of the gray bars in the left-hand bar graph represents the fraction of unrevised manuscripts in which the given item was rated three or above ( = 111). The diamonds are the corresponding scores of the revised versions. The black bars on the right-hand graph represent the change because of revision, that is, the difference between the after scores and the before scores (or distance in the left-hand graph between the end of the bar and the diamond). *  < 0.05, †  = 0.07.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Journal Club
Topic Collections
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.