The full content of Annals is available to subscribers

Subscribe/Learn More  >
Academia and the Profession |

Assessing the Generalizability of Prognostic Information

Amy C. Justice, MD, PhD; Kenneth E. Covinsky, MD, MPH; and Jesse A. Berlin, ScD
[+] Article, Author, and Disclosure Information

From Pittsburgh Veterans Affairs Health Care System and University of Pittsburgh, Pittsburgh, Pennsylvania; University of California, San Francisco, and San Francisco Veterans Affairs Medical Center, San Francisco, California; and School of Medicine and Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, Pennsylvania.

Ann Intern Med. 1999;130(6):515-524. doi:10.7326/0003-4819-130-6-199903160-00016
Text Size: A A A

Physicians are often asked to make prognostic assessments but often worry that their assessments will prove inaccurate. Prognostic systems were developed to enhance the accuracy of such assessments. This paper describes an approach for evaluating prognostic systems based on the accuracy (calibration and discrimination) and generalizability (reproducibility and transportability) of the system's predictions. Reproducibility is the ability to produce accurate predictions among patients not included in the development of the system but from the same population. Transportability is the ability to produce accurate predictions among patients drawn from a different but plausibly related population. On the basis of the observation that the generalizability of a prognostic system is commonly limited to a single historical period, geographic location, methodologic approach, disease spectrum, or follow-up interval, we describe a working hierarchy of the cumulative generalizability of prognostic systems.

This approach is illustrated in a structured review of the Dukes and Jass staging systems for colon and rectal cancer and applied to a young man with colon cancer. Because it treats the development of the system as a “black box” and evaluates only the performance of the predictions, the approach can be applied to any system that generates predicted probabilities. Although the Dukes and Jass staging systems are discrete, the approach can also be applied to systems that generate continuous predictions and, with some modification, to systems that predict over multiple time periods. Like any scientific hypothesis, the generalizability of a prognostic system is established by being tested and being found accurate across increasingly diverse settings. The more numerous and diverse the settings in which the system is tested and found accurate, the more likely it will generalize to an untested setting.


Grahic Jump Location
Figure 1.
Calibration of the Dukes staging system.Table 3

Points on lines correspond to the 5-year mortality rate by Dukes stage for each of five validation studies and the original development report ( ). To show point estimates for stage B1, B2, C1, and C2 disease when reported, point estimates for stage B and C were repeated in studies in which mortality rates were not reported by substages within stages B and C.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Calibration of the Jass staging system.Table 3

Points on lines correspond to the 5-year mortality rate by Jass stage for each of five validation studies and the original (development) report ( ). Because validation 2 reported no point estimate for stage III disease, a midpoint between stage II and stage III was imputed by averaging the difference between the stages.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.