Kaveh G. Shojania, MD; Margaret Sampson, MLIS; Mohammed T. Ansari, MBBS, MMedSc, MPhil; Jun Ji, MD, MHA; Steve Doucette, MSc; David Moher, PhD
Disclaimer: The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.
Acknowledgments: The authors thank Keith O'Rourke for statistical advice, Jessie McGowan and Tamara Rader for assistance with searches, and Alison Jennings for assistance with development of the meta-analytic worksheet. They also gratefully acknowledge Dr. David Atkins and the members of the technical advisory panel for the project funded by the Agency for Healthcare Research and Quality from which this work derives: Drs. Paul Shekelle, Evelyn Whitlock, Cynthia Mulrow, Doug Altman, Martin Eccles, and P.J. Devereaux.
Grant Support: By the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services (contract no. 290-02-0021). Dr. Shojania received additional salary support from the Government of Canada Research Chairs program. Dr. Moher is the recipient of a University of Ottawa Research Chair.
Potential Financial Conflicts of Interest: None disclosed.
Reproducible Research Statement: The data set is available to interested readers by contacting Dr. Shojania (e-mail, email@example.com); statistical code can be obtained from Mr. Doucette (e-mail, firstname.lastname@example.org).
Requests for Single Reprints: Kaveh G. Shojania, MD, The Ottawa Hospital–Civic Campus, 1053 Carling Avenue, Room C403, Box 693, Ottawa, Ontario K1Y 4E9, Canada; e-mail, email@example.com.
Current Author Addresses: Drs. Shojania and Ji: Ottawa Health Research Institute, 1053 Carling Avenue, Ottawa, Ontario K1Y 4E9, Canada.
Ms. Sampson, Mr. Ansari, and Dr. Moher: Children's Hospital of Eastern Ontario Research Institute, 401 Smyth Road, Ottawa, Ontario K1H 8L1, Canada.
Mr. Doucette: The Ottawa Hospital, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada.
Author Contributions: Conception and design: K.G. Shojania, M. Sampson, M.T. Ansari, D. Moher.
Analysis and interpretation of the data: K.G. Shojania, M. Sampson, M.T. Ansari, J. Ji, S. Doucette, D. Moher.
Drafting of the article: K.G. Shojania.
Critical revision of the article for important intellectual content: M. Sampson, M.T. Ansari, S. Doucette, D. Moher.
Final approval of the article: K.G. Shojania, M. Sampson, M.T. Ansari, D. Moher.
Provision of study materials or patients: M. Sampson.
Statistical expertise: K.G. Shojania, S. Doucette.
Obtaining of funding: M. Sampson, D. Moher.
Administrative, technical, or logistic support: D. Moher.
Collection and assembly of data: M. Sampson, M.T. Ansari, J.Ji.
Systematic reviews are often advocated as the best source of evidence to guide clinical decisions and health care policy, yet we know little about the extent to which they require updating.
To estimate the average time to changes in evidence that are sufficiently important to warrant updating systematic reviews.
Survival analysis of 100 quantitative systematic reviews.
Systematic reviews published from 1995 to 2005 and indexed in ACP Journal Club. Eligible reviews evaluated a specific drug or class of drug, device, or procedure and included only randomized or quasi-randomized, controlled trials.
Quantitative signals for updating were changes in statistical significance or relative changes in effect magnitude of at least 50% involving 1 of the primary outcomes of the original systematic review or any mortality outcome. Qualitative signals included substantial differences in characterizations of effectiveness, new information about harm, and caveats about the previously reported findings that would affect clinical decision making.
The cohort of 100 systematic reviews included a median of 13 studies and 2663 participants per review. A qualitative or quantitative signal for updating occurred for 57% of reviews (95% CI, 47% to 67%). Median duration of survival free of a signal for updating was 5.5 years (CI, 4.6 to 7.6 years). However, a signal occurred within 2 years for 23% of reviews and within 1 year for 15%. In 7%, a signal had already occurred at the time of publication. Only 4% of reviews had a signal within 1 year of the end of the reported search period; 11% had a signal within 2 years of the search. Shorter survival was associated with cardiovascular topics (hazard ratio, 2.70 [CI, 1.36 to 5.34]) and heterogeneity in the original review (hazard ratio, 2.15 [CI, 1.12 to 4.11]).
Judgments of the need for updating were made without involving content experts.
In a cohort of high-quality systematic reviews directly relevant to clinical practice, signals for updating occurred frequently and within a relatively short time.
Clinicians rely on systematic reviews for current, evidence-based information.
This survival analysis of 100 meta-analyses indexed in ACP Journal Club from 1995 to 2005 found that new evidence that substantively changed conclusions about the effectiveness or harms of therapies arose frequently and within relatively short time periods. The median survival time without substantive new evidence for the meta-analyses was 5.5 years. Significant new evidence was already available for 7% of the reviews at the time of publication and became available for 23% within 2 years.
Clinically important evidence that alters conclusions about the effectiveness and harms of treatments can accumulate rapidly.
Overall process for determining updating status.ACP Journal Club
Screening of potential systematic reviews for inclusion in cohort.
Table 1. Characteristics of the Cohort of 100 Systematic Reviews
Table 2. Examples of Quantitative and Qualitative Signals
Overall survival time (95% CI) free of signals for updating.
Table 3. Predictors of Signals for Updating in the Cohort of 100 Systematic Reviews
Survival of the original systematic review by clinical topic area.nn
Kaplan–Meier plot showing the effect on survival of increasing the total number of patients by more than a factor of 2.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis. Ann Intern Med. 2007;147:224–233. doi: 10.7326/0003-4819-147-4-200708210-00179
Download citation file:
Published: Ann Intern Med. 2007;147(4):224-233.
Cardiology, Education and Training, Healthcare Delivery and Policy.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use