Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological Rigor FREE

Compared with other study designs, well-done randomized trials provide the most valid estimate of the benefits of health interventions because they minimize bias. Systematic reviews of randomized trials identify all studies that have addressed a particular question, and meta-analyses combine the results by using methods that minimize bias (1). Many consider systematic reviews to be the best source of information for making clinical and policy decisions.

Such groups as the Cochrane Collaboration have established standards for the conduct and reporting of systematic reviews (2). In this issue, Shojania and colleagues (3) extend this tradition of subjecting research methods to close scrutiny. They describe how quickly the conclusions of 100 systematic reviews published in ACP Journal Club became out of date because of newly completed randomized trials that changed either the statistical significance or the magnitude of the summary treatment effect when they were added to the original review. Fifty percent of reviews were out of date within 5.5 years after publication and 23% were out of date within 2 years. Not surprisingly, the results were most likely to change for cardiovascular interventions (presumably because of the large number of trials in cardiovascular medicine) or if the trials in the original review were heterogeneous.

Unfortunately, Shojania and colleagues' findings do not allow authors to accurately identify the time at which their specific systematic review will be out of date. Because outdated reviews can be misleading, we believe that researchers should update a literature search annually to identify studies that might change the results of the original review, unless a search of clinical trial registries assures them that no randomized trials addressing the topic of the review are ongoing. Because the updating search should use the same search strategy as the original review, updating should take relatively little time.

In the past 2 decades, studies like that of Shojania and colleagues have improved the methodological quality of systematic reviews. We suggest that the time has come to devote similar energy to increasing the likelihood that the content and format of reviews are useful to a variety of decision makers. Despite advances in the conduct and reporting of systematic reviews, current evidence suggests that they are used less frequently by clinicians and policymakers than one might think. Patients use them even less frequently. A systematic review of the information-seeking behavior of physicians found that textbooks (many of which do not rely on evidence from systematic reviews) are still the most frequently consulted source of information, followed by advice from colleagues (4). Similarly, nurses and other health professionals seem to refer to evidence from systematic reviews infrequently in decision making (5 - 6). Analyses of selected policymaking processes in Canada and at the World Health Organization found that evidence from systematic reviews was used infrequently (7 - 8).

Given that systematic reviews of randomized trials are less susceptible to bias than the opinions of experts and observational data, why do people fail to use them when making policy, clinical, or personal decisions? In trying to answer these questions, we consider the clinical relevance of the questions addressed by the systematic reviews, the format of systematic reviews, and the failure of authors to place their findings in a clinical context.

Most systematic reviews address highly specific questions that interest the author. Authors seldom consult with policymakers, clinicians, patients, or health care managers about what's important to them. Clinicians often struggle with broad questions, such as, “What's the most effective treatment for insomnia?”, rather than the narrow questions that researchers prefer, such as, “What's the evidence that benzodiazepines are effective for the treatment of insomnia?” Also, many important clinical questions are not addressed by systematic reviews. For example, a review of the Cochrane Collaboration databases for “dizziness” and “syncope” yielded no completed reviews of the management or investigation of these common symptoms.

Systematic reviews are entirely dependent on the relevance of the randomized trials that have been conducted. Except for relatively rare pragmatic trials, the trials that most systematic reviews comprise involve highly selected patients who receive care from highly selected physicians, which in some circumstances raises important questions about their generalizability. (Actually, we feel that people sometimes worry too much about generalizablity. Clinical trials will never be done in every conceivable group of patients; therefore, clinicians must learn the art of applicability—that is, asking, “Is our patient really so different from patients included in the trials that we can't apply the results?”). Randomized trials generally underreport adverse events (9) and often do not study patients for a sufficient period to detect important side effects. These shortcomings can make them, and therefore systematic reviews, insufficient to drive clinical practice on their own.

Systematic reviews are often off-putting to clinical readers. To meet the criteria established for the reporting of systematic reviews, most reviews are lengthy (Cochrane reviews are often longer than 30 pages), appear complicated to those not trained in systematic reviews, and take a long time to read and appraise. Thus, it is not surprising that clinicians use them infrequently and prefer more user-friendly formats for accessing evidence.

Authors of systematic reviews tend to focus on documenting methodological rigor and fail to describe the policy or clinical context. Therefore, systematic reviews often do not provide crucial information for policymakers (10). Glasziou and Shepperd (11) found that fewer than 15% of systematic reviews published in Evidence-Based Medicine provide sufficient information about the intervention to allow clinicians or policymakers to implement it. Policymakers are interested in the governance, financial, and health care delivery strategies that are most likely to achieve their particular goals. Systematic reviews of studies that address these issues exist (see the Cochrane Effective Practice and Organization of Care group at http://www.epoc.uottawa.ca/), but the body of evidence is still quite small. In addition, systematic reviews generally do not consider cost-effectiveness or budget impact, which are crucial for policymakers.

While recognizing these shortcomings of existing systematic reviews, how can we encourage clinicians and policymakers to use them? First, we could provide more exposure to systematic reviews (which, despite their length and apparent complexity, are relatively easy to interpret) during clinical training. Second, we could encourage more interaction among researchers, policymakers, and frontline clinicians, so that trialists might design pragmatic studies aimed at answering questions of interest to policymakers and clinicians. Third, policymakers must be more willing to commission randomized trials of policy-relevant questions, which can then be incorporated into systematic reviews. Fourth, the authors of systematic reviews should provide more contextual information or refer readers to easy-to-find articles that contain this information. Finally, authors must develop user-friendly formats so that busy clinicians and health care managers can use systematic reviews in real-time decision making. We suggest making systematic reviews available in 3 formats: 20-second, 2-minute, and 2-hour versions. The 20-second version is a 1-paragraph to 1-page bottom-line summary. The 2-minute version is a 2- to 3-page summary that focuses on details of validity and applicability (with sufficient information to facilitate implementation of the evidence). The 2-hour version contains all of the relevant methodological and contextual details.

Because systematic reviews are comprehensive, they are undoubtedly a far more reliable basis for decision making than the traditional review article, with its selective citation of articles and the nonsystematic opinion of 1 or 2 experts. However, the traditional review has its virtues. The expert reviewers often go beyond the evidence from randomized trials to discuss issues that are of crucial importance to clinicians and policymakers. Moreover, the experts are usually willing to make recommendations in circumstances in which the evidence base is poor; in contrast, systematic reviews tend to call for more studies, and more than half of completed Cochrane reviews state that the evidence in support of the intervention is limited or poor (Glasziou P. Personal communication). We urgently need a new type of review. It would combine the scientific rigor of systematic reviews with the clinically nuanced contextualization and opinion of traditional review articles while clearly distinguishing between evidence and opinion (12).

Andreas Laupacis, MD

Keenan Research Centre, Li Ka Shing Knowledge Institute of St. Michael's Hospital

Toronto, Ontario M5B 1W8, Canada

Sharon Straus, MD

University of Calgary

Calgary, Alberta T2N 2T9, Canada