Challenges in Systematic Reviews That Assess Treatment Harms FREE

To be useful to decision makers, an evidence synthesis of a medical intervention should assess the balance of benefits and harms (1 - 2). Harms from medical interventions include adverse drug events (3) and complications following surgery or other procedures. An evidence synthesis that emphasizes only benefits is likely to lead to biased conclusions (4).

For most interventions, unfortunately, systematic reviews of harms are sparser than reviews of benefits. An analysis of more than 1000 systematic reviews or meta-analyses of health care interventions found that just 27% reviewed any harms data, and only 4% primarily focused on safety (5). Of 138 Cochrane systematic reviews of randomized trials with data from at least 4000 participants, only 18% included data on clearly defined harms (6). Difficulties in identifying studies of harms (7), poor quality of adverse event reporting in clinical trials (8), and uncertainty regarding whether systematic reviews of harms yield useful information (9) are some reasons systematic reviews have focused on evaluations of effectiveness.

Investigators performing systematic reviews of harms face challenges in finding and selecting data (1), rating the quality of harms reporting (9), and synthesizing (10) and displaying data from individual studies. In this article, we review these methodologic challenges and highlight examples of approaches to them from 96 Evidence-based Practice Center (EPC) evidence reports.

Most systematic reviews rely on searches of electronic databases of published articles and hand searches of relevant journals. Identifying important harms of treatment and quantifying the risk associated with them, however, often require a broader range of data sources. In addition, the types of studies included in an evidence synthesis may influence the quality or amount of evidence regarding harms.

In a sample of 60 meta-analyses that were published in 1995, 33 included only controlled trials (10). Properly designed and executed randomized, controlled trials are often considered the gold standard for evaluating efficacy because they minimize potential bias, but the quality and quantity of harms reporting in trials are often inadequate (11 - 14). Furthermore, surgical procedures and invasive diagnostic devices often become widely disseminated even though few or no randomized trial data are available (15 - 16). In these cases, relying on controlled trials for information about harms is impossible.

By contrast, randomized trials of drugs are numerous. Premarketing trials, conducted according to the requirements of the U.S. Food and Drug Administration (FDA) (15), assess the effects of drugs under ideal circumstances. These “efficacy trials” have limited ability, however, to assess adverse events or applicability to everyday practice. They may exclude patients at high risk for harms or may not be applicable to clinical practice for other reasons (16), may be too short to identify long-term or delayed harms, or may have sample sizes too small to detect uncommon events (1,17 - 20).

As required under the Freedom of Information Act, the FDA provides detailed information about the trials submitted in support of an application for approval of a new drug. Unfortunately, this information is often unavailable to the public at the time the drug is approved, and sometimes it is not disclosed until a year or more after the drug is marketed. When this information is available, the approval documents enable the reviewer to compare the results of published and unpublished trials and to compare the material published in journals to the material submitted to the FDA.

Unpublished clinical trials tend to report lower treatment effects than published trials (13,21). The impact of unpublished trials on assessments of harms has not been well studied, but a recent meta-analysis of antidepressants in children found that addition of data from unpublished trials changed risk–benefit profiles from favorable to unfavorable for several drugs (22). In addition, scientists affiliated with pharmaceutical companies sometimes publish meta-analyses of premarketing trials, some of which have not been fully published individually. These meta-analyses may focus on a single adverse effect or a narrow range of related adverse effects. The narrow focus can obscure the tradeoffs of benefits and harms for the drug in comparison to its competitors. For example, a meta-analysis of studies of venlafaxine reported similar overall withdrawal rates compared with other antidepressants but did not note that the rate of nausea and vomiting was significantly higher for venlafaxine (23). These meta-analyses also may exclude studies not performed by the manufacturer and may not describe the quality-related design characteristics of the individual trials. In the venlafaxine example, the FDA has not released the details of the unpublished premarketing studies, even though this drug was approved in 1993. The lack of information about these trials—how many were done, and whether efficacy, discontinuations, and adverse events were similar to or different from those associated with comparison drugs—make this type of meta-analysis an unreliable and potentially misleading source of information.

Journal publications may omit important information from trials because of space limitations. Although most useful for efficacy analysis (24), drug approval information—especially the clinical and statistical reviews prepared by FDA staff—frequently provides details about harms not included in the journal article. For example, for a major trial (Vioxx Gastrointestinal Outcomes Research Study, or VIGOR) of rofecoxib, an FDA statistical review made available to the public in 2001 contains 6 pages of analysis on the issue of cardiovascular risk (25), compared with 3 lines in the New England Journal of Medicine report (26). In fact, before the publication of VIGOR, most trials of rofecoxib did not report the rate of myocardial infarctions. One FDA analysis of trial data showed that the additional risk for cardiovascular events associated with rofecoxib did not appear until after 6 months of follow-up (27). This finding was of critical importance in evaluating the results of a major trial of celecoxib, the Celecoxib Long-term Arthritis Safety Study (CLASS). Findings from this study were published in the Journal of the American Medical Association as 6-month data; at 6 months, rates of serious gastrointestinal or cardiovascular events did not differ significantly between celecoxib and 2 comparators, diclofenac and ibuprofen (28). The article did not mention that some patients in the trial had been observed for longer than 6 months (29 - 30). In contrast, the FDA review explains the changes in the protocol for CLASS, compared the 6-month results to longer-term results, and explained the rationale for early termination (27).

Observational studies can be a useful supplement for systematically assessing harms, particularly when effectiveness trials are lacking (31 - 32). The term observational studies refers to a broad range of study designs, including case reports; retrospective analyses of large claims databases; population-based, longitudinal cohort studies; uncontrolled surgical series of patients receiving an intervention (33); and others (34 - 35). All have some potential for yielding useful information.

Well-controlled observational studies demonstrated the associations between maternal diethylstilbestrol use and vaginal adenocarcinoma in young women (36); angiotensin-converting enzyme inhibitors and scleroderma renal crisis; certain appetite suppressants and pulmonary hypertension; and aspirin use among children with influenza and the Reye syndrome (37). The traditional observational designs used in epidemiology—case–control and population-based cohort studies—are subject to confounding and biases that are encountered less commonly in randomized, controlled trials (10,38 - 39). However, those designs take stronger precautions against bias than other observational designs, and their strengths and weaknesses are well understood. Confounding by indication, for example, is usually not an issue with unexpected adverse drug events in these studies because such unpredictable outcomes are usually not associated with the indication for treatment (32,40).

Of 96 EPC evidence reports published between December 1998 and April 2004, 71 addressed health care interventions associated with potential harms (38). Of these, 11 did not assess harms. Two of the 11 evaluated surgical interventions, and the other 9 evaluated pharmacologic interventions or approaches to diagnostic testing and disease management. Of the 60 remaining EPC reports, 34 included observational studies of adverse events.

Thirteen of 16 EPC reports evaluating surgical, obstetric, dental, or other invasive interventions included observational studies of harms, compared with 21 of 44 that evaluated noninvasive interventions. Most EPC reports evaluating alternative medicine interventions included observational studies of harms. Two EPC reports specifically included observational studies to identify uncommon or rare adverse events (41 - 42); 1 included observational studies when there was insufficient evidence from clinical trials (43); and 1 included observational studies because of concerns about applicability of clinical trials to community practice (44). In other EPC reports, the investigators did not explicitly state how they decided whether to include observational studies of harms, although it appeared that these studies were more likely to be included when clinical trial data were lacking.

Although several studies have found that well-designed controlled observational studies and randomized trials can report similar estimates of effects (45 - 48), no study has systematically evaluated how frequently observational studies result in different, yet valid, conclusions about harms. One meta-analysis of serious gastrointestinal complications associated with nonsteroidal anti-inflammatory drugs (49) and an EPC report on complications from carotid endarterectomy (44) found that clinical trials reported higher risks for adverse events than observational studies. This could be due to poorer assessment of harms in the observational studies or that observational studies are more likely to be published if they report good results. An EPC report on management of cancer pain found that a sample of observational studies did not change conclusions drawn from randomized trials (43). In other EPC reports, it was not clear whether including observational studies changed assessments about harms. Similarly, systematic reviews by non-EPC investigators either did not assess the effects of including observational studies (9,50 - 53) or used observational studies only for areas where clinical trial data were lacking (54 - 55).

Large databases may provide useful information about harms. For example, Fowler and Wennberg and colleagues used Medicare claims data to identify patients undergoing prostate surgery and surveyed them to estimate the frequency of side effects in practice (56 - 57). They found that rates of sexual dysfunction, urinary incontinence, and urethral stricture were higher than reported in case series. Another study that compared Medicare data with data from randomized trials found a higher Medicare rate of acid-related upper gastrointestinal events in women receiving alendronate (58).

Pharmacoepidemiology is a developing science that uses large databases to study drug effects. Many pharmacoepidemiologic studies attempt to emulate 4 epidemiologic study designs: cohort studies, case–control studies, case–crossover studies, and case–time–control studies (34,59). However, databases used in pharmacoepidemiologic research sometimes use fewer, and often weaker, precautions against bias than prospectively designed epidemiologic databases, such as the Framingham Study and the Nurses' Health Study. While pharmacoepidemiologic studies may be very valuable for examining the frequency of uncommon adverse events, additional empirical research is needed to identify features that are associated with valid findings.

Because medical research has traditionally been based in academic centers, findings may not apply to most patients who receive their care in community settings. Data collected by practice-based research networks may provide better information about benefits and harms of health care interventions in everyday clinical practice (60). Practice-based research data sets are often richer in clinical detail than administrative databases, making it possible to identify and measure likely confounders with more confidence.

One of the largest and most well-known practice-based research networks is the United Kingdom–based General Practice Research Database (GPRD) (61). A recently published study of suicide risk associated with selective serotonin reuptake inhibitors based on GPRD data reported no clear association with increased risk (62). Although this finding is similar to those of meta-analyses of randomized trials (63 - 64), the GPRD analysis also suggests that the risks for suicide are not significantly higher in clinical practice, where patients may not be monitored as closely or be as highly selected as in the trials.

Compared with traditional observational studies, studies based on analyses of administrative or practice-based databases are often harder to find by using electronic searches because many such analyses are proprietary. In addition, administrative and practice-based databases often could address more questions than they do. The analyses that are published are determined largely by funders' interest in a particular question (for example, Is celecoxib safer than rofecoxib?) rather than the question that may be of interest to a systematic reviewer (for example, Is celecoxib safer than naproxen?). Because this can introduce a type of publication bias, systematic reviewers must be able to query databases (or their owners) to supplement published information on the questions a review addresses, rather than depending solely on questions posed by interested parties.

About 30% of the primary published literature on adverse drug events is in the form of case reports (65). Case reports can identify uncommon, unexpected, or long-term adverse drug events (18,40) that are often different from those detected in clinical trials (66). Thirteen of 18 confirmed and important adverse drug reactions first reported in 1963, for example, were initially identified by case reports (67). Of 548 new chemical entities approved by the FDA between 1975 and 1999, 56 (10.2%) subsequently received one or more prominent black-box safety warnings (n = 45) or were withdrawn from the market (n = 16) (68). Although the proportion of withdrawals due to case report data in the United States is not known, case reports were the primary source used to withdraw 18 of 22 drugs from the Spanish market (69).

The FDA receives about 280 000 reports of postmarketing adverse events annually and collects them into a database (70). Although pharmaceutical companies may perform high-quality analyses of such data, these analyses are not always made public in a timely fashion, as in the case of the withdrawn lipid-lowering drug cerivastatin (71).

In conducting systematic reviews of adverse events, it is always important to consider whether specific drugs are more likely to be harmful for specific populations. Compared with white persons, for example, African Americans have an increased risk for angioedema from angiotensin-receptor inhibitors (72). Systematic reviews should bring attention to studies that deliberately look at the risks and benefits of specific drugs in subgroups. In many situations, however, the risk in different populations can be difficult to address systematically because controlled trials do not analyze harmful effects in subgroups or exclude patients at higher risk for harms. Often, a comprehensive search for trials and for observational clinical studies proves to be inadequate for evaluating clinical outcomes of drugs in specific populations.

When clinical data on subpopulations are lacking, investigators should consider including pharmacodynamic and pharmacokinetic studies, even though such data do not always correlate with clinical effects. In the case of the lipid-lowering agent rosuvastatin, for example, the FDA required labeling indicating that drug levels are higher in Asian persons (73), although a recently published meta-analysis of trials submitted to the FDA found no differences in clinical adverse events according to ethnicity, sex, or age (74). One role of systematic reviews in these cases is to help distinguish concerns based on clinical data from what is based on pharmacologic properties or on other considerations. The uncertainty regarding the safety of rosuvastatin in Asian persons illustrates how choosing among drugs for specific populations is often an imprecise use of evidence. Although it may not be clear whether the “best drug in general” is the “best drug for everyone,” overstating or overemphasizing differences in kinetics could lead to more harm than good if a decision is made to deprive people of what may prove to be the best drug; thus, such data should be interpreted with caution.

Empirically validated criteria are available to judge the quality of randomized, controlled trials (13,75 - 76), although associations between quality measures and estimates of treatment effect (77 - 78) are not straightforward. Unique considerations suggest the need for a distinct set of criteria for judging harms reporting. A recent meta-analysis of rofecoxib, for example, found that having an external end point committee was associated with higher risks for myocardial infarction, but allocation concealment was not associated with differences in risk (55). Widely used criteria to assess the quality of observational studies (79 - 80) and randomized trials, however, were not designed specifically to assess the quality of harms assessment (81). Guidelines for evaluating adverse events in systematic reviews (1) have been published, but they do not give firm recommendations on how to assess the quality of included studies.

Because they lack randomization, observational studies should adhere to higher methodologic standards than randomized, controlled trials (1,10,37,39). This point is often overlooked in debates about the integrity of pharmacoepidemiologic data. Randomized, controlled trials are expected to have outcomes recorded by blinded personnel, and to include all participants who were randomly assigned in the analysis of results. Using blinded outcome assessors and using an inception cohort are at least as important in observational studies.

In addition to not being designed for evaluating quality of harms assessment, systematic reviews have found that quality rating instruments for observational studies varied greatly in scope, the number and types of items used, and developmental rigor, and concluded that further study is needed to determine which methodologic characteristics are associated with bias (82 - 84). Several systematic reviews have evaluated specific methodologic characteristics for their effects on estimates of harms, although the generalizability of their findings is unknown. They found that prospective or retrospective design (84 - 85), case–control compared with cohort studies (49,55), and smaller compared with larger case series (84) had no clear effect on estimates of harms. On the other hand, a recent meta-analysis of observational studies of naproxen found that estimates of cardiovascular risk were lower in studies sponsored by the manufacturer of the competing drug rofecoxib (55).

Assessing the validity of case reports can be particularly difficult because of difficulties in establishing causality. Modeling, however, suggests that more than 1 to 3 spontaneously reported cases of uncommon or rare adverse events is very unlikely to be coincidental (86). Of 47 case reports published in 1963 in 4 major general medical journals, 35 subsequently proved to be clearly correct (87).

Four EPC reports (41 - 42,88 - 89) used criteria to assess adverse drug events reported in case reports and series that included considerations of temporal relationship, lack of alternative causes, presence of toxic concentrations of the drug, response to discontinuation, dose–response relationship, and response to rechallenge. Other disease-specific (90 - 91) and non–disease-specific (92 - 93) methods for assessing the probability of adverse drug reactions from case reports have also been published. These methods have been validated by using expert opinion (90,92,94) or positivity on rechallenge (95) as the gold standard. Guidelines for improving the reporting of suspected adverse drug events in case reports have recently been proposed (96). Of the 19 recommended items, the median number mentioned in 35 reports of 48 patients published in BMJ was only 9 (range, 5 to 12), although effects of missing information on the validity of case reports have not been studied.

Studies of surgical interventions are often uncontrolled series and frequently do not meet standards for accurate and comprehensive reporting of complications (97). One EPC report created but did not empirically validate a 4-grade system to rate the quality of surgical series; this system incorporated several methodologic considerations, such as the number of centers, prospective or retrospective design, and use of intention-to-treat analysis (98). In our EPC report on carotid endarterectomy, we developed and empirically tested an 8-criteria quality-rating instrument for assessing harms reporting from randomized, controlled trials; cohort studies; and uncontrolled surgical series (44). It incorporated factors potentially associated with more rigorous adverse event assessment and assigned an overall quality score and rating (Table 1). Several of these criteria are similar to those proposed in recent guidelines to improve reporting of harms in randomized trials (8).

The quality-rating instrument was pilot tested on 47 studies of carotid endarterectomy. Univariate analyses found pooled rates of stroke or death of 3.8% in poor-quality studies (95% CI, 2.7% to 5.2%), 6.4% in fair-quality studies (CI, 4.5% to 8.7%), and 6.8% in good-quality studies (CI, 4.6% to 9.5%). As was found in an earlier systematic review (85), independent assessment was associated with higher complication rates. For 6 of the 7 other individual quality-ratings criteria in our quality-ratings tool (follow-up was reported as complete in all studies and could not be assessed), meeting the criterion adequately was also associated with higher complication rates.

Our quality-ratings instrument was tested only on a set of studies for 1 intervention, and analyses did not control for other patient or intervention factors (such as skill or experience of surgeon) that could affect complication rates. Nonetheless, we are aware of no other studies in which methodologic shortcomings identified by a quality-ratings tool for randomized trials and observational studies of surgical complications were associated with lower adverse event rates.

Although the need to incorporate, synthesize, and weigh data from different types of studies for systematic reviews of harms has become widely recognized (31,99 - 101), methods to combine data from different sources are just starting to be developed (102). Because confounding and selection bias can distort findings from observational studies, it is especially important that researchers who include such studies avoid inappropriate statistical combination of data, carefully describe the characteristics and quality of included studies (103), and thoroughly explore potential sources of heterogeneity (10,104 - 105).

An approach proposed by the Grades of Recommendation Assessment, Development and Evaluation (GRADE) Working Group attempts to balance the need to be concise with full and transparent consideration of all important issues. This group recommends that investigators display important factors related to quality assessment (study design, quality, consistency, directness, and other modifying factors) and results (number of patients, effect size, quality, and importance) in a summary table (2). One EPC developed summary tables for evaluation of surgical complications from adrenalectomy that efficiently convey information about the number and types of studies, quality assessment, and results that could be used as a template for systematic reviews of surgical complications (Table 2) (98). Another EPC report developed summary tables to display the number and type of studies and estimate the magnitude of the association for serious adverse antihypertensive drug reactions during pregnancy (Table 3) (106). Other examples of methods for summarizing results from different data sources include Forrest plots of effect size (51) or pooled estimates of risk stratified by study design (49), and summary tables displaying assessments of the likelihood of bias and generalizability of results for each included study (53). A recently published meta-analysis juxtaposed clinical trials of rofecoxib and observational studies of naproxen (trials were not available) to evaluate and compare risks for myocardial infarction (55). These examples all illustrate potential complexities associated with interpreting and synthesizing results from systematic reviews that include different data sources. Until more is known about the methods for performing such evidence syntheses, oversimplification of findings could do more harm than good (10).

Better data about harms are needed to conduct balanced systematic reviews. Systematic reviewers often focus on analyzing data from published clinical trials. Information from a broader range of sources, however, may help fill in gaps or provide a more comprehensive look at harms (19,31,40).

Additional research is needed to empirically determine the impact of including data from different sources on assessments of harms. Research on optimal strategies for identifying studies for systematic reviews has focused on clinical trials, and similar research on efficient identification of other published and unpublished studies of harms is needed (7,107). Further development and testing of criteria for rating the quality of harms reporting will help facilitate judgments of the validity of included studies and conclusions of systematic reviews (9,13,108).

It is encouraging that attention to methodologic issues associated with conducting systematic evidence reviews on harms is increasing (99). For now, investigators should consider several key points when conducting systematic reviews of harms (Table 4): Investigators should explicitly state and explain the rationale for any decision to exclude specific data sources. Evidence syntheses evaluating associations between interventions and rare adverse drug events, for example, should strongly consider including observational studies, as randomized, controlled trials are unlikely to identify such events (3,109). For assessments of adverse drug events, FDA documents may also provide important information not available in journal publications. In addition, investigators should specifically state what criteria are being used to evaluate the quality of studies of harms, and consider validating the quality ratings criteria used. This would help advance the science of performing systematic reviews of harms. Finally, investigators should clearly display their results (1 - 2,110), indicating the populations addressed by the included studies and the applicability of the results to other populations (45). They should avoid inappropriately combining data, and thoroughly investigate heterogeneity to promote insight into areas of uncertainty and future research needs (10,111).