Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care FREE

Quantitative systematic reviews have traditionally focused on estimating the effectiveness of specific therapeutic interventions, assessing the accuracy of diagnostic tests, and, to a lesser extent, quantifying epidemiologic relationships (1–3). Such meta-analyses lend themselves to clear formulation of problems with research questions such as the following: Does a particular intervention increase survival in patients with a particular diagnosis? What are the sensitivity and specificity of a given diagnostic test? What is the probability that patients with a particular risk factor are likely to develop a given disease? The literature searches, development of inclusion and exclusion criteria, and selection of quantitative methods for combining evidence follow readily from such research questions.

By contrast, systematic reviews of some important policy issues, such as those related to the delivery, organization, and financing of health care, present 3 key challenges to established methods of systematic reviews (4). First, such reviews may require considerable effort at the problem formulation stage to narrow the scope of the question to one that can feasibly be addressed and to define appropriate inclusion and exclusion criteria that facilitate literature searches. They may address research questions such as the following: What is the relative effect of private for-profit versus private not-for-profit delivery of care on patient outcomes? Which strategies are the most effective for promoting preventive care? Second, they may not lend themselves to straightforward literature searches, as some of the literature addressing these complex questions falls outside the traditional medical literature and may be difficult to retrieve (5). For example, the relevant literature may consist principally of institutional reports, white papers, and other “grey literature,” which are often not well cataloged in electronic databases. Third, the included articles may be highly heterogeneous with respect to study design (including the key reported outcomes) and quality. This can lead to difficulties appraising and interpreting the evidence and can severely limit opportunities to synthesize the data quantitatively, as well as qualitatively.

There is increasing interest in using high-quality systematic reviews of topics related to the delivery, organization, and financing of health care. For example, economic evaluations of technologies are increasingly being used to inform coverage decisions by organizations such as the National Institute for Clinical Excellence (NICE) in the United Kingdom and the Australian Government's Pharmaceutical Benefits Advisory Committee (PBAC) (6). Several research groups are developing expertise in performing high-quality systematic reviews on these topics. Notably, within the Cochrane Collaboration, the Cochrane Effective Practice and Organisation of Care Group has performed numerous reviews on related topics (7–8).

The Agency for Healthcare Research and Quality (AHRQ) has increasingly asked its Evidence-based Practice Centers to synthesize evidence on complex, policy-related topics that do not lend themselves to traditional methods of evidence synthesis. In this review, we discuss innovative methods used by Evidence-based Practice Centers and other researchers to synthesize evidence related to the study of the organization, delivery, and financing of health care. We reviewed 94 Evidence-based Practice Center reviews and other published systematic reviews to identify reviews that specified their method for problem formulation, discussed the challenges of using nontraditional or medical literatures, and synthesized broad-based literature relevant to health care delivery, organization, or financing. In this supplement article, we review approaches used in these systematic reviews for problem formulation, data collection, and analysis and interpretation.

We found 13 published reviews of broad topics related to the delivery, organization, and financing of health care that specified their methods for problem formulation, data collection, and data analysis and interpretation. The topics meeting these criteria broadly included patient safety (9–11), quality improvement (12–13), bioterrorism preparedness (14–15), health promotion (16), health care financing (17–19), telemedicine (20), and health literacy (21). We present the results of our review in the Appendix Table.

In the sections that follow, we summarize innovative approaches to solving the unusual methodologic challenges characteristic of these types of evidence synthesis topics. We have organized these sections according to 3 of the stages of evidence synthesis from the Cooper model of research synthesis: problem formulation, data collection, and data analysis and interpretation (22).

The problem formulation stage has 3 research objectives: development of the key research questions, construction of the definitions of the key concepts that distinguish relevant from irrelevant studies, and the establishment of inclusion and exclusion criteria for the review (22). In general, given the complexity and breadth of the topics addressed in the included reviews, the authors often described the use of conceptual frameworks for problem formulation. Specifically, they often adapted theoretical frameworks from related fields to develop conceptual models that they then used to specify key research questions and direct their literature searches.

The evidence related to health care delivery, organization, and financing often describes complex systems processes. Without clearly stated definitions of the processes of interest, it can be difficult to identify the research questions, develop appropriate literature searches, and formulate inclusion and exclusion criteria. Thus, for some projects, the essential task of the problem formulation stage is the defining of relevant concepts. For example, in the review “Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators,” the authors evaluated patient safety indicators developed from administrative data (9). They began by developing a set of standardized definitions of 9 key concepts relevant to their topic, including quality of health care, quality indicators, patient safety, patient safety indicators, medical error, and adverse events (9). They used these definitions to develop a list of potential indicators of medical errors, patient safety, or potentially preventable complications (9). The authors then invited 21 professional clinical organizations and 16 surgical subspecialty organizations to submit nominations of experts to evaluate potential patient safety indicators. An expanded expert advisory group of 76 clinicians (divided into 11 panels) reviewed and evaluated indicators according to a method adapted from the RAND/UCLA Appropriateness Method (23). This consisted of an initial independent assessment of each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists, followed by a final independent assessment by clinician panelists using the same questionnaire (9). The panel process served to refine definitions of some indicators, add new measures, and dismiss indicators with major concerns from further consideration (9). Finally, a subset of 34 of the most promising indicators was developed and additional literature was sought for studies describing these indicators. This unique use of a series of expert advisory panels produced inclusion criteria with considerable face validity given the breath and depth of the clinical specialists involved in the process.

The authors of the review “Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems” used an approach based on decision theory for defining the key research questions of their study (14). Their review evaluated whether existing information technologies and decision support systems would be useful for clinicians and public health officials preparing for and responding to bioterrorism (14). The authors were faced with a broad topic that involved highly heterogeneous technologies, deployed in different settings, for different users, and for different aspects of bioterrorism preparedness planning and responses. To specify the key research questions of their study, the authors used a decision analytic framework to identify the key decisions of clinicians and public health officials preparing for and responding to bioterrorism (Figures 1 and 2) (24). For example, they identified 4 key decisions of clinicians during a bioterrorism response: diagnostic decisions, management decisions (for example, how to select antibiotics for a given patient), prevention decisions (for example, which patients to place in isolation), and decisions about when to report suspicious cases to public health officials. The authors used a graphical method for representing decision problems, called an influence diagram, to help identify information needed to make such decisions (Figure 2) (24). The use of a decision analytic framework and influence diagram schematic also enabled the authors to identify and evaluate the relationships between the uncertain events that affect the decisions and to identify what is observable to the decision makers (24). They used this analytic framework to inform the key questions for their literature searches and syntheses of the literature. For example, this approach identified data about sensitivity, specificity, and post-test probability of disease for diagnostic tests as critical data for clinicians' diagnostic decisions.

Systematic reviewers faced with a complex topic that includes multiple decision makers (for example, patients and clinicians) or multiple decisions whose outcomes may be related (for example, diagnostic and treatment decisions) may find that the use of a decision analytic framework facilitates problem formulation.

Among the included articles, we found that many reviews adapted established theoretical frameworks to develop their conceptual models. For example, each of the 3 reviews on patient safety relied on previously existing models of patient safety, human work performance, and processes associated with human error (9–11). Figure 3 presents the 2 models that were used in the review “The Effect of Health Care Working Conditions on Patient Safety” (11). For this project, the authors adapted a framework that describes the major characteristics of work environments that can either improve work quality (called resources) or impede work quality (called demands). They combined this with a conceptual model of patient safety to develop their key questions and search strategies.

Adaptation of theoretical frameworks from relevant allied literatures for problem formulation is of particular relevance for topics with literature in its infancy, such as that related to the delivery, organization, and financing of health care. The structure of the Evidence-based Practice Centers, which requires the establishment and active solicitation of input from expert advisors, is particularly effective for developing such conceptual frameworks for problem formulation. These expert panels can direct the research team to theoretical resources in related fields, advise in the adaptation of these established frameworks, and provide feedback on iterative improvements in the development and application of the conceptual model (18).

The key research objectives of the data collection stage are the selection of data sources, development of search strategies for each data source, and abstraction of relevant data from retrieved articles (22). For topics related to the delivery, organization, and financing of health care, the relevant articles will almost always require searches beyond the traditional medical literature. We found 3 commonalities in the searches of the included reviews. First, the conceptual models developed during problem formulation directed the literature searches. Second, as we will describe in greater detail in the following section, several reviews used iterative search strategies. A recent systematic review of factors associated with effective diffusion of innovations emphasizes the importance of a structured search process that combines electronic searches directed by a conceptual model supplemented by input from experts with iterative refinements to the search (25). Third, most of the included reviews used a variety of types of data sources: tra-ditional medical databases such as MEDLINE, alternative databases of relevant associated literatures (for example, psychology, economics, and engineering), structured Internet searches, and input from expert advisors.

Most of the included reviews stated that their search strategies were directly informed by their conceptual frameworks and were iterative. For example, the decision analytic framework developed in the review on information technologies for bioterrorism preparedness and response also informed the authors' literature searches (14). In this case, the authors first sought articles describing bioterrorism-relevant events such as the 2001 anthrax cases, disasters, acts of terrorism, and infectious disease outbreaks to determine the key decisions and task of clinicians and public health officials (14). They then performed additional searches to identify articles about information technologies that could assist each of these decisions (14).

For systematic reviewers faced with broad topics, an initial search may help to define the key elements of heterogeneity. For example, an initial search for quality improvement strategies for diabetes may find potentially relevant articles that include evaluations directed at both inpatient and outpatient management, of type 1 and type 2 diabetes, and of pediatric and adult populations. Such a search could be used to establish inclusion criteria that minimize heterogeneity (for example, limiting the included studies to those that include an outpatient component) and direct subsequent searches.

Given the nascent nature of the quality-based purchasing literature, the authors of the review “Strategies to Support Quality-based Purchasing: A Review of the Evidence” performed searches to identify ongoing research in addition to published or completed research (18). The objective of the search strategy for this project was to identify all published and ongoing randomized trials of quality-based purchasing strategies. In addition to traditional medical literatures referenced in MEDLINE and the Cochrane databases, the authors searched Web sites of relevant organizations (for example, employer organizations pursuing quality-based purchasing) and reference lists provided by their expert advisors (18). To perform this search, they relied primarily on Web-based searches of organizations that fund relevant research (for example, AHRQ and the Robert Wood Johnson Foundation). They also sought direct information about ongoing research from personal contacts with individuals at these funding organizations and members of their expert advisor panel. The use of expert advisory panels by the Evidence-based Practice Centers provides 3 benefits to data collection: Experts can provide access to robust bibliographies of relevant articles, can advise about very recently published articles (that have often not yet been referenced in the electronic databases), and can direct systematic reviewers to unpublished data.

Several specialized databases may facilitate directed literature searches on topics related to the delivery, organization, and financing of health care. A review of these is outside the scope of this paper; however, we briefly highlight 4 databases that may be of particular interest to the systematic reviewer of related topics.

First, the Cochrane Collaboration's Specialized Register of Effective Practice and Organization of Care (EPOC) maintains a database of “educational, behavioral, financial, organizational, and regulatory interventions designed to improve health professional practice and the organization of health care services” (7). Interested systematic reviewers can submit a search request to the EPOC trials search coordinator.

Second, the Information Service Centre for Reviews and Dissemination at the University of York has recently prepared a review of resources of the health economics literature, including electronic sources of international health economics evaluations and repositories of research papers and working papers in health economics (26). This is in addition to the 3 other databases that can be searched directly from their Web site, including registries of health technology assessments and economic evaluations.

Third, the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) of the University of London performs systematic reviews on topics related to health promotion interventions (27). For systematic reviewers interested in topics related to health promotion, the EPPI-Center has contributed health promotion trials to the searchable Trial Registry (CENTRAL) of the Cochrane Library. Additionally, they have developed a publicly available database of these trials (Trials Register of Promoting Health Interventions) (TROPHI) (28).

Finally, reviewers addressing policy-oriented questions related to implementation research from fields such as education, psychology, and criminology may be interested in the 2 databases of intervention trials and policy evaluations maintained by the Campbell Collaboration (29).

The key research objective of the data analysis and interpretation stage is to evaluate the included articles for heterogeneity and select appropriate methods for combining homogeneous studies (22). The published literature on the delivery, organization, and financing of health care often comprises heterogeneous interventions described in small studies that often differ significantly from each other with respect to the study design and outcomes evaluated. This heterogeneity often prevents quantitative combination of studies. Thus, most of the included reviews present their data descriptively. For example, in the review of quality-based purchasing, given the very small number of included articles, the authors briefly summarized each individual article. In addition, organizing the included studies according to the key criteria developed in the conceptual framework can contribute significantly to an appreciation of the relevant literature. For example, the review on information technologies for bioterrorism preparedness organized the included articles according to the key types of decisions the technologies were designed to facilitate (for example, detection, diagnosis, management, surveillance, and communication). The Government Accounting Office recently used this classification system to evaluate federally funded information technologies for biodefense (30).

Two ongoing projects may provide additional insight and recommendations for the systematic reviewer interested in descriptive methods or narrative synthesis: One project sponsored by the Information Service Centre for Reviews and Dissemination at the University of York is attempting to developing methods for the narrative synthesis of quantitative and qualitative data in systematic reviews of effects (31). Second, the Cochrane/Campbell Qualitative Methods Network is actively working to develop methodologic standards for searching, appraising, and combining data from qualitative research for systematic reviews (32). We also direct interested readers to a recent review by Dixon-Woods and colleagues that discusses the advantages and disadvantages of various methods for synthesizing qualitative data, including thematic analysis, qualitative comparative analysis, and Bayesian meta-analysis (33).

Users of systematic reviews such NICE and PBAC have identified an increasing need for guidance in judging the quality of technology assessments and economic evaluations (6). Among the included studies, we found that the authors often used selected questions from published scales (34) to describe the quality of the literature in general or to stratify included studies before qualitative synthesis. This approach is consistent with the general movement away from the use of global quality scores as a weight in meta-analyses (35). However, the heterogeneity of studies used to evaluate the organization, delivery, and financing of health care can make it difficult to identify a short list of quality features that apply equally well to all relevant studies.

Because of the heterogeneity of the study designs and outcome measures encountered by syntheses of topics related to the organization, delivery, and financing of health care, systematic reviewers typically rely on methods that use the direction of the effect rather than the magnitude of the effect to combine studies quantitatively. Such analyses amount to “vote counting,” the weakest and potentially most misleading form of quantitative synthesis (36). The authors of “The Quality Gap: A Critical Analysis of Quality Improvement Strategies,” a recent synthesis of the evidence supporting quality improvement strategies for diabetes and hypertension, faced this issue as the multiple studies of changes in patient and provider behavior targeted differing processes of care (13). To preserve the essential meaning of the outcomes while still permitting pooling across studies, the investigators adopted a method developed for the United Kingdom Health Technology Assessment monograph on the impact of clinical practice guidelines (37). Following this method, they calculated a “median effect” among the outcomes evaluated in each study. For example, a study may have reported 3 outcomes involving provider adherence to a guideline for diabetes care: one outcome related to checking glycosylated hemoglobin, another outcome related to screening for retinal disease, and a third outcome for educating patients about important aspects of self-care. The authors ranked the outcomes according to the change in provider adherence, and chose the median of these to characterize the outcome of the study. The authors then used this outcome, which represented neither the best-case nor the worst-case findings, as the result that would be combined from studies with similar interventions (13).

The authors of this review also developed a novel approach for handling heterogeneity among complex, multifaceted interventions (13). When quality improvement interventions use multiple distinct strategies (for example, audit and feedback to providers combined with a financial incentive and supplemented by patient education), the pooled effect associated with any single strategy is confounded by effects of multiple co-interventions. To address this problem, investigators calculated the median effect of a target outcome (as was just described) for all interventions with a given quality improvement strategy (for example, provider education) and compared this effect with that associated with all interventions not using that strategy (for example, all interventions that lacked component of provider education). This analysis, performed initially with a simple comparison of the median effects (using the Mann–Whitney test to determine whether medians could be from the same population) and subsequently performed with a regression model of mean effect sizes, estimated the effect associated with adding a specific quality improvement strategy to a multifaceted intervention.

The focus on the incremental benefit of a particular quality improvement strategy represents an advance over analyses that focus on single quality improvement strategies despite their coming from complex interventions (38–40), but it relies on 2 assumptions: that the combinations of strategies are close to random (for example, that it is not the case that 2 particular strategies always appear together) and that the effects of the strategies are additive. From an analytic perspective, this approach of summarizing effect using a median effect size is not as susceptible to the effects of outliers; however, it does not incorporate information about study size. To address this limitation, the authors of the quality improvement review stratified the results by study size and other design characteristics to evaluate whether the median effects of large studies differed from those of small studies.

Several included reviews developed simulation models to address gaps in the primary literature (15, 18). For example, in the review “Regionalization of Bioterrorism Preparedness and Response,” the authors found minimal published evidence that specifically addressed key policy questions regarding the development of local inventories of medical and pharmaceutical supplies for bioterrorism responses (15). Therefore, they used the data collected from the systematic review on the expected progression of disease for a single bioterrorism agent (anthrax) to develop a dynamic compartmental model (41, 42) that evaluates anthrax disease progression in a population of exposed and potentially exposed individuals. The model calculates expected anthrax-associated mortality and costs given assumptions about stockpiling of local antibiotics, event detection, and strategies for dispensing antibiotics. Assumptions were informed by the available published evidence and expert opinion. This model enabled the authors to investigate several additional policy questions of national relevance for which there is currently very limited evaluative evidence but for which policy decisions are being formulated. These included questions of the effect of early detection of a covert bioterrorism attack on expected mortality (for example, the role of bioterrorism surveillance systems) and the effect of poor adherence to prophylactic antibiotics on expected mortality. The results of the model in combination with the results of the systematic review highlighted key gaps in the literature and the key targets of future research by identifying those assumptions to which the model results were most sensitive, and for which there was little or no published literature to inform the assumptions.

The use of simulation models may be helpful for topics related to the delivery, organization, and financing of health care because the literature in these areas is somewhat less developed than that in other medical fields. Models can serve to complement the often limited primary evidence by highlighting data that may affect the conclusions of the systematic review, and thus identify key targets for future research. Although model-based analyses can identify and help prioritize targets for research, when data for key model inputs are lacking, other conclusions drawn from the model will usually be limited.

We identified several key approaches used in rigorous syntheses of topics related to the delivery, organization, and financing of health care. We did not attempt to evaluate whether the methodologic approaches used in the included reviews are better or worse than alternative approaches. We described the methods used and, on the basis of our findings, we make the following recommendations about potentially useful methodologic approaches for the syntheses of broad-based health care topics (Table).

First, start with a conceptual model. Seek established theoretical frameworks from relevant fields to provide structure to the problem formulation and data analysis stages of the research. For example, the use of a decision analytic framework may facilitate problem formulation of topics that include multiple decision makers and multiple decisions whose outcomes may be related, or that require the explicit description of information required for decision making. Second, adopt an iterative approach to problem formulation and literature searches. Initial (somewhat broad) literature searches can assist in problem formulation. Subsequent, tailored searches can then be performed for relevant studies. The use of expert advisors is particularly helpful in terms of receiving guidance after iterations of the problem formulation and literature searches. Third, identify relevant literature sources outside those referenced in MEDLINE and other traditional medical bibliographic databases. The Appendix Table provides example data sources. Fourth, use the conceptual model to organize narrative descriptions of the included studies. Fifth, consider using selected, validated evaluation criteria to help organize groups of studies and select them for either qualitative or quantitative synthesis. Sixth, if quantitative synthesis is possible, consider collapsing related outcomes to facilitate multivariate assessments of the effects of interventions on complex processes. If this type of quantitative analysis is not possible, model-based analyses may be useful. Seventh, consider simulation models that use data from the review to explore key gaps in the literature and offer insights to policymakers and people who fund research.

Topics related to health care delivery, organization, and financing pose inherent challenges for systematic reviews. Problem formulation may be difficult, evaluative studies may be scarce, and the common quantitative approaches for meta-analysis may not be applicable. Nonetheless, we found systematic reviews that made important contributions and helped provide a coherent overview of the studies they assessed. As the literature in these areas grows, so will the opportunities to refine conceptual frameworks for problem formulation, to develop rigorous methods for searching nontraditional literature sources, and to explore multivariate methods for data synthesis.