0
Challenges of Summarizing Better Information for Better Health: The Evidence-based Practice Center Experience |

Challenges in Systematic Reviews: Synthesis of Topics Related to the Delivery, Organization, and Financing of Health Care FREE

Dena M. Bravata, MD, MS; Kathryn M. McDonald, MM; Kaveh G. Shojania, MD; Vandana Sundaram, MPH; and Douglas K. Owens, MD, MS
[+] Article and Author Information

From Stanford University–University of California, San Francisco, Evidence-based Practice Center and the Center for Primary Care and Outcomes Research, Stanford University School of Medicine, Stanford, California; University of Ottawa, Ottawa Health Research Institute, Ottawa, Canada; and Veterans Affairs Palo Alto Health Care System, Palo Alto, California.


Acknowledgments: The authors thank Jody L. Mechanic, RN, MS, for her assistance in preparing this manuscript.

Grant Support: This work was performed by the Stanford–University of California, San Francisco, Evidence-based Practice Center under contract number 290-02-0017 to the Agency for Healthcare Research and Quality, Rockville, Maryland. Dr. Owens was supported by the Department of Veterans Affairs.

Potential Financial Conflicts of Interest: Authors of this paper have received funding for Evidence-based Practice Center reports.

Requests for Single Reprints: Dena M. Bravata, MD, MS, Center for Primary Care and Outcomes Research, Stanford University, 117 Encina Commons, Stanford, CA 94305-6019; e-mail, dbravata@stanford.edu.

Current Author Addresses: Drs. Bravata, McDonald, and Sundaram: Center for Primary Care and Outcomes Research, Stanford University, 117 Encina Commons, Stanford, CA 94305-6019.

Dr. Shojania: Ottawa Health Research Institute; The Ottawa Hospital–Civic Campus, 1053 Carling Avenue, Room C403, Box 693, Ottawa K1Y 4E9, Ontario.

Dr. Owens: Veterans Affairs Palo Alto Health Care System, Palo Alto, California, and Center for Primary Care and Outcomes Research, Stanford University, 117 Encina Commons, Stanford, CA 94305-6019.


Ann Intern Med. 2005;142(12_Part_2):1056-1065. doi:10.7326/0003-4819-142-12_Part_2-200506211-00005
Text Size: A A A

Some important health policy topics, such as those related to the delivery, organization, and financing of health care, present substantial challenges to established methods for evidence synthesis. For example, such reviews may ask: What is the effect of for-profit versus not-for-profit delivery of care on patient outcomes? Or, which strategies are the most effective for promoting preventive care? This paper desc ribes innovative methods for synthesizing evidence related to the delivery, organization, and financing of health care. We found 13 systematic reviews on these topics that described novel methodologic approaches. Several of these syntheses used 3 approaches: conceptual frameworks to inform problem formulation, systematic searches that included nontraditional literature sources, and hybrid synthesis methods that included simulations to address key gaps in the literature. As the primary literature on these topics expands, so will opportunities to develop additional novel methods for performing high-quality comprehensive syntheses.

Quantitative systematic reviews have traditionally focused on estimating the effectiveness of specific therapeutic interventions, assessing the accuracy of diagnostic tests, and, to a lesser extent, quantifying epidemiologic relationships (13). Such meta-analyses lend themselves to clear formulation of problems with research questions such as the following: Does a particular intervention increase survival in patients with a particular diagnosis? What are the sensitivity and specificity of a given diagnostic test? What is the probability that patients with a particular risk factor are likely to develop a given disease? The literature searches, development of inclusion and exclusion criteria, and selection of quantitative methods for combining evidence follow readily from such research questions.

By contrast, systematic reviews of some important policy issues, such as those related to the delivery, organization, and financing of health care, present 3 key challenges to established methods of systematic reviews (4). First, such reviews may require considerable effort at the problem formulation stage to narrow the scope of the question to one that can feasibly be addressed and to define appropriate inclusion and exclusion criteria that facilitate literature searches. They may address research questions such as the following: What is the relative effect of private for-profit versus private not-for-profit delivery of care on patient outcomes? Which strategies are the most effective for promoting preventive care? Second, they may not lend themselves to straightforward literature searches, as some of the literature addressing these complex questions falls outside the traditional medical literature and may be difficult to retrieve (5). For example, the relevant literature may consist principally of institutional reports, white papers, and other “grey literature,” which are often not well cataloged in electronic databases. Third, the included articles may be highly heterogeneous with respect to study design (including the key reported outcomes) and quality. This can lead to difficulties appraising and interpreting the evidence and can severely limit opportunities to synthesize the data quantitatively, as well as qualitatively.

There is increasing interest in using high-quality systematic reviews of topics related to the delivery, organization, and financing of health care. For example, economic evaluations of technologies are increasingly being used to inform coverage decisions by organizations such as the National Institute for Clinical Excellence (NICE) in the United Kingdom and the Australian Government's Pharmaceutical Benefits Advisory Committee (PBAC) (6). Several research groups are developing expertise in performing high-quality systematic reviews on these topics. Notably, within the Cochrane Collaboration, the Cochrane Effective Practice and Organisation of Care Group has performed numerous reviews on related topics (78).

The Agency for Healthcare Research and Quality (AHRQ) has increasingly asked its Evidence-based Practice Centers to synthesize evidence on complex, policy-related topics that do not lend themselves to traditional methods of evidence synthesis. In this review, we discuss innovative methods used by Evidence-based Practice Centers and other researchers to synthesize evidence related to the study of the organization, delivery, and financing of health care. We reviewed 94 Evidence-based Practice Center reviews and other published systematic reviews to identify reviews that specified their method for problem formulation, discussed the challenges of using nontraditional or medical literatures, and synthesized broad-based literature relevant to health care delivery, organization, or financing. In this supplement article, we review approaches used in these systematic reviews for problem formulation, data collection, and analysis and interpretation.

We found 13 published reviews of broad topics related to the delivery, organization, and financing of health care that specified their methods for problem formulation, data collection, and data analysis and interpretation. The topics meeting these criteria broadly included patient safety (911), quality improvement (1213), bioterrorism preparedness (1415), health promotion (16), health care financing (1719), telemedicine (20), and health literacy (21). We present the results of our review in the Appendix Table.

In the sections that follow, we summarize innovative approaches to solving the unusual methodologic challenges characteristic of these types of evidence synthesis topics. We have organized these sections according to 3 of the stages of evidence synthesis from the Cooper model of research synthesis: problem formulation, data collection, and data analysis and interpretation (22).

The problem formulation stage has 3 research objectives: development of the key research questions, construction of the definitions of the key concepts that distinguish relevant from irrelevant studies, and the establishment of inclusion and exclusion criteria for the review (22). In general, given the complexity and breadth of the topics addressed in the included reviews, the authors often described the use of conceptual frameworks for problem formulation. Specifically, they often adapted theoretical frameworks from related fields to develop conceptual models that they then used to specify key research questions and direct their literature searches.

Systematic Methods for Defining Concepts and Inclusion Criteria

The evidence related to health care delivery, organization, and financing often describes complex systems processes. Without clearly stated definitions of the processes of interest, it can be difficult to identify the research questions, develop appropriate literature searches, and formulate inclusion and exclusion criteria. Thus, for some projects, the essential task of the problem formulation stage is the defining of relevant concepts. For example, in the review “Measures of Patient Safety Based on Hospital Administrative Data: The Patient Safety Indicators,” the authors evaluated patient safety indicators developed from administrative data (9). They began by developing a set of standardized definitions of 9 key concepts relevant to their topic, including quality of health care, quality indicators, patient safety, patient safety indicators, medical error, and adverse events (9). They used these definitions to develop a list of potential indicators of medical errors, patient safety, or potentially preventable complications (9). The authors then invited 21 professional clinical organizations and 16 surgical subspecialty organizations to submit nominations of experts to evaluate potential patient safety indicators. An expanded expert advisory group of 76 clinicians (divided into 11 panels) reviewed and evaluated indicators according to a method adapted from the RAND/UCLA Appropriateness Method (23). This consisted of an initial independent assessment of each indicator by clinician panelists using an initial questionnaire, a conference call among all panelists, followed by a final independent assessment by clinician panelists using the same questionnaire (9). The panel process served to refine definitions of some indicators, add new measures, and dismiss indicators with major concerns from further consideration (9). Finally, a subset of 34 of the most promising indicators was developed and additional literature was sought for studies describing these indicators. This unique use of a series of expert advisory panels produced inclusion criteria with considerable face validity given the breath and depth of the clinical specialists involved in the process.

Use of a Decision Analytic Framework To Define Key Research Questions

The authors of the review “Bioterrorism Preparedness and Response: Use of Information Technologies and Decision Support Systems” used an approach based on decision theory for defining the key research questions of their study (14). Their review evaluated whether existing information technologies and decision support systems would be useful for clinicians and public health officials preparing for and responding to bioterrorism (14). The authors were faced with a broad topic that involved highly heterogeneous technologies, deployed in different settings, for different users, and for different aspects of bioterrorism preparedness planning and responses. To specify the key research questions of their study, the authors used a decision analytic framework to identify the key decisions of clinicians and public health officials preparing for and responding to bioterrorism (Figures 1 and 2) (24). For example, they identified 4 key decisions of clinicians during a bioterrorism response: diagnostic decisions, management decisions (for example, how to select antibiotics for a given patient), prevention decisions (for example, which patients to place in isolation), and decisions about when to report suspicious cases to public health officials. The authors used a graphical method for representing decision problems, called an influence diagram, to help identify information needed to make such decisions (Figure 2) (24). The use of a decision analytic framework and influence diagram schematic also enabled the authors to identify and evaluate the relationships between the uncertain events that affect the decisions and to identify what is observable to the decision makers (24). They used this analytic framework to inform the key questions for their literature searches and syntheses of the literature. For example, this approach identified data about sensitivity, specificity, and post-test probability of disease for diagnostic tests as critical data for clinicians' diagnostic decisions.

Grahic Jump Location
Figure 1.
Approach to problem formulation, data collection, and analysis and interpretation of reports of information technologies and decision support systems for bioterrorism preparedness and response.

This figure describes 3 stages of evidence synthesis used in a systematic review evaluating reports of information technologies and decision support systems for bioterrorism preparedness and response. The description of the full conceptual framework (24) and results of this systematic review (14) are available elsewhere. Figure modified with permission from reference 24: Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK. Med Decis Making, pp. 192-206, copyright 2004 by Society for Medical Decision Making. Reprinted by permission of Sage Publications.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Schematic of decisions made by clinicians and public health officials during a bioterrorism response.

This figure uses influence diagram notation to depict the key decisions of clinicians and public health officials responding to bioterrorism (rectangular decision nodes), to identify the uncertain events affecting these decisions (elliptic chance nodes), and to evaluate the information that is observable by the decision makers at the time they make their decisions. The schematic depicts 3 critical time periods as follows: Time period 1 refers to the interval in which decisions are made by clinicians regarding the events associated with the initial cases; time period 2 refers to the interval in which decisions are made by public health officials regarding the events associated with the initial cases; and time period 3 refers to the interval in which decisions are made by clinicians regarding the events associated with the subsequent cases. The decisions and processes depicted in this figure could be supported by information technologies and decision support systems designed to facilitate bioterrorism preparedness and response. Figure reprinted with permission from reference 24: Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK. Med Decis Making, pp. 192-206, copyright 2004 by Society for Medical Decision Making. Reprinted by permission of Sage Publications.

Grahic Jump Location

Systematic reviewers faced with a complex topic that includes multiple decision makers (for example, patients and clinicians) or multiple decisions whose outcomes may be related (for example, diagnostic and treatment decisions) may find that the use of a decision analytic framework facilitates problem formulation.

Adaptation of Established Theoretical Frameworks

Among the included articles, we found that many reviews adapted established theoretical frameworks to develop their conceptual models. For example, each of the 3 reviews on patient safety relied on previously existing models of patient safety, human work performance, and processes associated with human error (911). Figure 3 presents the 2 models that were used in the review “The Effect of Health Care Working Conditions on Patient Safety” (11). For this project, the authors adapted a framework that describes the major characteristics of work environments that can either improve work quality (called resources) or impede work quality (called demands). They combined this with a conceptual model of patient safety to develop their key questions and search strategies.

Grahic Jump Location
Figure 3.
Analytic framework and conceptual model of patient safety from the evidence report “The Effect of Health Care Working Conditions on Patient Safety.”

Top. An analytic framework with the linkages between antecedent factors (generally not modifiable), work environment conditions (which can either impede or improve work quality, and are therefore denoted as “demands” or “resources”), and the resultant influence on patient safety and outcomes. Bottom . Conceptual model of patient safety derived from several sources. The research team used these representations to develop a series of research questions focusing on the effects of the 5 categories of working conditions on patient outcomes related to patient safety, rate of medical error, rate of error recognition, and probability of adverse events. For example, one key question of this review was the following: Does the complexity of the plan of care influence whether working conditions affect patient outcomes that are related to patient safety? An asterisk denotes where this key question fits in the conceptual framework.

Grahic Jump Location

Adaptation of theoretical frameworks from relevant allied literatures for problem formulation is of particular relevance for topics with literature in its infancy, such as that related to the delivery, organization, and financing of health care. The structure of the Evidence-based Practice Centers, which requires the establishment and active solicitation of input from expert advisors, is particularly effective for developing such conceptual frameworks for problem formulation. These expert panels can direct the research team to theoretical resources in related fields, advise in the adaptation of these established frameworks, and provide feedback on iterative improvements in the development and application of the conceptual model (18).

The key research objectives of the data collection stage are the selection of data sources, development of search strategies for each data source, and abstraction of relevant data from retrieved articles (22). For topics related to the delivery, organization, and financing of health care, the relevant articles will almost always require searches beyond the traditional medical literature. We found 3 commonalities in the searches of the included reviews. First, the conceptual models developed during problem formulation directed the literature searches. Second, as we will describe in greater detail in the following section, several reviews used iterative search strategies. A recent systematic review of factors associated with effective diffusion of innovations emphasizes the importance of a structured search process that combines electronic searches directed by a conceptual model supplemented by input from experts with iterative refinements to the search (25). Third, most of the included reviews used a variety of types of data sources: tra-ditional medical databases such as MEDLINE, alternative databases of relevant associated literatures (for example, psychology, economics, and engineering), structured Internet searches, and input from expert advisors.

Iterative Literature Searches Directed by Problem Formulation

Most of the included reviews stated that their search strategies were directly informed by their conceptual frameworks and were iterative. For example, the decision analytic framework developed in the review on information technologies for bioterrorism preparedness and response also informed the authors' literature searches (14). In this case, the authors first sought articles describing bioterrorism-relevant events such as the 2001 anthrax cases, disasters, acts of terrorism, and infectious disease outbreaks to determine the key decisions and task of clinicians and public health officials (14). They then performed additional searches to identify articles about information technologies that could assist each of these decisions (14).

For systematic reviewers faced with broad topics, an initial search may help to define the key elements of heterogeneity. For example, an initial search for quality improvement strategies for diabetes may find potentially relevant articles that include evaluations directed at both inpatient and outpatient management, of type 1 and type 2 diabetes, and of pediatric and adult populations. Such a search could be used to establish inclusion criteria that minimize heterogeneity (for example, limiting the included studies to those that include an outpatient component) and direct subsequent searches.

Using a Variety of Literature Sources

Given the nascent nature of the quality-based purchasing literature, the authors of the review “Strategies to Support Quality-based Purchasing: A Review of the Evidence” performed searches to identify ongoing research in addition to published or completed research (18). The objective of the search strategy for this project was to identify all published and ongoing randomized trials of quality-based purchasing strategies. In addition to traditional medical literatures referenced in MEDLINE and the Cochrane databases, the authors searched Web sites of relevant organizations (for example, employer organizations pursuing quality-based purchasing) and reference lists provided by their expert advisors (18). To perform this search, they relied primarily on Web-based searches of organizations that fund relevant research (for example, AHRQ and the Robert Wood Johnson Foundation). They also sought direct information about ongoing research from personal contacts with individuals at these funding organizations and members of their expert advisor panel. The use of expert advisory panels by the Evidence-based Practice Centers provides 3 benefits to data collection: Experts can provide access to robust bibliographies of relevant articles, can advise about very recently published articles (that have often not yet been referenced in the electronic databases), and can direct systematic reviewers to unpublished data.

Several specialized databases may facilitate directed literature searches on topics related to the delivery, organization, and financing of health care. A review of these is outside the scope of this paper; however, we briefly highlight 4 databases that may be of particular interest to the systematic reviewer of related topics.

First, the Cochrane Collaboration's Specialized Register of Effective Practice and Organization of Care (EPOC) maintains a database of “educational, behavioral, financial, organizational, and regulatory interventions designed to improve health professional practice and the organization of health care services” (7). Interested systematic reviewers can submit a search request to the EPOC trials search coordinator.

Second, the Information Service Centre for Reviews and Dissemination at the University of York has recently prepared a review of resources of the health economics literature, including electronic sources of international health economics evaluations and repositories of research papers and working papers in health economics (26). This is in addition to the 3 other databases that can be searched directly from their Web site, including registries of health technology assessments and economic evaluations.

Third, the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Centre) of the University of London performs systematic reviews on topics related to health promotion interventions (27). For systematic reviewers interested in topics related to health promotion, the EPPI-Center has contributed health promotion trials to the searchable Trial Registry (CENTRAL) of the Cochrane Library. Additionally, they have developed a publicly available database of these trials (Trials Register of Promoting Health Interventions) (TROPHI) (28).

Finally, reviewers addressing policy-oriented questions related to implementation research from fields such as education, psychology, and criminology may be interested in the 2 databases of intervention trials and policy evaluations maintained by the Campbell Collaboration (29).

The key research objective of the data analysis and interpretation stage is to evaluate the included articles for heterogeneity and select appropriate methods for combining homogeneous studies (22). The published literature on the delivery, organization, and financing of health care often comprises heterogeneous interventions described in small studies that often differ significantly from each other with respect to the study design and outcomes evaluated. This heterogeneity often prevents quantitative combination of studies. Thus, most of the included reviews present their data descriptively. For example, in the review of quality-based purchasing, given the very small number of included articles, the authors briefly summarized each individual article. In addition, organizing the included studies according to the key criteria developed in the conceptual framework can contribute significantly to an appreciation of the relevant literature. For example, the review on information technologies for bioterrorism preparedness organized the included articles according to the key types of decisions the technologies were designed to facilitate (for example, detection, diagnosis, management, surveillance, and communication). The Government Accounting Office recently used this classification system to evaluate federally funded information technologies for biodefense (30).

Two ongoing projects may provide additional insight and recommendations for the systematic reviewer interested in descriptive methods or narrative synthesis: One project sponsored by the Information Service Centre for Reviews and Dissemination at the University of York is attempting to developing methods for the narrative synthesis of quantitative and qualitative data in systematic reviews of effects (31). Second, the Cochrane/Campbell Qualitative Methods Network is actively working to develop methodologic standards for searching, appraising, and combining data from qualitative research for systematic reviews (32). We also direct interested readers to a recent review by Dixon-Woods and colleagues that discusses the advantages and disadvantages of various methods for synthesizing qualitative data, including thematic analysis, qualitative comparative analysis, and Bayesian meta-analysis (33).

Study Assessment

Users of systematic reviews such NICE and PBAC have identified an increasing need for guidance in judging the quality of technology assessments and economic evaluations (6). Among the included studies, we found that the authors often used selected questions from published scales (34) to describe the quality of the literature in general or to stratify included studies before qualitative synthesis. This approach is consistent with the general movement away from the use of global quality scores as a weight in meta-analyses (35). However, the heterogeneity of studies used to evaluate the organization, delivery, and financing of health care can make it difficult to identify a short list of quality features that apply equally well to all relevant studies.

Pooling Heterogeneous Interventions and Outcomes

Because of the heterogeneity of the study designs and outcome measures encountered by syntheses of topics related to the organization, delivery, and financing of health care, systematic reviewers typically rely on methods that use the direction of the effect rather than the magnitude of the effect to combine studies quantitatively. Such analyses amount to “vote counting,” the weakest and potentially most misleading form of quantitative synthesis (36). The authors of “The Quality Gap: A Critical Analysis of Quality Improvement Strategies,” a recent synthesis of the evidence supporting quality improvement strategies for diabetes and hypertension, faced this issue as the multiple studies of changes in patient and provider behavior targeted differing processes of care (13). To preserve the essential meaning of the outcomes while still permitting pooling across studies, the investigators adopted a method developed for the United Kingdom Health Technology Assessment monograph on the impact of clinical practice guidelines (37). Following this method, they calculated a “median effect” among the outcomes evaluated in each study. For example, a study may have reported 3 outcomes involving provider adherence to a guideline for diabetes care: one outcome related to checking glycosylated hemoglobin, another outcome related to screening for retinal disease, and a third outcome for educating patients about important aspects of self-care. The authors ranked the outcomes according to the change in provider adherence, and chose the median of these to characterize the outcome of the study. The authors then used this outcome, which represented neither the best-case nor the worst-case findings, as the result that would be combined from studies with similar interventions (13).

The authors of this review also developed a novel approach for handling heterogeneity among complex, multifaceted interventions (13). When quality improvement interventions use multiple distinct strategies (for example, audit and feedback to providers combined with a financial incentive and supplemented by patient education), the pooled effect associated with any single strategy is confounded by effects of multiple co-interventions. To address this problem, investigators calculated the median effect of a target outcome (as was just described) for all interventions with a given quality improvement strategy (for example, provider education) and compared this effect with that associated with all interventions not using that strategy (for example, all interventions that lacked component of provider education). This analysis, performed initially with a simple comparison of the median effects (using the Mann–Whitney test to determine whether medians could be from the same population) and subsequently performed with a regression model of mean effect sizes, estimated the effect associated with adding a specific quality improvement strategy to a multifaceted intervention.

The focus on the incremental benefit of a particular quality improvement strategy represents an advance over analyses that focus on single quality improvement strategies despite their coming from complex interventions (3840), but it relies on 2 assumptions: that the combinations of strategies are close to random (for example, that it is not the case that 2 particular strategies always appear together) and that the effects of the strategies are additive. From an analytic perspective, this approach of summarizing effect using a median effect size is not as susceptible to the effects of outliers; however, it does not incorporate information about study size. To address this limitation, the authors of the quality improvement review stratified the results by study size and other design characteristics to evaluate whether the median effects of large studies differed from those of small studies.

Simulations Fill Gaps in Available Evidence

Several included reviews developed simulation models to address gaps in the primary literature (15, 18). For example, in the review “Regionalization of Bioterrorism Preparedness and Response,” the authors found minimal published evidence that specifically addressed key policy questions regarding the development of local inventories of medical and pharmaceutical supplies for bioterrorism responses (15). Therefore, they used the data collected from the systematic review on the expected progression of disease for a single bioterrorism agent (anthrax) to develop a dynamic compartmental model (41, 42) that evaluates anthrax disease progression in a population of exposed and potentially exposed individuals. The model calculates expected anthrax-associated mortality and costs given assumptions about stockpiling of local antibiotics, event detection, and strategies for dispensing antibiotics. Assumptions were informed by the available published evidence and expert opinion. This model enabled the authors to investigate several additional policy questions of national relevance for which there is currently very limited evaluative evidence but for which policy decisions are being formulated. These included questions of the effect of early detection of a covert bioterrorism attack on expected mortality (for example, the role of bioterrorism surveillance systems) and the effect of poor adherence to prophylactic antibiotics on expected mortality. The results of the model in combination with the results of the systematic review highlighted key gaps in the literature and the key targets of future research by identifying those assumptions to which the model results were most sensitive, and for which there was little or no published literature to inform the assumptions.

The use of simulation models may be helpful for topics related to the delivery, organization, and financing of health care because the literature in these areas is somewhat less developed than that in other medical fields. Models can serve to complement the often limited primary evidence by highlighting data that may affect the conclusions of the systematic review, and thus identify key targets for future research. Although model-based analyses can identify and help prioritize targets for research, when data for key model inputs are lacking, other conclusions drawn from the model will usually be limited.

We identified several key approaches used in rigorous syntheses of topics related to the delivery, organization, and financing of health care. We did not attempt to evaluate whether the methodologic approaches used in the included reviews are better or worse than alternative approaches. We described the methods used and, on the basis of our findings, we make the following recommendations about potentially useful methodologic approaches for the syntheses of broad-based health care topics (Table).

Table Jump PlaceholderTable.  Recommendations for Improving Systematic Reviews of Topics Related to the Delivery, Organization, and Financing of Health Care
Table Jump PlaceholderAppendix Table.  Summary of the Methodologic Approaches of Reviews of Topics Related to the Delivery, Organization, and Financing of Health Care

First, start with a conceptual model. Seek established theoretical frameworks from relevant fields to provide structure to the problem formulation and data analysis stages of the research. For example, the use of a decision analytic framework may facilitate problem formulation of topics that include multiple decision makers and multiple decisions whose outcomes may be related, or that require the explicit description of information required for decision making. Second, adopt an iterative approach to problem formulation and literature searches. Initial (somewhat broad) literature searches can assist in problem formulation. Subsequent, tailored searches can then be performed for relevant studies. The use of expert advisors is particularly helpful in terms of receiving guidance after iterations of the problem formulation and literature searches. Third, identify relevant literature sources outside those referenced in MEDLINE and other traditional medical bibliographic databases. The Appendix Table provides example data sources. Fourth, use the conceptual model to organize narrative descriptions of the included studies. Fifth, consider using selected, validated evaluation criteria to help organize groups of studies and select them for either qualitative or quantitative synthesis. Sixth, if quantitative synthesis is possible, consider collapsing related outcomes to facilitate multivariate assessments of the effects of interventions on complex processes. If this type of quantitative analysis is not possible, model-based analyses may be useful. Seventh, consider simulation models that use data from the review to explore key gaps in the literature and offer insights to policymakers and people who fund research.

Topics related to health care delivery, organization, and financing pose inherent challenges for systematic reviews. Problem formulation may be difficult, evaluative studies may be scarce, and the common quantitative approaches for meta-analysis may not be applicable. Nonetheless, we found systematic reviews that made important contributions and helped provide a coherent overview of the studies they assessed. As the literature in these areas grows, so will the opportunities to refine conceptual frameworks for problem formulation, to develop rigorous methods for searching nontraditional literature sources, and to explore multivariate methods for data synthesis.

Lau J, Ioannidis JP, Schmid CH.  Quantitative synthesis in systematic reviews. Ann Intern Med. 1997; 127:820-6. PubMed
 
Egger M, Smith G, Altman D.  Systematic Reviews in Health Care: Meta-Analysis in Context. 2nd ed. London: BMJ Publishing Group; 2001.
 
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D. et al.  Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000; 283:2008-12. PubMed
CrossRef
 
Bero LA, Jadad AR.  How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997; 127:37-42. PubMed
 
McAuley L, Pham B, Tugwell P, Moher D.  Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000; 356:1228-31. PubMed
 
. Donaldson C, Mugford M, Vale L Evidence-Based Health Economics: From Effectiveness to Efficiency in Systematic Review. London: BMJ Publishing Group; 2002.
 
EPOC: Effective Practice and Organisation of Care Group. The Cochrane Collaboration. Accessed athttp://www.epoc.uottawa.ca/on 5 January 2005.
 
Mowatt G, Grimshaw JM, Davis DA, Mazmanian PE.  Getting evidence into practice: the work of the Cochrane Effective Practice and Organization of care Group (EPOC). J Contin Educ Health Prof. 2001; 21:55-60. PubMed
 
McDonald KM, Romano PS, Geppert J, Davies SM, Duncan B, Shojania KG, et al.  Measures of patient safety based on hospital administrative data: the patient safety indicators. Evidence Report/Technology Assessment No. 5 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-0038.
 
Shojania K, Duncan B, McDonald K, Wachter RM.  Making health care safer: a critical analysis of patient safety practices. Evidence Report/Technology Assessment No. 43 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2001. AHRQ publication no. 01-E058.
 
Hickam DH, Severance S, Feldstein A, et al.  The effect of health care working conditions on patient safety. Evidence Report/Technology Assessment No. 74 (Prepared by the Oregon Health & Science University Evidence-based Practice Center under contract 290-97-0018). Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 03-E031.
 
Stone EG, Morton SC, Hulscher ME, Maglione MA, Roth EA, Grimshaw JM. et al.  Interventions that increase use of adult immunization and cancer screening services: a meta-analysis. Ann Intern Med. 2002; 136:641-51. PubMed
 
Shojania K, McDonald K, Wachter R, Owens D.  Closing the quality gap: a critical analysis of quality improvement strategies, volume 1–series overview and methodology. (Prepared by the UCSF–Stanford University Evidence-based Practices Center under contract 290-02-0017.) Rockville, MD: Agency for Healthcare Research and Quality; July 2004. AHRQ publication no. 04-0051-1.
 
Bravata DM, McDonald K, Owens DK, Buckeridge D, Haberland C, Rydzak C, et al.  Bioterrorism preparedness and response: use of information technologies and decision support systems. Evidence Report/Technology Assessment No. 59 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-E061.
 
Bravata DM, McDonald KM, Owens DK, Wilhelm-Leen E, Brandeau ML, Zaric GS, et al.  Regionalization of bioterrorism preparedness and response. Evidence Report/Technology Assessment No. 96. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-E016-2.
 
Ellis P, Robinson P, Ciliska D, Armour T, Raina P, Brouwers M, et al.  Diffusion and dissemination of evidence-based cancer control interventions. Evidence Report/Technology Assessment No. 79 (Prepared by McMaster University Evidence-based Practice Center under contract 290-97-0017). Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 03-E033.
 
Devereaux PJ, Choi PT, Lacchetti C, Weaver B, Schunemann HJ, Haines T. et al.  A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. CMAJ. 2002; 166:1399-406. PubMed
 
Dudley R, Frolich A, Robinowitz D, Talavera J, Broadhead P, Luft H.  Strategies to support quality-based purchasing: a review of the evidence. Evidence Report/Technology Assessment No. 10 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-02-0017). Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-0057.
 
Buntin MB, Laugesen M, Escarce J, Goldman D, Kan H, Shekelle P, et al.  Determinants of increases in Medicare expenditures for physicians' services. Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 04-0008, under contract 290-97-0001.
 
Hersh WR, Wallace JA, Patterson PK, Shapiro SE, Kraemer DF, Eilers GM, et al.  Telemedicine for the Medicare population. Evidence Report/Technology Assessment No. 24 (Prepared by the Oregon Health Sciences University Evidence-based Practice Center under contract 290-97-0018). Rockville, MD: Agency for Healthcare Research and Quality; 2001. AHRQ publication no. 01-E012.
 
Berkman ND, DeWalt DA, Pignone MP, Sheridan SL, Lohr KN, Lux L, et al.  Literacy and health outcomes. Evidence Report/Technology Assessment No. 87 (Prepared by the RTI-University of North Carolina Evidence-based Practice Center under contract 290-02-0016). Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-E007-2.
 
Cooper HM.  Scientific guidelines for conducting integrative research reviews. Review of Educational Research. 1982; 52:291-302.
 
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. et al.  The RAND/UCLA Appropriateness Method User's Manual. Los Angeles: RAND; 2001.
 
Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK.  A conceptual framework for evaluating information technologies and decision support systems for bioterrorism preparedness and response. Med Decis Making. 2004; 24:192-206. PubMed
 
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O.  Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004; 82:581-629. PubMed
 
Information Resources in Health Economics. (Information ServiceCentre for Reviews and Dissemination, University of York). December 2004. Accessed athttp://www.york.ac.uk/inst/crd/econ.htmon 6 January 2005.
 
The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) of the Social Science Research Unit (SSRU), Institute of Education, University of London. Accessed athttp://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/intro.htmon 15 January 2005.
 
Search the Trials Register of Promoting Health Interventions. Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre). Accessed athttp://eppi.ioe.ac.uk/EPPIWeb/home.aspx?Control=Search&SearchDB=trials&page=/hp/on 15 January 2005.
 
The Campbell Collaboration. Accessed athttp://www.campbellcollaboration.org/index.htmlon 15 January 2005.
 
Bioterrorism: Information Technology Strategy Could Strengthen Federal Agencies' Abilities to Respond to Public Health Emergencies. Washington, DC: General Accounting Office; 2003. Report No. GAO-03-139.
 
Developing methods for the narrative synthesis of quantitative and qualitative data in systematic reviews of effects. Centre for Reviews and Dissemination, University of York. Accessed athttp://www.york.ac.uk/inst/crd/listong.htmon 19 January 2005.
 
Popay J.  Cochrane Qualitative Methods Network. Accessed athttp://www.iphrp.salford.ac.uk/cochraneon 15 January 2005.
 
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A.  Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005; 10:45-53. PubMed
 
West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al.  Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the RTI-University of North Carolina Evidence-based Practice Center under contract 290-97-0011). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-E016.
 
Juni P, Witschi A, Bloch R, Egger M.  The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999; 282:1054-60. PubMed
 
Cooper H, Hedges LV.  The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994.
 
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L. et al.  Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004; 8:iii-iv, 1-72. PubMed
 
Hunt DL, Haynes RB, Hanna SE, Smith K.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998; 280:1339-46. PubMed
 
Thomson O'Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL.  Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000(2):CD000125.
 
Weingarten SR, Henning JM, Badamgarav E, Knight K, Hasselblad V, Gano A Jr, et al.  Interventions used in disease management programmes for patients with chronic illness-which ones work? Meta-analysis of published reports. BMJ. 2002; 325:925. PubMed
 
Anderson RM, May RM.  Infectious Diseases of Humans: Dynamics and Control. Oxford: Oxford Univ Pr; 1991.
 
Bailey NJ.  The Mathematical Theory of Infectious Diseases and its Applications. New York: Hafner Pr; 1975.
 

Figures

Grahic Jump Location
Figure 1.
Approach to problem formulation, data collection, and analysis and interpretation of reports of information technologies and decision support systems for bioterrorism preparedness and response.

This figure describes 3 stages of evidence synthesis used in a systematic review evaluating reports of information technologies and decision support systems for bioterrorism preparedness and response. The description of the full conceptual framework (24) and results of this systematic review (14) are available elsewhere. Figure modified with permission from reference 24: Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK. Med Decis Making, pp. 192-206, copyright 2004 by Society for Medical Decision Making. Reprinted by permission of Sage Publications.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Schematic of decisions made by clinicians and public health officials during a bioterrorism response.

This figure uses influence diagram notation to depict the key decisions of clinicians and public health officials responding to bioterrorism (rectangular decision nodes), to identify the uncertain events affecting these decisions (elliptic chance nodes), and to evaluate the information that is observable by the decision makers at the time they make their decisions. The schematic depicts 3 critical time periods as follows: Time period 1 refers to the interval in which decisions are made by clinicians regarding the events associated with the initial cases; time period 2 refers to the interval in which decisions are made by public health officials regarding the events associated with the initial cases; and time period 3 refers to the interval in which decisions are made by clinicians regarding the events associated with the subsequent cases. The decisions and processes depicted in this figure could be supported by information technologies and decision support systems designed to facilitate bioterrorism preparedness and response. Figure reprinted with permission from reference 24: Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK. Med Decis Making, pp. 192-206, copyright 2004 by Society for Medical Decision Making. Reprinted by permission of Sage Publications.

Grahic Jump Location
Grahic Jump Location
Figure 3.
Analytic framework and conceptual model of patient safety from the evidence report “The Effect of Health Care Working Conditions on Patient Safety.”

Top. An analytic framework with the linkages between antecedent factors (generally not modifiable), work environment conditions (which can either impede or improve work quality, and are therefore denoted as “demands” or “resources”), and the resultant influence on patient safety and outcomes. Bottom . Conceptual model of patient safety derived from several sources. The research team used these representations to develop a series of research questions focusing on the effects of the 5 categories of working conditions on patient outcomes related to patient safety, rate of medical error, rate of error recognition, and probability of adverse events. For example, one key question of this review was the following: Does the complexity of the plan of care influence whether working conditions affect patient outcomes that are related to patient safety? An asterisk denotes where this key question fits in the conceptual framework.

Grahic Jump Location

Tables

Table Jump PlaceholderTable.  Recommendations for Improving Systematic Reviews of Topics Related to the Delivery, Organization, and Financing of Health Care
Table Jump PlaceholderAppendix Table.  Summary of the Methodologic Approaches of Reviews of Topics Related to the Delivery, Organization, and Financing of Health Care

References

Lau J, Ioannidis JP, Schmid CH.  Quantitative synthesis in systematic reviews. Ann Intern Med. 1997; 127:820-6. PubMed
 
Egger M, Smith G, Altman D.  Systematic Reviews in Health Care: Meta-Analysis in Context. 2nd ed. London: BMJ Publishing Group; 2001.
 
Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D. et al.  Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000; 283:2008-12. PubMed
CrossRef
 
Bero LA, Jadad AR.  How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med. 1997; 127:37-42. PubMed
 
McAuley L, Pham B, Tugwell P, Moher D.  Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000; 356:1228-31. PubMed
 
. Donaldson C, Mugford M, Vale L Evidence-Based Health Economics: From Effectiveness to Efficiency in Systematic Review. London: BMJ Publishing Group; 2002.
 
EPOC: Effective Practice and Organisation of Care Group. The Cochrane Collaboration. Accessed athttp://www.epoc.uottawa.ca/on 5 January 2005.
 
Mowatt G, Grimshaw JM, Davis DA, Mazmanian PE.  Getting evidence into practice: the work of the Cochrane Effective Practice and Organization of care Group (EPOC). J Contin Educ Health Prof. 2001; 21:55-60. PubMed
 
McDonald KM, Romano PS, Geppert J, Davies SM, Duncan B, Shojania KG, et al.  Measures of patient safety based on hospital administrative data: the patient safety indicators. Evidence Report/Technology Assessment No. 5 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-0038.
 
Shojania K, Duncan B, McDonald K, Wachter RM.  Making health care safer: a critical analysis of patient safety practices. Evidence Report/Technology Assessment No. 43 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2001. AHRQ publication no. 01-E058.
 
Hickam DH, Severance S, Feldstein A, et al.  The effect of health care working conditions on patient safety. Evidence Report/Technology Assessment No. 74 (Prepared by the Oregon Health & Science University Evidence-based Practice Center under contract 290-97-0018). Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 03-E031.
 
Stone EG, Morton SC, Hulscher ME, Maglione MA, Roth EA, Grimshaw JM. et al.  Interventions that increase use of adult immunization and cancer screening services: a meta-analysis. Ann Intern Med. 2002; 136:641-51. PubMed
 
Shojania K, McDonald K, Wachter R, Owens D.  Closing the quality gap: a critical analysis of quality improvement strategies, volume 1–series overview and methodology. (Prepared by the UCSF–Stanford University Evidence-based Practices Center under contract 290-02-0017.) Rockville, MD: Agency for Healthcare Research and Quality; July 2004. AHRQ publication no. 04-0051-1.
 
Bravata DM, McDonald K, Owens DK, Buckeridge D, Haberland C, Rydzak C, et al.  Bioterrorism preparedness and response: use of information technologies and decision support systems. Evidence Report/Technology Assessment No. 59 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-E061.
 
Bravata DM, McDonald KM, Owens DK, Wilhelm-Leen E, Brandeau ML, Zaric GS, et al.  Regionalization of bioterrorism preparedness and response. Evidence Report/Technology Assessment No. 96. Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-E016-2.
 
Ellis P, Robinson P, Ciliska D, Armour T, Raina P, Brouwers M, et al.  Diffusion and dissemination of evidence-based cancer control interventions. Evidence Report/Technology Assessment No. 79 (Prepared by McMaster University Evidence-based Practice Center under contract 290-97-0017). Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 03-E033.
 
Devereaux PJ, Choi PT, Lacchetti C, Weaver B, Schunemann HJ, Haines T. et al.  A systematic review and meta-analysis of studies comparing mortality rates of private for-profit and private not-for-profit hospitals. CMAJ. 2002; 166:1399-406. PubMed
 
Dudley R, Frolich A, Robinowitz D, Talavera J, Broadhead P, Luft H.  Strategies to support quality-based purchasing: a review of the evidence. Evidence Report/Technology Assessment No. 10 (Prepared by the UCSF–Stanford Evidence-based Practice Center under contract 290-02-0017). Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-0057.
 
Buntin MB, Laugesen M, Escarce J, Goldman D, Kan H, Shekelle P, et al.  Determinants of increases in Medicare expenditures for physicians' services. Rockville, MD: Agency for Healthcare Research and Quality; 2003. AHRQ publication no. 04-0008, under contract 290-97-0001.
 
Hersh WR, Wallace JA, Patterson PK, Shapiro SE, Kraemer DF, Eilers GM, et al.  Telemedicine for the Medicare population. Evidence Report/Technology Assessment No. 24 (Prepared by the Oregon Health Sciences University Evidence-based Practice Center under contract 290-97-0018). Rockville, MD: Agency for Healthcare Research and Quality; 2001. AHRQ publication no. 01-E012.
 
Berkman ND, DeWalt DA, Pignone MP, Sheridan SL, Lohr KN, Lux L, et al.  Literacy and health outcomes. Evidence Report/Technology Assessment No. 87 (Prepared by the RTI-University of North Carolina Evidence-based Practice Center under contract 290-02-0016). Rockville, MD: Agency for Healthcare Research and Quality; 2004. AHRQ publication no. 04-E007-2.
 
Cooper HM.  Scientific guidelines for conducting integrative research reviews. Review of Educational Research. 1982; 52:291-302.
 
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. et al.  The RAND/UCLA Appropriateness Method User's Manual. Los Angeles: RAND; 2001.
 
Bravata DM, McDonald KM, Szeto H, Smith WM, Rydzak C, Owens DK.  A conceptual framework for evaluating information technologies and decision support systems for bioterrorism preparedness and response. Med Decis Making. 2004; 24:192-206. PubMed
 
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O.  Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004; 82:581-629. PubMed
 
Information Resources in Health Economics. (Information ServiceCentre for Reviews and Dissemination, University of York). December 2004. Accessed athttp://www.york.ac.uk/inst/crd/econ.htmon 6 January 2005.
 
The Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) of the Social Science Research Unit (SSRU), Institute of Education, University of London. Accessed athttp://eppi.ioe.ac.uk/EPPIWeb/home.aspx?page=/hp/intro.htmon 15 January 2005.
 
Search the Trials Register of Promoting Health Interventions. Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre). Accessed athttp://eppi.ioe.ac.uk/EPPIWeb/home.aspx?Control=Search&SearchDB=trials&page=/hp/on 15 January 2005.
 
The Campbell Collaboration. Accessed athttp://www.campbellcollaboration.org/index.htmlon 15 January 2005.
 
Bioterrorism: Information Technology Strategy Could Strengthen Federal Agencies' Abilities to Respond to Public Health Emergencies. Washington, DC: General Accounting Office; 2003. Report No. GAO-03-139.
 
Developing methods for the narrative synthesis of quantitative and qualitative data in systematic reviews of effects. Centre for Reviews and Dissemination, University of York. Accessed athttp://www.york.ac.uk/inst/crd/listong.htmon 19 January 2005.
 
Popay J.  Cochrane Qualitative Methods Network. Accessed athttp://www.iphrp.salford.ac.uk/cochraneon 15 January 2005.
 
Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A.  Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005; 10:45-53. PubMed
 
West S, King V, Carey TS, Lohr KN, McKoy N, Sutton SF, et al.  Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the RTI-University of North Carolina Evidence-based Practice Center under contract 290-97-0011). Rockville, MD: Agency for Healthcare Research and Quality; 2002. AHRQ publication no. 02-E016.
 
Juni P, Witschi A, Bloch R, Egger M.  The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999; 282:1054-60. PubMed
 
Cooper H, Hedges LV.  The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994.
 
Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L. et al.  Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004; 8:iii-iv, 1-72. PubMed
 
Hunt DL, Haynes RB, Hanna SE, Smith K.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998; 280:1339-46. PubMed
 
Thomson O'Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL.  Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000(2):CD000125.
 
Weingarten SR, Henning JM, Badamgarav E, Knight K, Hasselblad V, Gano A Jr, et al.  Interventions used in disease management programmes for patients with chronic illness-which ones work? Meta-analysis of published reports. BMJ. 2002; 325:925. PubMed
 
Anderson RM, May RM.  Infectious Diseases of Humans: Dynamics and Control. Oxford: Oxford Univ Pr; 1991.
 
Bailey NJ.  The Mathematical Theory of Infectious Diseases and its Applications. New York: Hafner Pr; 1975.
 

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)