Skip Navigation
American College of Physicians Logo
  • Subscribe
  • Submit a Manuscript
  • Sign In
    Sign in below to access your subscription for full content
    INDIVIDUAL SIGN IN
    Sign In|Set Up Account
    You will be directed to acponline.org to register and create your Annals account
    INSTITUTIONAL SIGN IN
    Open Athens|Shibboleth|Log In
    Annals of Internal Medicine
    SUBSCRIBE
    Subscribe to Annals of Internal Medicine.
    You will be directed to acponline.org to complete your purchase.
Annals of Internal Medicine Logo Menu
  • Latest
  • Issues
  • Channels
  • CME/MOC
  • In the Clinic
  • Journal Club
  • Web Exclusives
  • Author Info
Advanced Search
  • ‹ PREV ARTICLE
  • This Issue
  • NEXT ARTICLE ›
Reviews |3 July 2012

Effect of Clinical Decision-Support Systems: A Systematic Review Free

Tiffani J. Bright, PhD; Anthony Wong, MTech; Ravi Dhurjati, PhD; Erin Bristow, BA; Lori Bastian, MD, MS; Remy R. Coeytaux, MD, PhD; Gregory Samsa, PhD; Vic Hasselblad, PhD; John W. Williams, MD, MHS; Michael D. Musty, BA; Liz Wing, MA; Amy S. Kendrick, RN, MSN; Gillian D. Sanders, PhD; David Lobach, MD, PhD

Tiffani J. Bright, PhD

Anthony Wong, MTech

Ravi Dhurjati, PhD

Erin Bristow, BA

Lori Bastian, MD, MS

Remy R. Coeytaux, MD, PhD

Gregory Samsa, PhD

Vic Hasselblad, PhD

John W. Williams, MD, MHS

Michael D. Musty, BA

Liz Wing, MA

Amy S. Kendrick, RN, MSN

Gillian D. Sanders, PhD

David Lobach, MD, PhD

Article, Author, and Disclosure Information
Author, Article, and Disclosure Information
  • From Duke Evidence-based Practice Center, Duke Clinical Research Institute, Duke University School of Medicine, Durham, North Carolina, and School of Medicine, University of Utah, Salt Lake City, Utah.

    Disclaimer: The authors of this report are responsible for its content. Statements in the report should not be construed as endorsements by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

    Acknowledgment: The authors thank Connie Schardt, MSLS, for help with the literature search and retrieval.

    Grant Support: This project was funded under contract 290-2007-10066-I from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services.

    Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-1215.

    Requests for Single Reprints: Gillian D. Sanders, PhD, Evidence-based Practice Center, Director, Duke Clinical Research Institute, 2400 Pratt Street, Durham, NC 27705; e-mail, gillian.sanders@duke.edu.

    Current Author Addresses: Dr. Bright: 16 Kenilworth Drive, Hampton, VA 23666.

    Mr. Wong: 2618 Briar Trail, Apartment 202, Schaumburg, IL 60173.

    Dr. Dhurjati: D330-1 Mayo (MMC 729), 420 Delaware Street SE, University of Minnesota, Minneapolis, MN 55455.

    Ms. Bristow: 728 Irolo Street, Apartment D, Los Angeles, CA 90005.

    Dr. Bastian: Health Services Research & Development, 152 Veterans Affairs Medical Center, 508 Fulton Street, Durham, NC 27705.

    Drs. Coeytaux, Hasselblad, Williams, and Sanders; Mr. Musty; Ms. Wing; and Ms. Kendrick: Duke Evidence-based Practice Center, Duke Clinical Research Institute, 2400 Pratt Street, Durham, NC 27705.

    Dr. Samsa: Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27705.

    Dr. Lobach: Klesis LLC, 6 Harvey Place, Durham, NC 27705.

    Author Contributions: Conception and design: T.J. Bright, A. Wong, J.W. Williams, G.D. Sanders, D. Lobach.

    Analysis and interpretation of the data: T.J. Bright, A. Wong, R. Dhurjati, G. Samsa, V. Hasselblad, G.D. Sanders, D. Lobach.

    Drafting of the article: T.J. Bright, A. Wong, E. Bristow, G.D. Sanders, D. Lobach.

    Critical revision of the article for important intellectual content: T.J. Bright, G. Samsa, G.D. Sanders, D. Lobach.

    Final approval of the article: T.J. Bright, G. Samsa, G.D. Sanders, D. Lobach.

    Provision of study materials or patients: T.J. Bright, M.D. Musty.

    Statistical expertise: G. Samsa, V. Hasselblad.

    Obtaining of funding: G.D. Sanders, D. Lobach.

    Administrative, technical, or logistic support: T.J. Bright, M.D. Musty, L. Wing, A.S. Kendrick, G.D. Sanders.

    Collection and assembly of data: T.J. Bright, A. Wong, R. Dhurjati, E. Bristow, L. Bastian, R.R. Coeytaux, M.D. Musty.

×
  • ‹ PREV ARTICLE
  • This Issue
  • NEXT ARTICLE ›
Jump To
  • Full Article
  • FULL ARTICLE
  • FULL ARTICLE
    • Abstract
    • Methods
    • Results
    • Discussion
      1. References
  • Figures
  • Tables
  • Supplements
  • Audio/Video
  • Summary for Patients
  • Clinical Slide Sets
  • CME / MOC
  • Comments
  • Twitter Link
  • Facebook Link
  • Email Link
More
  • LinkedIn Link

Abstract

Background:

Despite increasing emphasis on the role of clinical decision-support systems (CDSSs) for improving care and reducing costs, evidence to support widespread use is lacking.

Purpose:

To evaluate the effect of CDSSs on clinical outcomes, health care processes, workload and efficiency, patient satisfaction, cost, and provider use and implementation.

Data Sources:

MEDLINE, CINAHL, PsycINFO, and Web of Science through January 2011.

Study Selection:

Investigators independently screened reports to identify randomized trials published in English of electronic CDSSs that were implemented in clinical settings; used by providers to aid decision making at the point of care; and reported clinical, health care process, workload, relationship-centered, economic, or provider use outcomes.

Data Extraction:

Investigators extracted data about study design, participant characteristics, interventions, outcomes, and quality.

Data Synthesis:

148 randomized, controlled trials were included. A total of 128 (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services (n = 25; odds ratio [OR], 1.42 [95% CI, 1.27 to 1.58]), ordering clinical studies (n = 20; OR, 1.72 [CI, 1.47 to 2.00]), and prescribing therapies (n = 46; OR, 1.57 [CI, 1.35 to 1.82]). Few studies measured potential unintended consequences or adverse effects.

Limitations:

Studies were heterogeneous in interventions, populations, settings, and outcomes. Publication bias and selective reporting cannot be excluded.

Conclusion:

Both commercially and locally developed CDSSs are effective at improving health care process measures across diverse settings, but evidence for clinical, economic, workload, and efficiency outcomes remains sparse. This review expands knowledge in the field by demonstrating the benefits of CDSSs outside of experienced academic centers.

Primary Funding Source:

Agency for Healthcare Research and Quality.

Despite increasing emphasis on clinical decision-support systems (CDSSs) in improving care and reducing costs, evidence supporting widespread use is limited. A CDSS is “any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration” (1). This review examines 3 types of decision-support interventions.
Classic CDSSs include alerts, reminders, order sets, drug-dose calculations that automatically remind the clinician of a specific action, or care summary dashboards that provide performance feedback on quality indicators. Information retrieval tools, such as an “infobutton” embedded in a clinical information system, are designed to aid clinicians in the search and retrieval of context-specific knowledge from information sources based on patient-specific information from a clinical information system. Knowledge resources, such as UpToDate, Epocrates, and MD Consult, consist of distilled primary literature that allows selection of content germane to a specific patient to facilitate decision making at the point of care or for a specific care situation.
Until recently, most studies of CDSSs came from 4 benchmark settings (Brigham and Women's Hospital/Partners HealthCare, the Department of Veterans Affairs, LDS Hospital/Intermountain Healthcare, and the Regenstrief Institute) (2). Although several reviews have examined the effects of CDSSs (1, 3–9), many questions remain about their impact. This systematic review adds to the literature by summarizing trials of CDSSs implemented in a clinical setting to aid decision making at the point of care or for a specific care situation.

Methods

We developed and followed a standard protocol for our review. Full details of our methods, search strategies, results, and conclusions are provided in a report commissioned by the Agency for Healthcare Research and Quality, available at www.effectivehealthcare.ahrq.gov.

Data Sources and Searches

We searched for studies done between January 1976 and January 2011 in MEDLINE accessed through PubMed, CINAHL, PsycINFO, and Web of Science.

Study Selection

We identified randomized trials of CDSSs implemented in a real clinical setting and used by health care providers to aid decision making at the point of care or for a specific care situation. Studies had to report at least one of the following types of outcomes: clinical (length of stay, morbidity, mortality, health-related quality of life, and adverse events), health care process (recommended preventive care, clinical study, or treatment ordered or completed), user workload and efficiency (user knowledge, number of patients seen, clinician workload, and efficiency), relationship-centered (patient satisfaction), economic (cost and cost-effectiveness), or use and implementation by a health care provider (acceptance, satisfaction, use, and implementation). We excluded studies that described nonelectronic CDSSs, included fewer than 50 participants, were not published in English, described closed-loop systems that did not involve a provider, evaluated systems that required mandatory compliance with the CDSS, or evaluated only the performance of the system as opposed to its effect on clinical practice.

Data Extraction and Quality Assessment

Data related to study setting and design, sample characteristics, intervention characteristics, comparators, and outcomes were extracted by 1 reviewer and confirmed by another. Two reviewers used a standardized approach to independently categorize the quality of individual studies as good, fair, or poor (10) and evaluated the overall strength of evidence for each outcome as high, moderate, low, or insufficient (11). Reviewers also identified issues related to study setting, interventions, and outcomes that limited applicability of evidence (10, 12).

Data Synthesis and Analysis

A priori–defined outcomes believed to be important in measuring the effect of CDSSs in improving clinical practice guided our synthesis process. Studies with a common outcome were grouped together to facilitate qualitative analysis. Quantitative analysis was done where 4 or more studies assessed the same outcome in the same manner, regardless of the specific CDSS intervention. Summary estimates were calculated by using the DerSimonian and Laird (13) random-effects model as implemented in Comprehensive Meta-analysis, version 2.2.055 (Biostat, Englewood, New Jersey).

Role of the Funding Source

Primary funding was provided by the Agency for Healthcare Research and Quality. The funding source formulated the initial study questions but otherwise had no role in the design, analysis, or interpretation of the data or in the decision to submit the manuscript for publication.

Results

We screened 15 176 abstracts, evaluated 1407 full-text articles, and included 160 articles, representing 148 unique studies (Appendix Figure). Appendix Table 1 summarizes important characteristics of the studies and their quality rating. A total of 128 studies (86%) assessed health care process measures, 29 (20%) assessed clinical outcomes, and 22 (15%) measured costs. Many studies (n = 51) were performed in environments with established health information technology (IT); many were multisite studies involving multiple institutions (n = 46).
Appendix Figure.

Summary of evidence search and selection.

CDSS = clinical decision-support system; KQ = key question; RCT = randomized, controlled trial.

Appendix Table 1.

Study Characteristics

Appendix Table 1.
The Table summarizes the most important findings and the strength of supporting evidence for those findings, whereas Appendix Table 2 provides examples of interventions that assessed the outcomes of interest. Both commercially and locally developed CDSSs improved health care process measures related to performing preventive services, ordering clinical studies, and prescribing therapies. Few studies (n = 15) indicated conceptualization of potential unintended consequences or measured potential adverse effects of implementing decision-support tools.

Table.

Summary of Evidence, by Outcome

Table.

Appendix Table 2.

Examples of Clinical Decision-Support Interventions

Appendix Table 2.

Clinical Outcomes

Morbidity

Several studies assessed morbidity outcomes (14–38), such as hospitalizations, Apgar scores, surgical site infections, cardiovascular events, colorectal cancer, deep venous thrombosis, and hypoglycemia events. Topics addressed included diagnosis (16, 21, 29, 30, 34), pharmacotherapy (15, 18–22, 25, 28, 33), chronic disease management (14, 19, 20, 22, 26–28, 31, 32, 36, 38), laboratory test ordering (19, 37), immunizations (19, 35), preventive care (17, 19, 28–30, 37), and discharge planning (23, 24). Approximately 50% of the studies were performed in an academic setting.
Many studies evaluated locally developed interventions implemented in the ambulatory environment. Typical interventions were automatically delivered, system-initiated recommendations provided synchronously at the point of care to enable decision making during the health care provider–patient encounter. Three such interventions required a mandatory response (that is, required that the user respond to the given recommendation, whether that response was to accept or dismiss the recommendation or to modify the user's action) (17, 18, 37).
Comparators included usual care or no CDSS and the same CDSS with additional features. Limitations included short follow-up, low statistical power to detect important differences, and the potential for contamination of providers in control groups that improved because of knowledge of interventions.
Meta-analysis of these heterogeneous studies (n = 16) suggested that CDSSs improved morbidity outcomes (relative risk, 0.88 [95% CI, 0.80 to 0.96]). We rated this level of evidence as moderate. Most studies were good quality, and many of the interventions were evaluated in multiple institutions. However, the interventions were often paper-based or standalone systems implemented in academic or Veterans Affairs settings and were targeted toward a single condition.

Mortality

Seven studies (17, 20–22, 33, 39, 40) reported mortality outcomes. Issues addressed included diagnosis (21), pharmacotherapy (20–22, 33, 40), chronic disease management (20, 22), preventing deep venous thrombosis (17), and detecting and notifying clinicians of critical laboratory values (39). Most CDSSs were locally developed and integrated into a computerized physician order entry (CPOE) or electronic health record (EHR) system and had system-initiated recommendations delivered synchronously at the point of care that did not require a clinician response.
Interventions were evaluated against usual care or no CDSS, except for 2 studies (20, 22) that compared the same intervention with additional features. Limitations included small sample size, duration shorter than 1 year, and possible contamination of control providers.
Meta-analysis of these heterogeneous studies (n = 6) suggested no significant effect of CDSSs on mortality, although CIs were wide (odds ratio [OR], 0.79 [CI, 0.54 to 1.15]). Of note, 2 studies reported a significant reduction in mortality with use of CDSSs (20, 22). We rated the overall strength of evidence related to mortality outcomes as low. Most studies were conducted in a single academic or Veterans Affairs setting with a comprehensive, well-established health IT infrastructure.

Adverse Events

Five studies assessed the effectiveness of CDSSs in reducing or preventing adverse events (23, 24, 39–42). Studies were mostly implemented in an academic, inpatient setting. Studies evaluated the effect of these interventions to improve the timing of warfarin therapy (41), improve discharge planning (23, 24), prevent adverse drug events (42), detect critical laboratory values (39), and detect potentially inappropriate or inadequate antimicrobial therapy (40).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-imitated recommendations in real time to enable decision making during the provider–patient encounter. Only 1 study clearly required a mandatory response (39).
All of the CDSSs were evaluated against usual care or no CDSS. Limitations included evaluation at a single institution, evaluation periods less than 1 year, and potential improvement in physician performance because of their knowledge of the intervention.
Meta-analysis of these heterogeneous studies estimated a relative risk of 1.01 (CI, 0.90 to 1.14), and neither this summary nor any individual studies demonstrated a significant effect. We rated this level of evidence as low. Most studies were good quality, and 2 were conducted at multiple sites; however, these interventions primarily contained locally developed knowledge, and results may not be generalizable to nonteaching settings.

Health Care Process Measures

Recommended Preventive Care Service Ordered or Completed

Forty-three studies examined the effect of CDSSs on the rates of ordering or completing recommended preventive care services (17, 19, 27, 28, 35, 37, 43–82). Most studies were conducted in the academic or ambulatory environment. Topics addressed included diagnosis (43, 47, 62, 71, 82), pharmacotherapy (19, 28, 49, 53, 54, 70, 77, 80), chronic disease management (19, 26, 28, 43, 44, 48, 51, 70, 72, 74, 76), laboratory test ordering (19, 37, 55, 60, 70, 71, 74, 75, 80, 81), preventive care (17, 19, 28, 37, 43, 45, 48, 52–60, 62–65, 68–71, 73–75, 79–82), immunizations (19, 35, 48–50, 52, 56, 66, 67, 78, 79, 81), and initiating discussions with patients (60, 61, 72).
Most interventions were locally developed, paper-based, or standalone systems and automatically delivered recommendations in real time to enable decision making during the health care provider–patient encounter. Only 7 of the interventions required a mandatory response (17, 37, 47, 49) or justification (64, 65, 68, 69, 75) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data measuring patient or economic outcomes; few assessments of long-term outcomes of interventions; and the Hawthorne effect, which probably stimulated more comprehensive preventive care across groups.
In meta-analysis of these 25 heterogeneous studies (17, 27, 28, 35, 37, 43–63), the effect of CDSSs on preventive care services was significant (OR, 1.42 [CI, 1.27 to 1.58]) (Figure 1). We rated this level of evidence as high. Approximately one half of the studies were good quality, one third were evaluated in multicenter trials, and one fourth addressed multiple clinical conditions. However, most CDSSs were locally developed, not integrated into a CPOE or an EHR, and evaluated in academic medical centers, all of which can affect the generalizability of these findings.
Figure 1.

Results of studies that examined whether recommended preventive care services were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or completing preventive care services of CDSS vs. control groups. In the 25 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to preventive care recommendations was 1.42 (95% CI, 1.27 to 1.58). CDSS = clinical decision-support system.

Recommended Clinical Study Ordered or Completed

Twenty-nine studies evaluated the effect of CDSSs on ordering and completing recommended clinical studies (26, 31, 32, 83–109). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (85, 89–91, 95, 98, 101–103), pharmacotherapy (94, 98), chronic disease management (26, 31, 32, 84, 85, 103), laboratory test ordering (83, 87, 89, 92–94, 96, 97, 99–101, 104–108), initiating discussions with patients (108, 109), and additional clinical tasks (86, 88, 90, 109).
Typical interventions were locally developed, were integrated CDSS recommendations in a CPOE or an EHR system, and automatically delivered system-initiated recommendations to enable decision making during the provider–patient encounter. Eight interventions required a mandatory response (90, 99, 100, 104, 106, 108) or justification (83, 105) for not adhering to the recommendation.
Comparators included usual care or no CDSS, direct comparison against the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations of the evidence base included diverse metrics to assess adherence to ordering and completing a recommended action, studies not designed to evaluate the clinical or economic outcomes associated with the CDSS interventions, and limited evidence of the effect of CDSSs on a broad set of conditions.
Meta-analysis of these 20 heterogeneous studies (26, 31, 32, 83, 84, 88, 89, 91, 92, 94–96, 98–103, 105, 108, 109) found a significant effect of CDSSs on ordering or completing of clinical studies (OR, 1.72 [CI, 1.47 to 2.00]) (Figure 2). We rated this level of evidence as moderate. Most studies were good quality, approximately one third were implemented in multiple sites, and almost one fourth included a direct comparison of the effectiveness of the CDSS against the same intervention with additional features. However, we noted a strong suggestion of publication bias in these studies, and these results may not be generalizable to all settings because most studies were either evaluated in environments with a well-established health IT infrastructure or conducted outside of the United States.
Figure 2.

Results of studies that examined whether recommended clinical studies were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or completing recommended clinical studies of CDSS vs. control groups. In the 20 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to clinical study recommendations was 1.72 (95% CI, 1.47 to 2.00). CDSS = clinical decision-support system.

Recommended Treatment Ordered or Prescribed

Sixty-seven studies evaluated the effect of CDSSs on ordering and prescribing therapy (14, 18, 20–22, 25–28, 33, 36, 38–41, 43, 44, 53, 54, 70, 80, 82, 84, 88, 94, 98, 110–154). Many studies were conducted in the academic setting, and most were evaluated in the ambulatory environment. Topics addressed included diagnosis (21, 43, 82, 98, 112, 123, 129, 138, 151, 152), pharmacotherapy (18, 20–22, 25, 28, 33, 40, 53, 54, 70, 80, 94, 98, 110, 111, 114, 116, 117, 119, 122, 123, 125–134, 136, 138–150, 153, 154), laboratory test ordering (70, 80, 94, 110, 130), chronic disease management (14, 20, 22, 26–28, 36, 38, 43, 44, 70, 84, 110, 112, 113, 115, 118, 120, 121, 123–125, 132, 133, 135, 141), preventive care (28, 43, 53, 54, 70, 80, 82, 120, 148), and additional clinical tasks (39, 41, 82, 88, 110, 137, 151, 152).
Typical interventions were locally developed, were integrated into a CPOE or an EHR system, and automatically delivered system-initiated recommendations in real time to enable decision making during the provider–patient encounter. Eighteen CDSSs required a mandatory response (18, 39, 119, 122, 134, 139–143, 147, 148, 151, 152) or justification (112, 113, 136, 137, 145) for not adhering to the recommendation. Limitations included inadequate follow-up periods to observe sustained results, sparse data demonstrating how changes in clinician ordering and prescribing led to improvements in clinical or economic outcomes, and study designs that did not capture the extent to which nonadherence with recommended therapy resulted in adverse events.
Meta-analysis of 46 heterogeneous studies (14, 18, 20–22, 25–28, 33, 36, 38, 40, 43, 44, 53, 54, 70, 82, 84, 94, 98, 110, 112–117, 121–124, 129, 130, 134–136, 141–144, 146, 148, 151–153) showed that intervention providers with decision support were more likely to order the appropriate treatment or therapy (OR, 1.57 [CI, 1.35 to 1.82]) (Figure 3). We rated this level of evidence as high. Most studies were good quality, and most were evaluated in multisite trials. However, generalizability may be limited because most studies were implemented in the ambulatory environment, were evaluated in settings where clinicians were experienced EHR users or provided care in an established health IT infrastructure, and incorporated knowledge that was targeted toward specific conditions.
Figure 3.

Results of studies that examined whether recommended treatments were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or prescribing treatment of CDSS vs. control groups. In the 46 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to treatment recommendations was 1.57 (95% CI, 1.35 to 1.82). CDSS = clinical decision-support system.

User Workload and Efficiency Outcomes

Evidence on the effect of CDSSs on clinician knowledge or improved confidence in managing patient care was insufficient (71, 72, 86, 155, 156). Seven studies examined the effect of CDSSs on efficiency (23, 24, 40, 106, 141, 155–157). Limitations included contamination of clinicians in the control group that improved because of knowledge of the intervention, evaluation periods that were too brief to demonstrate an effect on efficiency, and small clinician sample sizes.
We rated the level of evidence as low. Most interventions contained locally developed knowledge, such as protocols or algorithms derived on the basis of local performance, quality, and outcome data not representative of other sites, and were evaluated in academic settings.

Economic Outcomes

Cost

Twenty-two studies reported costs (21, 26, 27, 31, 32, 36, 40, 43, 53, 54, 71, 83, 90, 106, 109, 113, 118, 130, 141, 158–163). Objectives of the CDSSs included diagnosis (21, 43, 71, 90, 160), pharmacotherapy (21, 40, 53, 54, 130, 141), chronic disease management (26, 27, 31, 32, 36, 43, 113, 118, 141, 161, 162), laboratory test ordering (71, 83, 106, 130, 160, 163), preventive care (43, 53, 54, 71, 158, 159), initiating discussions with patients (109, 159), and additional clinical tasks (90, 109). One study reported reduced hospitalization expenses with CDSS use (31, 32), and 12 studies reported that use had a positive effect on costs compared with control groups and other non-CDSS groups (21, 27, 40, 53, 54, 83, 90, 106, 113, 130, 141, 159, 160).
Modest evidence from academic and community inpatient and ambulatory settings showed that locally and commercially developed CDSSs had lower treatment costs, total costs, and reduced costs compared with control groups and other non-CDSS intervention groups. Most studies were conducted in the academic ambulatory setting and evaluated locally developed, integrated CDSSs in CPOE or EHR systems that automatically delivered system-initiated recommendations synchronously at the point of care and did not require a mandatory clinician response.

Cost-Effectiveness

Six studies (53, 54, 56, 57, 78, 95, 96, 161, 162) examined the cost-effectiveness of CDSSs or their effect on cost-effectiveness of care. These demonstrated conflicting findings, with 3 studies suggesting that CDSSs were cost-effective (53, 54, 78, 95) and 3 reporting that CDSSs were not cost-effective (56, 57, 161, 162). Objectives included diagnosis (95), pharmacotherapy (53, 54), chronic disease management (161, 162), preventive care (53, 54, 56, 57), and immunizations (56, 78).

Use and Implementation Outcomes

Twenty-four studies assessed the effect of provider acceptance of CDSSs (19, 41, 55, 62, 75, 90, 105, 113, 119, 120, 136, 137, 145–147, 149, 158, 159, 164–170). Topics addressed included diagnosis (62, 90, 166), pharmacotherapy (19, 119, 136, 145–147, 149, 164, 165), chronic disease management (19, 113, 120, 168–170), laboratory test ordering (19, 55, 75, 105), preventive care (19, 55, 62, 75, 120, 147, 158, 159, 167), immunizations (19), initiating discussions with patients (159), and additional clinical tasks (41, 90, 137).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. One half of the studies required a mandatory response (55, 90, 119, 147, 166) or justification (75, 105, 113, 120, 136, 137, 145) for not adhering to the recommendation; however, there was no significant effect on provider acceptance. Limitations included an inconsistent definition of provider acceptance; small sample sizes; and scarce data on clinical outcomes, such as morbidity, length of stay, or adverse events. We rated this level of evidence as low. Most of these studies were fair quality and were evaluated in academic medical settings with established health IT infrastructures and experienced EHR users, which may limit the generalizability of the findings.
Provider satisfaction with CDSSs was examined in 19 studies (14, 23–25, 37, 43, 80, 86, 105, 109, 112, 119, 127, 128, 141, 151–153, 155, 156, 158, 165). Topics addressed included diagnosis (43, 112, 151, 152), pharmacotherapy (25, 80, 119, 127, 128, 141, 153, 165), chronic disease management (14, 43, 112, 141), laboratory test ordering (37, 80, 105), preventive care (37, 43, 80), and initiating discussions with patients (109).
Comparators included usual care or no CDSS and direct comparison with the same CDSS with additional features. Seven CDSSs required a mandatory response (37, 119, 141, 151, 152, 155) or justification (105, 112) for not adhering to the recommendation. Limitations included the narrow assessment of the role of provider satisfaction with CDSSs on patient-specific outcomes and small sample sizes of clinicians.
Twelve studies demonstrated provider satisfaction with CDSSs (25, 37, 80, 109, 112, 119, 127, 128, 141, 155, 156, 158, 165); 4 showed a significant effect of satisfaction among intervention providers compared with control providers (109, 112, 155, 165). Provider dissatisfaction with CDSSs was also reported in 6 studies (14, 43, 86, 105, 151–153). We rated this level of evidence as moderate. Most studies were good quality and evaluated CDSSs integrated into CPOE or EHR systems in multiple interventions outside of environments with an established and robust health IT. However, most CDSSs were locally developed and implemented in the ambulatory setting.
Seventeen studies examined provider use of CDSSs by using such metrics as the number of times the CDSS was accessed by the clinician or provided a recommendation to the clinician (51, 71, 80, 86, 107, 110, 117, 119, 123, 138, 142, 145, 156, 165, 168–172). Objectives included diagnosis (71, 123, 138), pharmacotherapy (80, 110, 117, 119, 123, 138, 142, 145, 165), chronic disease management (51, 110, 123, 168–172), laboratory test ordering (71, 80, 107, 110), preventive care (71, 80), and additional clinical tasks (86, 110, 156).
Comparators included usual care or no CDSS, direct comparison with the same CDSS with additional features, or comparison of the same CDSS for different conditions. Limitations included sparse data demonstrating how provider use translated into more appropriate patient care and small sample sizes of clinicians. We rated this level of evidence as low.
Among the 12 studies (80, 107, 117, 119, 123, 138, 145, 168–172) that provided statistical data about provider use, 8 (80, 110, 119, 123, 145, 165, 168–170) documented low use (<50% of the clinician's time or of patient visits) or that less than 50% of clinicians used the CDSS or received alerts to guide therapeutic action. Most of these studies were fair quality and evaluated locally developed interventions in multiple community and ambulatory settings. Additional results are available from the technical report at www.effectivehealthcare.ahrq.gov.

Discussion

Our systematic review investigated the continuum of information support for clinical care, including traditional CDSSs, as well as information retrieval systems and knowledge resources developed for access at the point of care. Studies were primarily conducted outside of institutions with an established health IT infrastructure. Most interventions targeted specific medical conditions and were evaluated in single settings.
Clinical decision support had a favorable effect on prescribing treatments, facilitating preventive care services, and ordering clinical studies across diverse venues and systems. This finding contrasts with that of another review (2), which showed that most reports of successful CDSS implementation were based on locally developed systems at 4 sites.
Evidence demonstrating positive effects of CDSSs on clinical and economic outcomes remains surprisingly sparse, although this could be because of the relative difficulty of implementing randomized, controlled trials in real clinical settings, as well as the logistics of measuring the direct clinical effect of CDSSs. Evidence was also limited in showing an effect of CDSSs on clinical workload and efficiency. Furthermore, available evidence is insufficient to draw conclusions about the potential negative effect of implementing decision-support tools, which is necessary to truly fulfill the goal of evaluating these interventions and to better address implementation challenges (173).
Our findings are important in light of the increasing political interest and financial investment of the U.S. government in resources for health IT. Meaningful use of CDSSs needs to be objectively informed about the role that they can and should play in reshaping health care delivery. Further understanding is increasingly important to optimally define their role in the context of meaningful use for EHRs.
Our systematic review has several limitations. The heterogeneity of the studies limited general observations about CDSSs. We minimized this limitation in our meta-analyses by including studies that assessed the same outcome in the same manner. Although this investigation was a comprehensive review of randomized, controlled trials that provided the best evidence on CDSS effectiveness, these studies may provide less information about issues related to CDSS implementation, effect on workflow, and factors affecting usability.
Most studies (76%) evaluated the effectiveness of the intervention by using usual care rather than a direct comparator, which may contribute to more positive results. Finally, we acknowledge the possibility of selective reporting or publication bias. However, a recent review by Buntin and colleagues (173) found that 62% of included studies on health IT reported positive effects where the technology was associated with improvement in 1 or more aspects of care (173). Formal assessment by using funnel plots found no consistent bias for most outcomes, except for ordering or completing of clinical studies, where there was a strong suggestion of publication bias.
Significant research is still required to promote widespread use of CDSSs and to augment their clinical effectiveness. Future studies should investigate how to expand CDSS content to accommodate multiple comorbid conditions simultaneously and to determine which members of the care team should receive clinical decision support, what effect CDSSs have on clinical and economic outcomes, and how CDSSs can be most effectively integrated into workflow and deployed across diverse settings. Further work is also needed to understand how CDSSs can aid in the transformation of care delivery models, such as accountable care organizations and patient-centered medical homes; how to incorporate CDSSs into workflow tools, such as medical registries and provider–provider messaging capabilities; and how to integrate CDSSs with workflow-oriented quality improvement programs.
Promoting extensive use of CDSSs will require a better definition of the clinical decision-support infrastructure. Such infrastructure could include consistent underlying frameworks for describing CDSSs, such as the “Clinical Decision Support Five Rights” (174), to aid in the aggregation and synthesis of results; development and evaluation of models for porting CDSSs across settings; and improved identification of characteristics of the environment and workflow into which a CDSS is deployed, as well as characteristics of the intended users.
In summary, evidence demonstrated the efficacy of CDSSs on health care process outcomes across diverse settings by using both commercially and locally developed systems, but data showing an effect on clinical and economic outcomes were sparse. Broad penetration of clinical decision-support tools will require aggressively seeking a better understanding of what the right information is and when and how it should be delivered to the right person, and a critical examination of the unintended consequences of CDSS implementation.

References

  1. Kawamoto
    K
    ,  
    Houlihan
    CA
    ,  
    Balas
    EA
    ,  
    Lobach
    DF
    .  
    Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success.
    BMJ
    2005
    330
    765
     PubMed
    CrossRef
     PubMed
  2. Chaudhry
    B
    ,  
    Wang
    J
    ,  
    Wu
    S
    ,  
    Maglione
    M
    ,  
    Mojica
    W
    ,  
    Roth
    E
    .  
    et al.
    Systematic review: impact of health information technology on quality, efficiency, and costs of medical care.
    Ann Intern Med
    2006
    144
    742-52
     PubMed
     PubMed
    CrossRef
  3. Bryan
    C
    ,  
    Boren
    SA
    .  
    The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature.
    Inform Prim Care
    2008
    16
    79-91
     PubMed
     PubMed
  4. Garg
    AX
    ,  
    Adhikari
    NK
    ,  
    McDonald
    H
    ,  
    Rosas-Arellano
    MP
    ,  
    Devereaux
    PJ
    ,  
    Beyene
    J
    .  
    et al.
    Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.
    JAMA
    2005
    293
    1223-38
     PubMed
    CrossRef
     PubMed
  5. Grimshaw
    J
    ,  
    Freemantle
    N
    ,  
    Wallace
    S
    ,  
    Russell
    I
    ,  
    Hurwitz
    B
    ,  
    Watt
    I
    .  
    et al.
    Developing and implementing clinical practice guidelines.
    Qual Health Care
    1995
    4
    55-64
     PubMed
     PubMed
    CrossRef
  6. Hunt
    DL
    ,  
    Haynes
    RB
    ,  
    Hanna
    SE
    ,  
    Smith
    K
    .  
    Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review.
    JAMA
    1998
    280
    1339-46
     PubMed
     PubMed
  7. Sintchenko
    V
    ,  
    Magrabi
    F
    ,  
    Tipper
    S
    .  
    Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review.
    Med Inform Internet Med
    2007
    32
    225-40
     PubMed
     PubMed
    CrossRef
  8. Shojania
    KG
    ,  
    Jennings
    A
    ,  
    Mayhew
    A
    ,  
    Ramsay
    C
    ,  
    Eccles
    M
    ,  
    Grimshaw
    J
    .  
    Effect of point-of-care computer reminders on physician behaviour: a systematic review.
    Can Med Assoc J
    2010
    182
    216-25
    CrossRef
  9. Kaushal
    R
    ,  
    Shojania
    KG
    ,  
    Bates
    DW
    .  
    Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review.
    Arch Intern Med
    2003
    163
    1409-16
     PubMed
    CrossRef
     PubMed
  10. Agency for Healthcare Research and Quality. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2011. Accessed at www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318 on 24 January 2012.
  11. Owens
    DK
    ,  
    Lohr
    KN
    ,  
    Atkins
    D
    ,  
    Treadwell
    JR
    ,  
    Reston
    JT
    ,  
    Bass
    EB
    .  
    et al.
    AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health-care program.
    J Clin Epidemiol
    2010
    63
    513-23
     PubMed
    CrossRef
     PubMed
  12. Atkins
    D
    ,  
    Chang
    SM
    ,  
    Gartlehner
    G
    ,  
    Buckley
    DI
    ,  
    Whitlock
    EP
    ,  
    Berliner
    E
    .  
    et al.
    Assessing applicability when comparing medical interventions: AHRQ and the Effective Health Care Program.
    J Clin Epidemiol
    2011
    64
    1198-207
     PubMed
    CrossRef
     PubMed
  13. DerSimonian
    R
    ,  
    Laird
    N
    .  
    Meta-analysis in clinical trials.
    Control Clin Trials
    1986
    7
    177-88
     PubMed
    CrossRef
     PubMed
  14. McCowan
    C
    ,  
    Neville
    RG
    ,  
    Ricketts
    IW
    ,  
    Warner
    FC
    ,  
    Hoskins
    G
    ,  
    Thomas
    GE
    .  
    Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma.
    Med Inform Internet Med
    2001
    26
    191-201
     PubMed
     PubMed
  15. Cavalcanti
    AB
    ,  
    Silva
    E
    ,  
    Pereira
    AJ
    ,  
    Caldeira-Filho
    M
    ,  
    Almeida
    FP
    ,  
    Westphal
    GA
    .  
    et al.
    A randomized controlled trial comparing a computer-assisted insulin infusion protocol with a strict and a conventional protocol for glucose control in critically ill patients.
    J Crit Care
    2009
    24
    371-8
     PubMed
    CrossRef
     PubMed
  16. Kline
    JA
    ,  
    Zeitouni
    RA
    ,  
    Hernandez-Nino
    J
    ,  
    Jones
    AE
    .  
    Randomized trial of computerized quantitative pretest probability in low-risk chest pain patients: effect on safety and resource use.
    Ann Emerg Med
    2009
    53
    727-35.e1
     PubMed
    CrossRef
     PubMed
  17. Kucher
    N
    ,  
    Koo
    S
    ,  
    Quiroz
    R
    ,  
    Cooper
    JM
    ,  
    Paterno
    MD
    ,  
    Soukonnikov
    B
    .  
    et al.
    Electronic alerts to prevent venous thromboembolism among hospitalized patients.
    N Engl J Med
    2005
    352
    969-77
     PubMed
    CrossRef
     PubMed
  18. Zanetti
    G
    ,  
    Flanagan
    HL
    Jr
    ,  
    Cohn
    LH
    ,  
    Giardina
    R
    ,  
    Platt
    R
    .  
    Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room.
    Infect Control Hosp Epidemiol
    2003
    24
    13-6
     PubMed
    CrossRef
     PubMed
  19. McDonald
    CJ
    ,  
    Hui
    SL
    ,  
    Smith
    DM
    ,  
    Tierney
    WM
    ,  
    Cohen
    SJ
    ,  
    Weinberger
    M
    .  
    et al.
    Reminders to physicians from an introspective computer medical record. A two-year randomized trial.
    Ann Intern Med
    1984
    100
    130-8
     PubMed
     PubMed
    CrossRef
  20. Roumie
    CL
    ,  
    Elasy
    TA
    ,  
    Greevy
    R
    ,  
    Griffin
    MR
    ,  
    Liu
    X
    ,  
    Stone
    WJ
    .  
    et al.
    Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial.
    Ann Intern Med
    2006
    145
    165-75
     PubMed
     PubMed
    CrossRef
  21. Paul
    M
    ,  
    Andreassen
    S
    ,  
    Tacconelli
    E
    ,  
    Nielsen
    AD
    ,  
    Almanasreh
    N
    ,  
    Frank
    U
    .  
    et al.
    TREAT Study Group
    Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial.
    J Antimicrob Chemother
    2006
    58
    1238-45
     PubMed
     PubMed
  22. Ansari
    M
    ,  
    Shlipak
    MG
    ,  
    Heidenreich
    PA
    ,  
    Van Ostaeyen
    D
    ,  
    Pohl
    EC
    ,  
    Browner
    WS
    .  
    et al.
    Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure.
    Circulation
    2003
    107
    2799-804
     PubMed
    CrossRef
     PubMed
  23. Graumlich
    JF
    ,  
    Novotny
    NL
    ,  
    Nace
    GS
    ,  
    Aldag
    JC
    .  
    Patient and physician perceptions after software-assisted hospital discharge: cluster randomized trial.
    J Hosp Med
    2009
    4
    356-63
     PubMed
    CrossRef
     PubMed
  24. Graumlich
    JF
    ,  
    Novotny
    NL
    ,  
    Stephen Nace
    G
    ,  
    Kaushal
    H
    ,  
    Ibrahim-Ali
    W
    ,  
    Theivanayagam
    S
    .  
    et al.
    Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial.
    J Hosp Med
    2009
    4
    11-9
     PubMed
    CrossRef
  25. Heidenreich
    PA
    ,  
    Gholami
    P
    ,  
    Sahay
    A
    ,  
    Massie
    B
    ,  
    Goldstein
    MK
    .  
    Clinical reminders attached to echocardiography reports of patients with reduced left ventricular ejection fraction increase use of beta-blockers: a randomized trial.
    Circulation
    2007
    115
    2829-34
     PubMed
    CrossRef
     PubMed
  26. Tierney
    WM
    ,  
    Overhage
    JM
    ,  
    Murray
    MD
    ,  
    Harris
    LE
    ,  
    Zhou
    XH
    ,  
    Eckert
    GJ
    .  
    et al.
    Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial.
    Health Serv Res
    2005
    40
    477-97
     PubMed
    CrossRef
     PubMed
  27. Tierney
    WM
    ,  
    Overhage
    JM
    ,  
    Murray
    MD
    ,  
    Harris
    LE
    ,  
    Zhou
    XH
    ,  
    Eckert
    GJ
    .  
    et al.
    Effects of computerized guidelines for managing heart disease in primary care.
    J Gen Intern Med
    2003
    18
    967-76
     PubMed
    CrossRef
     PubMed
  28. Gilutz
    H
    ,  
    Novack
    L
    ,  
    Shvartzman
    P
    ,  
    Zelingher
    J
    ,  
    Bonneh
    DY
    ,  
    Henkin
    Y
    .  
    et al.
    Computerized community cholesterol control (4C): meeting the challenge of secondary prevention.
    Isr Med Assoc J
    2009
    11
    23-9
     PubMed
     PubMed
  29. Holt
    TA
    ,  
    Thorogood
    M
    ,  
    Griffiths
    F
    ,  
    Munday
    S
    .  
    Protocol for the ‘e-Nudge trial’: a randomised controlled trial of electronic feedback to reduce the cardiovascular risk of individuals in general practice [ISRCTN64828380].
    Trials
    2006
    7
    11
     PubMed
    CrossRef
     PubMed
  30. Holt
    TA
    ,  
    Thorogood
    M
    ,  
    Griffiths
    F
    ,  
    Munday
    S
    ,  
    Friede
    T
    ,  
    Stables
    D
    .  
    Automated electronic reminders to facilitate primary cardiovascular disease prevention: randomised controlled trial.
    Br J Gen Pract
    2010
    60
    137-43
     PubMed
    CrossRef
  31. Khan
    S
    ,  
    Maclean
    CD
    ,  
    Littenberg
    B
    .  
    The effect of the Vermont Diabetes Information System on inpatient and emergency room use: results from a randomized trial.
    Health Outcomes Res Med
    2010
    1
    61-e66
     PubMed
    CrossRef
  32. Maclean
    CD
    ,  
    Gagnon
    M
    ,  
    Callas
    P
    ,  
    Littenberg
    B
    .  
    The Vermont diabetes information system: a cluster randomized trial of a population based decision support system.
    J Gen Intern Med
    2009
    24
    1303-10
     PubMed
    CrossRef
     PubMed
  33. Brier
    ME
    ,  
    Gaweda
    AE
    ,  
    Dailey
    A
    ,  
    Aronoff
    GR
    ,  
    Jacobs
    AA
    .  
    Randomized trial of model predictive control for improved anemia management.
    Clin J Am Soc Nephrol
    2010
    5
    814-20
     PubMed
    CrossRef
     PubMed
  34. Hamilton
    E
    ,  
    Platt
    R
    ,  
    Gauthier
    R
    ,  
    McNamara
    H
    ,  
    Miner
    L
    ,  
    Rothenberg
    S
    .  
    et al.
    The effect of computer-assisted evaluation of labor on cesarean rates.
    J Healthc Qual
    2004
    26
    37-44
     PubMed
    CrossRef
     PubMed
  35. McDonald
    CJ
    ,  
    Hui
    SL
    ,  
    Tierney
    WM
    .  
    Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics.
    MD Comput
    1992
    9
    304-12
     PubMed
     PubMed
  36. Murray
    MD
    ,  
    Harris
    LE
    ,  
    Overhage
    JM
    ,  
    Zhou
    XH
    ,  
    Eckert
    GJ
    ,  
    Smith
    FE
    .  
    et al.
    Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.
    Pharmacotherapy
    2004
    24
    324-37
     PubMed
    CrossRef
     PubMed
  37. Sequist
    TD
    ,  
    Zaslavsky
    AM
    ,  
    Marshall
    R
    ,  
    Fletcher
    RH
    ,  
    Ayanian
    JZ
    .  
    Patient and physician reminders to promote colorectal cancer screening: a randomized controlled trial.
    Arch Intern Med
    2009
    169
    364-71
     PubMed
     PubMed
  38. Subramanian
    U
    ,  
    Fihn
    SD
    ,  
    Weinberger
    M
    ,  
    Plue
    L
    ,  
    Smith
    FE
    ,  
    Udris
    EM
    .  
    et al.
    A controlled trial of including symptom data in computer-based care suggestions for managing patients with chronic heart failure.
    Am J Med
    2004
    116
    375-84
     PubMed
    CrossRef
     PubMed
  39. Kuperman
    GJ
    ,  
    Teich
    JM
    ,  
    Tanasijevic
    MJ
    ,  
    Ma'Luf
    N
    ,  
    Rittenberg
    E
    ,  
    Jha
    A
    .  
    et al.
    Improving response to critical laboratory results with automation: results of a randomized controlled trial.
    J Am Med Inform Assoc
    1999
    6
    512-22
     PubMed
    CrossRef
     PubMed
  40. McGregor
    JC
    ,  
    Weekes
    E
    ,  
    Forrest
    GN
    ,  
    Standiford
    HC
    ,  
    Perencevich
    EN
    ,  
    Furuno
    JP
    .  
    et al.
    Impact of a computerized clinical decision support system on reducing inappropriate antimicrobial use: a randomized controlled trial.
    J Am Med Inform Assoc
    2006
    13
    378-84
     PubMed
    CrossRef
     PubMed
  41. Fihn
    SD
    ,  
    McDonell
    MB
    ,  
    Vermes
    D
    ,  
    Henikoff
    JG
    ,  
    Martin
    DC
    ,  
    Callahan
    CM
    .  
    et al.
    A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics.
    J Gen Intern Med
    1994
    9
    131-9
     PubMed
    CrossRef
     PubMed
  42. Gurwitz
    JH
    ,  
    Field
    TS
    ,  
    Rochon
    P
    ,  
    Judge
    J
    ,  
    Harrold
    LR
    ,  
    Bell
    CM
    .  
    et al.
    Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting.
    J Am Geriatr Soc
    2008
    56
    2225-33
     PubMed
    CrossRef
     PubMed
  43. Apkon
    M
    ,  
    Mattera
    JA
    ,  
    Lin
    Z
    ,  
    Herrin
    J
    ,  
    Bradley
    EH
    ,  
    Carbone
    M
    .  
    et al.
    A randomized outpatient trial of a decision-support information technology tool.
    Arch Intern Med
    2005
    165
    2388-94
     PubMed
    CrossRef
     PubMed
  44. Bertoni
    AG
    ,  
    Bonds
    DE
    ,  
    Chen
    H
    ,  
    Hogan
    P
    ,  
    Crago
    L
    ,  
    Rosenberger
    E
    .  
    et al.
    Impact of a multifaceted intervention on cholesterol management in primary care practices: guideline adherence for heart health randomized trial.
    Arch Intern Med
    2009
    169
    678-86
     PubMed
    CrossRef
     PubMed
  45. Burack
    RC
    ,  
    Gimotty
    PA
    ,  
    George
    J
    ,  
    McBride
    S
    ,  
    Moncrease
    A
    ,  
    Simon
    MS
    .  
    et al.
    How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial.
    Cancer
    1998
    82
    2391-400
     PubMed
    CrossRef
     PubMed
  46. Burack
    RC
    ,  
    Gimotty
    PA
    ,  
    Simon
    M
    ,  
    Moncrease
    A
    ,  
    Dews
    P
    .  
    The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial.
    Prev Med
    2003
    36
    547-54
     PubMed
    CrossRef
     PubMed
  47. Cannon
    DS
    ,  
    Allen
    SN
    .  
    A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline.
    J Am Med Inform Assoc
    2000
    7
    196-203
     PubMed
    CrossRef
     PubMed
  48. Demakis
    JG
    ,  
    Beauchamp
    C
    ,  
    Cull
    WL
    ,  
    Denwood
    R
    ,  
    Eisen
    SA
    ,  
    Lofgren
    R
    .  
    et al.
    Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders.
    JAMA
    2000
    284
    1411-6
     PubMed
    CrossRef
     PubMed
  49. Dexter
    PR
    ,  
    Perkins
    S
    ,  
    Overhage
    JM
    ,  
    Maharry
    K
    ,  
    Kohler
    RB
    ,  
    McDonald
    CJ
    .  
    A computerized reminder system to increase the use of preventive care for hospitalized patients.
    N Engl J Med
    2001
    345
    965-70
     PubMed
    CrossRef
     PubMed
  50. Dexter
    PR
    ,  
    Perkins
    SM
    ,  
    Maharry
    KS
    ,  
    Jones
    K
    ,  
    McDonald
    CJ
    .  
    Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial.
    JAMA
    2004
    292
    2366-71
     PubMed
    CrossRef
     PubMed
  51. Eccles
    M
    ,  
    McColl
    E
    ,  
    Steen
    N
    ,  
    Rousseau
    N
    ,  
    Grimshaw
    J
    ,  
    Parkin
    D
    .  
    et al.
    Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.
    BMJ
    2002
    325
    941
     PubMed
    CrossRef
     PubMed
  52. Frank
    O
    ,  
    Litt
    J
    ,  
    Beilby
    J
    .  
    Opportunistic electronic reminders. Improving performance of preventive care in general practice.
    Aust Fam Physician
    2004
    33
    87-90
     PubMed
     PubMed
  53. Fretheim
    A
    ,  
    Aaserud
    M
    ,  
    Oxman
    AD
    .  
    Rational prescribing in primary care (RaPP): economic evaluation of an intervention to improve professional practice.
    PLoS Med
    2006
    3
    216
     PubMed
    CrossRef
  54. Fretheim
    A
    ,  
    Oxman
    AD
    ,  
    Håvelsrud
    K
    ,  
    Treweek
    S
    ,  
    Kristoffersen
    DT
    ,  
    Bjørndal
    A
    .  
    Rational prescribing in primary care (RaPP): a cluster randomized trial of a tailored intervention.
    PLoS Med
    2006
    3
    134
     PubMed
    CrossRef
  55. Litzelman
    DK
    ,  
    Dittus
    RS
    ,  
    Miller
    ME
    ,  
    Tierney
    WM
    .  
    Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols.
    J Gen Intern Med
    1993
    8
    311-7
     PubMed
    CrossRef
     PubMed
  56. McDowell
    I
    ,  
    Newell
    C
    ,  
    Rosser
    W
    .  
    Comparison of three methods of recalling patients for influenza vaccination.
    CMAJ
    1986
    135
    991-7
     PubMed
     PubMed
  57. McDowell
    I
    ,  
    Newell
    C
    ,  
    Rosser
    W
    .  
    Computerized reminders to encourage cervical screening in family practice.
    J Fam Pract
    1989
    28
    420-4
     PubMed
     PubMed
  58. Overhage
    JM
    ,  
    Tierney
    WM
    ,  
    McDonald
    CJ
    .  
    Computer reminders to implement preventive care guidelines for hospitalized patients.
    Arch Intern Med
    1996
    156
    1551-6
     PubMed
    CrossRef
     PubMed
  59. Price
    M
    .  
    Can hand-held computers improve adherence to guidelines? A (Palm) Pilot study of family doctors in British Columbia.
    Can Fam Physician
    2005
    51
    1506-7
     PubMed
     PubMed
  60. Taylor
    V
    ,  
    Thompson
    B
    ,  
    Lessler
    D
    ,  
    Yasui
    Y
    ,  
    Montano
    D
    ,  
    Johnson
    KM
    .  
    et al.
    A clinic-based mammography intervention targeting inner-city women.
    J Gen Intern Med
    1999
    14
    104-11
     PubMed
    CrossRef
     PubMed
  61. Unrod
    M
    ,  
    Smith
    M
    ,  
    Spring
    B
    ,  
    DePue
    J
    ,  
    Redd
    W
    ,  
    Winkel
    G
    .  
    Randomized controlled trial of a computer-based, tailored intervention to increase smoking cessation counseling by primary care physicians.
    J Gen Intern Med
    2007
    22
    478-84
     PubMed
    CrossRef
     PubMed
  62. Dykes
    PC
    ,  
    Carroll
    DL
    ,  
    Hurley
    A
    ,  
    Lipsitz
    S
    ,  
    Benoit
    A
    ,  
    Chang
    F
    .  
    et al.
    Fall prevention in acute care hospitals: a randomized trial.
    JAMA
    2010
    304
    1912-8
     PubMed
    CrossRef
     PubMed
  63. Chambers
    CV
    ,  
    Balaban
    DJ
    ,  
    Carlson
    BL
    ,  
    Ungemack
    JA
    ,  
    Grasberger
    DM
    .  
    Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines.
    J Fam Pract
    1989
    29
    273-80
     PubMed
     PubMed
  64. Burack
    RC
    ,  
    Gimotty
    PA
    .  
    Promoting screening mammography in inner-city settings. The sustained effectiveness of computerized reminders in a randomized controlled trial.
    Med Care
    1997
    35
    921-31
     PubMed
    CrossRef
     PubMed
  65. Burack
    RC
    ,  
    Gimotty
    PA
    ,  
    George
    J
    ,  
    Stengle
    W
    ,  
    Warbasse
    L
    ,  
    Moncrease
    A
    .  
    Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography.
    Med Care
    1994
    32
    609-24
     PubMed
    CrossRef
     PubMed
  66. Fiks
    AG
    ,  
    Hunter
    KF
    ,  
    Localio
    AR
    ,  
    Grundmeier
    RW
    ,  
    Bryant-Stephens
    T
    ,  
    Luberti
    AA
    .  
    et al.
    Impact of electronic health record-based alerts on influenza vaccination for children with asthma.
    Pediatrics
    2009
    124
    159-69
     PubMed
    CrossRef
     PubMed
  67. Flanagan
    JR
    ,  
    Doebbeling
    BN
    ,  
    Dawson
    J
    ,  
    Beekmann
    S
    .  
    Randomized study of online vaccine reminders in adult primary care.
    Proc AMIA Symp
    1999
    755-9
     PubMed
  68. Fordham
    D
    ,  
    McPhee
    SJ
    ,  
    Bird
    JA
    ,  
    Rodnick
    JE
    ,  
    Detmer
    WM
    .  
    The Cancer Prevention Reminder System.
    MD Comput
    1990
    7
    289-95
     PubMed
     PubMed
  69. McPhee
    SJ
    ,  
    Bird
    JA
    ,  
    Jenkins
    CN
    ,  
    Fordham
    D
    .  
    Promoting cancer screening. A randomized, controlled trial of three interventions.
    Arch Intern Med
    1989
    149
    1866-72
     PubMed
    CrossRef
     PubMed
  70. Gill
    JM
    ,  
    Chen
    YX
    ,  
    Glutting
    JJ
    ,  
    Diamond
    JJ
    ,  
    Lieberman
    MI
    .  
    Impact of decision support in electronic medical records on lipid management in primary care.
    Popul Health Manag
    2009
    12
    221-6
     PubMed
    CrossRef
     PubMed
  71. Hobbs
    FD
    ,  
    Delaney
    BC
    ,  
    Carson
    A
    ,  
    Kenkre
    JE
    .  
    A prospective controlled trial of computerized decision support for lipid management in primary care.
    Fam Pract
    1996
    13
    133-7
     PubMed
     PubMed
  72. Holbrook
    A
    ,  
    Thabane
    L
    ,  
    Keshavjee
    K
    ,  
    Dolovich
    L
    ,  
    Bernstein
    B
    ,  
    Chan
    D
    .  
    et al.
    COMPETE II Investigators
    Individualized electronic decision support and reminders to improve diabetes care in the community: COMPETE II randomized trial.
    CMAJ
    2009
    181
    37-44
     PubMed
     PubMed
    CrossRef
  73. Kenealy
    T
    ,  
    Arroll
    B
    ,  
    Petrie
    KJ
    .  
    Patients and computers as reminders to screen for diabetes in family practice. Randomized-controlled trial.
    J Gen Intern Med
    2005
    20
    916-21
     PubMed
    CrossRef
     PubMed
  74. Lobach
    DF
    ,  
    Hammond
    WE
    .  
    Development and evaluation of a Computer-Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus.
    Proc Annu Symp Comput Appl Med Care
    1994
    787-91
     PubMed
  75. Ornstein
    SM
    ,  
    Garr
    DR
    ,  
    Jenkins
    RG
    ,  
    Rust
    PF
    ,  
    Arnon
    A
    .  
    Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services.
    J Fam Pract
    1991
    32
    82-90
     PubMed
     PubMed
  76. Peterson
    KA
    ,  
    Radosevich
    DM
    ,  
    O'Connor
    PJ
    ,  
    Nyman
    JA
    ,  
    Prineas
    RJ
    ,  
    Smith
    SA
    .  
    et al.
    Improving diabetes care in practice: findings from the TRANSLATE trial.
    Diabetes Care
    2008
    31
    2238-43
     PubMed
    CrossRef
     PubMed
  77. Reeve
    JF
    ,  
    Tenni
    PC
    ,  
    Peterson
    GM
    .  
    An electronic prompt in dispensing software to promote clinical interventions by community pharmacists: a randomized controlled trial.
    Br J Clin Pharmacol
    2008
    65
    377-85
     PubMed
    CrossRef
     PubMed
  78. Rosser
    WW
    ,  
    Hutchison
    BG
    ,  
    McDowell
    I
    ,  
    Newell
    C
    .  
    Use of reminders to increase compliance with tetanus booster vaccination.
    CMAJ
    1992
    146
    911-7
     PubMed
     PubMed
  79. Rosser
    WW
    ,  
    McDowell
    I
    ,  
    Newell
    C
    .  
    Use of reminders for preventive procedures in family medicine.
    CMAJ
    1991
    145
    807-14
     PubMed
     PubMed
  80. Sequist
    TD
    ,  
    Gandhi
    TK
    ,  
    Karson
    AS
    ,  
    Fiskio
    JM
    ,  
    Bugbee
    D
    ,  
    Sperling
    M
    .  
    et al.
    A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease.
    J Am Med Inform Assoc
    2005
    12
    431-7
     PubMed
    CrossRef
     PubMed
  81. Tierney
    WM
    ,  
    Hui
    SL
    ,  
    McDonald
    CJ
    .  
    Delayed feedback of physician performance versus immediate reminders to perform preventive care. Effects on physician compliance.
    Med Care
    1986
    24
    659-66
     PubMed
    CrossRef
     PubMed
  82. van Wyk
    JT
    ,  
    van Wijk
    MA
    ,  
    Sturkenboom
    MC
    ,  
    Mosseveld
    M
    ,  
    Moorman
    PW
    ,  
    van der Lei
    J
    .  
    Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial.
    Circulation
    2008
    117
    371-8
     PubMed
    CrossRef
     PubMed
  83. Bates
    DW
    ,  
    Kuperman
    GJ
    ,  
    Rittenberg
    E
    ,  
    Teich
    JM
    ,  
    Fiskio
    J
    ,  
    Ma'luf
    N
    .  
    et al.
    A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests.
    Am J Med
    1999
    106
    144-50
     PubMed
    CrossRef
     PubMed
  84. Bell
    LM
    ,  
    Grundmeier
    R
    ,  
    Localio
    R
    ,  
    Zorc
    J
    ,  
    Fiks
    AG
    ,  
    Zhang
    X
    .  
    et al.
    Electronic health record-based decision support to improve asthma care: a cluster-randomized trial.
    Pediatrics
    2010
    125
    770-7
     PubMed
    CrossRef
  85. Downs
    M
    ,  
    Turner
    S
    ,  
    Bryans
    M
    ,  
    Wilcock
    J
    ,  
    Keady
    J
    ,  
    Levin
    E
    .  
    et al.
    Effectiveness of educational interventions in improving detection and management of dementia in primary care: cluster randomised controlled study.
    BMJ
    2006
    332
    692-6
     PubMed
    CrossRef
     PubMed
  86. Emery
    J
    ,  
    Morris
    H
    ,  
    Goodchild
    R
    ,  
    Fanshawe
    T
    ,  
    Prevost
    AT
    ,  
    Bobrow
    M
    .  
    et al.
    The GRAIDS Trial: a cluster randomised controlled trial of computer decision support for the management of familial cancer risk in primary care.
    Br J Cancer
    2007
    97
    486-93
     PubMed
    CrossRef
     PubMed
  87. Feldstein
    AC
    ,  
    Smith
    DH
    ,  
    Perrin
    N
    ,  
    Yang
    X
    ,  
    Rix
    M
    ,  
    Raebel
    MA
    .  
    et al.
    Improved therapeutic monitoring with several interventions: a randomized trial.
    Arch Intern Med
    2006
    166
    1848-54
     PubMed
     PubMed
  88. Flottorp
    S
    ,  
    Oxman
    AD
    ,  
    Håvelsrud
    K
    ,  
    Treweek
    S
    ,  
    Herrin
    J
    .  
    Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat.
    BMJ
    2002
    325
    367
     PubMed
    CrossRef
     PubMed
  89. Greiver
    M
    ,  
    Drummond
    N
    ,  
    White
    D
    ,  
    Weshler
    J
    ,  
    Moineddin
    R
    .  
    North Toronto Primary Care Research Network (Nortren)
    Angina on the Palm: randomized controlled pilot trial of Palm PDA software for referrals for cardiac testing.
    Can Fam Physician
    2005
    51
    382-3
     PubMed
     PubMed
  90. Harpole
    LH
    ,  
    Khorasani
    R
    ,  
    Fiskio
    J
    ,  
    Kuperman
    GJ
    ,  
    Bates
    DW
    .  
    Automated evidence-based critiquing of orders for abdominal radiographs: impact on utilization and appropriateness.
    J Am Med Inform Assoc
    1997
    4
    511-21
     PubMed
    CrossRef
     PubMed
  91. Lee
    NJ
    ,  
    Chen
    ES
    ,  
    Currie
    LM
    ,  
    Donovan
    M
    ,  
    Hall
    EK
    ,  
    Jia
    H
    .  
    et al.
    The effect of a mobile clinical decision support system on the diagnosis of obesity and overweight in acute and primary care encounters.
    ANS Adv Nurs Sci
    2009
    32
    211-21
     PubMed
     PubMed
    CrossRef
  92. Lo
    HG
    ,  
    Matheny
    ME
    ,  
    Seger
    DL
    ,  
    Bates
    DW
    ,  
    Gandhi
    TK
    .  
    Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care.
    J Am Med Inform Assoc
    2009
    16
    66-71
     PubMed
    CrossRef
     PubMed
  93. Matheny
    ME
    ,  
    Sequist
    TD
    ,  
    Seger
    AC
    ,  
    Fiskio
    JM
    ,  
    Sperling
    M
    ,  
    Bugbee
    D
    .  
    et al.
    A randomized trial of electronic clinical reminders to improve medication laboratory monitoring.
    J Am Med Inform Assoc
    2008
    15
    424-9
     PubMed
    CrossRef
     PubMed
  94. McDonald
    CJ
    .  
    Use of a computer to detect and respond to clinical events: its effect on clinician behavior.
    Ann Intern Med
    1976
    84
    162-7
     PubMed
     PubMed
    CrossRef
  95. McDowell
    I
    ,  
    Newell
    C
    ,  
    Rosser
    W
    .  
    A randomized trial of computerized reminders for blood pressure screening in primary care.
    Med Care
    1989
    27
    297-305
     PubMed
    CrossRef
     PubMed
  96. Palen
    TE
    ,  
    Raebel
    M
    ,  
    Lyons
    E
    ,  
    Magid
    DM
    .  
    Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders.
    Am J Manag Care
    2006
    12
    389-95
     PubMed
     PubMed
  97. Palen
    TE
    ,  
    Price
    DW
    ,  
    Snyder
    AJ
    ,  
    Shetterly
    SM
    .  
    Computerized alert reduced D-dimer testing in the elderly.
    Am J Manag Care
    2010
    16
    267-75
     PubMed
  98. Player
    MS
    ,  
    Gill
    JM
    ,  
    Mainous
    AG
    3rd
    ,  
    Everett
    CJ
    ,  
    Koopman
    RJ
    ,  
    Diamond
    JJ
    .  
    et al.
    An electronic medical record-based intervention to improve quality of care for gastro-esophageal reflux disease (GERD) and atypical presentations of GERD.
    Qual Prim Care
    2010
    18
    223-9
     PubMed
     PubMed
  99. Raebel
    MA
    ,  
    Lyons
    EE
    ,  
    Chester
    EA
    ,  
    Bodily
    MA
    ,  
    Kelleher
    JA
    ,  
    Long
    CL
    .  
    et al.
    Improving laboratory monitoring at initiation of drug therapy in ambulatory care: a randomized trial.
    Arch Intern Med
    2005
    165
    2395-401
     PubMed
    CrossRef
     PubMed
  100. Raebel
    MA
    ,  
    Chester
    EA
    ,  
    Newsom
    EE
    ,  
    Lyons
    EE
    ,  
    Kelleher
    JA
    ,  
    Long
    C
    .  
    et al.
    Randomized trial to improve laboratory safety monitoring of ongoing drug therapy in ambulatory patients.
    Pharmacotherapy
    2006
    26
    619-26
     PubMed
    CrossRef
     PubMed
  101. Roukema
    J
    ,  
    Steyerberg
    EW
    ,  
    van der Lei
    J
    ,  
    Moll
    HA
    .  
    Randomized trial of a clinical decision support system: impact on the management of children with fever without apparent source.
    J Am Med Inform Assoc
    2008
    15
    107-13
     PubMed
     PubMed
    CrossRef
  102. Roy
    PM
    ,  
    Durieux
    P
    ,  
    Gillaizeau
    F
    ,  
    Legall
    C
    ,  
    Armand-Perroux
    A
    ,  
    Martino
    L
    .  
    et al.
    A computerized handheld decision-support system to improve pulmonary embolism diagnosis: a randomized trial.
    Ann Intern Med
    2009
    151
    677-86
     PubMed
     PubMed
    CrossRef
  103. Schriefer
    SP
    ,  
    Landis
    SE
    ,  
    Turbow
    DJ
    ,  
    Patch
    SC
    .  
    Effect of a computerized body mass index prompt on diagnosis and treatment of adult obesity.
    Fam Med
    2009
    41
    502-7
     PubMed
     PubMed
  104. Stiell
    IG
    ,  
    Clement
    CM
    ,  
    Grimshaw
    J
    ,  
    Brison
    RJ
    ,  
    Rowe
    BH
    ,  
    Schull
    MJ
    .  
    et al.
    Implementation of the Canadian C-Spine Rule: prospective 12 centre cluster randomised trial.
    BMJ
    2009
    339
    b4146
     PubMed
    CrossRef
     PubMed
  105. Sundaram
    V
    ,  
    Lazzeroni
    LC
    ,  
    Douglass
    LR
    ,  
    Sanders
    GD
    ,  
    Tempio
    P
    ,  
    Owens
    DK
    .  
    A randomized trial of computer-based reminders and audit and feedback to improve HIV screening in a primary care setting.
    Int J STD AIDS
    2009
    20
    527-33
     PubMed
     PubMed
  106. Tierney
    WM
    ,  
    McDonald
    CJ
    ,  
    Martin
    DK
    ,  
    Rogers
    MP
    .  
    Computerized display of past test results. Effect on outpatient testing.
    Ann Intern Med
    1987
    107
    569-74
     PubMed
     PubMed
    CrossRef
  107. van Wijk
    MA
    ,  
    van der Lei
    J
    ,  
    Mosseveld
    M
    ,  
    Bohnen
    AM
    ,  
    van Bemmel
    JH
    .  
    Assessment of decision support for blood test ordering in primary care. a randomized trial.
    Ann Intern Med
    2001
    134
    274-81
     PubMed
     PubMed
    CrossRef
  108. Walker
    J
    ,  
    Fairley
    CK
    ,  
    Walker
    SM
    ,  
    Gurrin
    LC
    ,  
    Gunn
    JM
    ,  
    Pirotta
    MV
    .  
    et al.
    Computer reminders for Chlamydia screening in general practice: a randomized controlled trial.
    Sex Transm Dis
    2010
    37
    445-50
     PubMed
     PubMed
  109. Wilson
    BJ
    ,  
    Torrance
    N
    ,  
    Mollison
    J
    ,  
    Watson
    MS
    ,  
    Douglas
    A
    ,  
    Miedzybrodzka
    Z
    .  
    et al.
    Cluster randomized trial of a multifaceted primary care decision-support intervention for inherited breast cancer risk.
    Fam Pract
    2006
    23
    537-44
     PubMed
    CrossRef
     PubMed
  110. Bourgeois
    FC
    ,  
    Linder
    J
    ,  
    Johnson
    SA
    ,  
    Co
    JP
    ,  
    Fiskio
    J
    ,  
    Ferris
    TG
    .  
    Impact of a computerized template on antibiotic prescribing for acute respiratory infections in children and adolescents.
    Clin Pediatr (Phila)
    2010
    49
    976-83
     PubMed
    CrossRef
     PubMed
  111. Christakis
    DA
    ,  
    Zimmerman
    FJ
    ,  
    Wright
    JA
    ,  
    Garrison
    MM
    ,  
    Rivara
    FP
    ,  
    Davis
    RL
    .  
    A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children.
    Pediatrics
    2001
    107
    15
     PubMed
    CrossRef
  112. Co
    JP
    ,  
    Johnson
    SA
    ,  
    Poon
    EG
    ,  
    Fiskio
    J
    ,  
    Rao
    SR
    ,  
    Van Cleave
    J
    .  
    et al.
    Electronic health record decision support and quality of care for children with ADHD.
    Pediatrics
    2010
    126
    239-46
     PubMed
    CrossRef
     PubMed
  113. Cobos
    A
    ,  
    Vilaseca
    J
    ,  
    Asenjo
    C
    ,  
    Pedro-Botet
    J
    ,  
    Sánchez
    E
    ,  
    Val
    A
    .  
    et al.
    Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: report of a cluster-randomized trial.
    Disease Management & Health Outcomes
    2005
    13
    421-32
    CrossRef
  114. Davis
    RL
    ,  
    Wright
    J
    ,  
    Chalmers
    F
    ,  
    Levenson
    L
    ,  
    Brown
    JC
    ,  
    Lozano
    P
    .  
    et al.
    A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics.
    PLoS Clin Trials
    2007
    2
    25
     PubMed
    CrossRef
  115. Feldstein
    A
    ,  
    Elmer
    PJ
    ,  
    Smith
    DH
    ,  
    Herson
    M
    ,  
    Orwoll
    E
    ,  
    Chen
    C
    .  
    et al.
    Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial.
    J Am Geriatr Soc
    2006
    54
    450-7
     PubMed
    CrossRef
     PubMed
  116. Field
    TS
    ,  
    Rochon
    P
    ,  
    Lee
    M
    ,  
    Gavendo
    L
    ,  
    Baril
    JL
    ,  
    Gurwitz
    JH
    .  
    Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency.
    J Am Med Inform Assoc
    2009
    16
    480-5
     PubMed
    CrossRef
     PubMed
  117. Filippi
    A
    ,  
    Sabatini
    A
    ,  
    Badioli
    L
    ,  
    Samani
    F
    ,  
    Mazzaglia
    G
    ,  
    Catapano
    A
    .  
    et al.
    Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial.
    Diabetes Care
    2003
    26
    1497-500
     PubMed
    CrossRef
     PubMed
  118. Fitzmaurice
    DA
    ,  
    Hobbs
    FD
    ,  
    Murray
    ET
    ,  
    Holder
    RL
    ,  
    Allan
    TF
    ,  
    Rose
    PE
    .  
    Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial.
    Arch Intern Med
    2000
    160
    2343-8
     PubMed
    CrossRef
     PubMed
  119. Fortuna
    RJ
    ,  
    Zhang
    F
    ,  
    Ross-Degnan
    D
    ,  
    Campion
    FX
    ,  
    Finkelstein
    JA
    ,  
    Kotch
    JB
    .  
    et al.
    Reducing the prescribing of heavily marketed medications: a randomized controlled trial.
    J Gen Intern Med
    2009
    24
    897-903
     PubMed
    CrossRef
     PubMed
  120. Goud
    R
    ,  
    de Keizer
    NF
    ,  
    ter Riet
    G
    ,  
    Wyatt
    JC
    ,  
    Hasman
    A
    ,  
    Hellemans
    IM
    .  
    et al.
    Effect of guideline based computerised decision support on decision making of multidisciplinary teams: cluster randomised trial in cardiac rehabilitation.
    BMJ
    2009
    338
    b1440
     PubMed
    CrossRef
     PubMed
  121. Hicks
    LS
    ,  
    Sequist
    TD
    ,  
    Ayanian
    JZ
    ,  
    Shaykevich
    S
    ,  
    Fairchild
    DG
    ,  
    Orav
    EJ
    .  
    et al.
    Impact of computerized decision support on blood pressure management and control: a randomized controlled trial.
    J Gen Intern Med
    2008
    23
    429-41
     PubMed
     PubMed
  122. Krall
    MA
    ,  
    Traunweiser
    K
    ,  
    Towery
    W
    .  
    Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis.
    Stud Health Technol Inform
    2004
    107
    135-9
     PubMed
     PubMed
  123. Linder
    JA
    ,  
    Rigotti
    NA
    ,  
    Schneider
    LI
    ,  
    Kelley
    JH
    ,  
    Brawarsky
    P
    ,  
    Haas
    JS
    .  
    An electronic health record-based intervention to improve tobacco treatment in primary care: a cluster-randomized controlled trial.
    Arch Intern Med
    2009
    169
    781-7
     PubMed
    CrossRef
     PubMed
  124. Locatelli
    F
    ,  
    Covic
    A
    ,  
    Macdougall
    IC
    ,  
    Scherhag
    A
    ,  
    Wiecek
    A
    .  
    ORAMA Study Group
    Effect of computer-assisted European Best Practice Guideline implementation on adherence and target attainment: ORAMA results.
    J Nephrol
    2009
    22
    662-74
     PubMed
     PubMed
  125. Manotti
    C
    ,  
    Moia
    M
    ,  
    Palareti
    G
    ,  
    Pengo
    V
    ,  
    Ria
    L
    ,  
    Dettori
    AG
    .  
    Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment).
    Haematologica
    2001
    86
    1060-70
     PubMed
     PubMed
  126. Marco
    F
    ,  
    Sedano
    C
    ,  
    Bermúdez
    A
    ,  
    López-Duarte
    M
    ,  
    Fernández-Fontecha
    E
    ,  
    Zubizarreta
    A
    .  
    A prospective controlled study of a computer-assisted acenocoumarol dosage program.
    Pathophysiol Haemost Thromb
    2003
    33
    59-63
     PubMed
    CrossRef
     PubMed
  127. Martens
    JD
    ,  
    van der Aa
    A
    ,  
    Panis
    B
    ,  
    van der Weijden
    T
    ,  
    Winkens
    RA
    ,  
    Severens
    JL
    .  
    Design and evaluation of a computer reminder system to improve prescribing behaviour of GPs.
    Stud Health Technol Inform
    2006
    124
    617-23
     PubMed
     PubMed
  128. Martens
    JD
    ,  
    van der Weijden
    T
    ,  
    Severens
    JL
    ,  
    de Clercq
    PA
    ,  
    de Bruijn
    DP
    ,  
    Kester
    AD
    .  
    et al.
    The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial.
    Int J Med Inform
    2007
    76
    Suppl 3
    S403-16
     PubMed
    CrossRef
     PubMed
  129. Montgomery
    AA
    ,  
    Fahey
    T
    ,  
    Peters
    TJ
    ,  
    MacIntosh
    C
    ,  
    Sharp
    DJ
    .  
    Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial.
    BMJ
    2000
    320
    686-90
     PubMed
    CrossRef
     PubMed
  130. Overhage
    JM
    ,  
    Tierney
    WM
    ,  
    Zhou
    XH
    ,  
    McDonald
    CJ
    .  
    A randomized trial of “corollary orders” to prevent errors of omission.
    J Am Med Inform Assoc
    1997
    4
    364-75
     PubMed
    CrossRef
     PubMed
  131. Peterson
    JF
    ,  
    Rosenbaum
    BP
    ,  
    Waitman
    LR
    ,  
    Habermann
    R
    ,  
    Powers
    J
    ,  
    Harrell
    D
    .  
    et al.
    Physicians' response to guided geriatric dosing: initial results from a randomized trial.
    Stud Health Technol Inform
    2007
    129
    1037-40
     PubMed
     PubMed
  132. Phillips
    LS
    ,  
    Ziemer
    DC
    ,  
    Doyle
    JP
    ,  
    Barnes
    CS
    ,  
    Kolm
    P
    ,  
    Branch
    WT
    .  
    et al.
    An endocrinologist-supported intervention aimed at providers improves diabetes management in a primary care site: improving primary care of African Americans with diabetes (IPCAAD) 7.
    Diabetes Care
    2005
    28
    2352-60
     PubMed
    CrossRef
     PubMed
  133. Ziemer
    DC
    ,  
    Doyle
    JP
    ,  
    Barnes
    CS
    ,  
    Branch
    WT
    Jr
    ,  
    Cook
    CB
    ,  
    El-Kebbi
    IM
    .  
    et al.
    An intervention to overcome clinical inertia and improve diabetes mellitus control in a primary care setting: Improving Primary Care of African Americans with Diabetes (IPCAAD) 8.
    Arch Intern Med
    2006
    166
    507-13
     PubMed
     PubMed
    CrossRef
  134. Raebel
    MA
    ,  
    Charles
    J
    ,  
    Dugan
    J
    ,  
    Carroll
    NM
    ,  
    Korner
    EJ
    ,  
    Brand
    DW
    .  
    et al.
    Randomized trial to improve prescribing safety in ambulatory elderly patients.
    J Am Geriatr Soc
    2007
    55
    977-85
     PubMed
    CrossRef
     PubMed
  135. Rood
    E
    ,  
    Bosman
    RJ
    ,  
    van der Spoel
    JI
    ,  
    Taylor
    P
    ,  
    Zandstra
    DF
    .  
    Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation.
    J Am Med Inform Assoc
    2005
    12
    172-80
     PubMed
     PubMed
  136. Rossi
    RA
    ,  
    Every
    NR
    .  
    A computerized intervention to decrease the use of calcium channel blockers in hypertension.
    J Gen Intern Med
    1997
    12
    672-8
     PubMed
    CrossRef
     PubMed
  137. Rothschild
    JM
    ,  
    McGurk
    S
    ,  
    Honour
    M
    ,  
    Lu
    L
    ,  
    McClendon
    AA
    ,  
    Srivastava
    P
    .  
    et al.
    Assessment of education and computerized decision support interventions for improving transfusion practice.
    Transfusion
    2007
    47
    228-39
     PubMed
    CrossRef
     PubMed
  138. Samore
    MH
    ,  
    Bateman
    K
    ,  
    Alder
    SC
    ,  
    Hannah
    E
    ,  
    Donnelly
    S
    ,  
    Stoddard
    GJ
    .  
    et al.
    Clinical decision support and appropriateness of antimicrobial prescribing: a randomized trial.
    JAMA
    2005
    294
    2305-14
     PubMed
    CrossRef
     PubMed
  139. Shojania
    KG
    ,  
    Yokoe
    D
    ,  
    Platt
    R
    ,  
    Fiskio
    J
    ,  
    Ma'luf
    N
    ,  
    Bates
    DW
    .  
    Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial.
    J Am Med Inform Assoc
    1998
    5
    554-62
     PubMed
    CrossRef
     PubMed
  140. Simon
    SR
    ,  
    Smith
    DH
    ,  
    Feldstein
    AC
    ,  
    Perrin
    N
    ,  
    Yang
    X
    ,  
    Zhou
    Y
    .  
    et al.
    Computerized prescribing alerts and group academic detailing to reduce the use of potentially inappropriate medications in older people.
    J Am Geriatr Soc
    2006
    54
    963-8
     PubMed
    CrossRef
     PubMed
  141. Smith
    SA
    ,  
    Shah
    ND
    ,  
    Bryant
    SC
    ,  
    Christianson
    TJ
    ,  
    Bjornsen
    SS
    ,  
    Giesler
    PD
    .  
    et al.
    Evidens Research Group
    Chronic care model and shared care in diabetes: randomized trial of an electronic decision support system.
    Mayo Clin Proc
    2008
    83
    747-57
     PubMed
     PubMed
    CrossRef
  142. Strom
    BL
    ,  
    Schinnar
    R
    ,  
    Aberra
    F
    ,  
    Bilker
    W
    ,  
    Hennessy
    S
    ,  
    Leonard
    CE
    .  
    et al.
    Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial.
    Arch Intern Med
    2010
    170
    1578-83
     PubMed
    CrossRef
     PubMed
  143. Strom
    BL
    ,  
    Schinnar
    R
    ,  
    Bilker
    W
    ,  
    Hennessy
    S
    ,  
    Leonard
    CE
    ,  
    Pifer
    E
    .  
    Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co-prescribing as a test case.
    J Am Med Inform Assoc
    2010
    17
    411-5
     PubMed
    CrossRef
     PubMed
  144. Tamblyn
    R
    ,  
    Huang
    A
    ,  
    Perreault
    R
    ,  
    Jacques
    A
    ,  
    Roy
    D
    ,  
    Hanley
    J
    .  
    et al.
    The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care.
    CMAJ
    2003
    169
    549-56
     PubMed
     PubMed
  145. Tamblyn
    R
    ,  
    Huang
    A
    ,  
    Taylor
    L
    ,  
    Kawasumi
    Y
    ,  
    Bartlett
    G
    ,  
    Grad
    R
    .  
    et al.
    A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care.
    J Am Med Inform Assoc
    2008
    15
    430-8
     PubMed
    CrossRef
     PubMed
  146. Tamblyn
    R
    ,  
    Reidel
    K
    ,  
    Huang
    A
    ,  
    Taylor
    L
    ,  
    Winslade
    N
    ,  
    Bartlett
    G
    .  
    et al.
    Increasing the detection and response to adherence problems with cardiovascular medication in primary care through computerized drug management systems: a randomized controlled trial.
    Med Decis Making
    2010
    30
    176-88
     PubMed
    CrossRef
     PubMed
  147. Terrell
    KM
    ,  
    Perkins
    AJ
    ,  
    Dexter
    PR
    ,  
    Hui
    SL
    ,  
    Callahan
    CM
    ,  
    Miller
    DK
    .  
    Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: a randomized, controlled trial.
    J Am Geriatr Soc
    2009
    57
    1388-94
     PubMed
    CrossRef
     PubMed
  148. Terrell
    KM
    ,  
    Perkins
    AJ
    ,  
    Hui
    SL
    ,  
    Callahan
    CM
    ,  
    Dexter
    PR
    ,  
    Miller
    DK
    .  
    Computerized decision support for medication dosing in renal insufficiency: a randomized, controlled trial.
    Ann Emerg Med
    2010
    56
    623-9
     PubMed
    CrossRef
     PubMed
  149. Vadher
    BD
    ,  
    Patterson
    DL
    ,  
    Leaning
    M
    .  
    Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians.
    Clin Lab Haematol
    1997
    19
    203-7
     PubMed
    CrossRef
     PubMed
  150. Vadher
    B
    ,  
    Patterson
    DL
    ,  
    Leaning
    M
    .  
    Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial.
    BMJ
    1997
    314
    1252-6
     PubMed
    CrossRef
     PubMed
  151. Vissers
    MC
    ,  
    Biert
    J
    ,  
    van der Linden
    CJ
    ,  
    Hasman
    A
    .  
    Effects of a supportive protocol processing system (ProtoVIEW) on clinical behaviour of residents in the accident and emergency department.
    Comput Methods Programs Biomed
    1996
    49
    177-84
     PubMed
    CrossRef
     PubMed
  152. Vissers
    MC
    ,  
    Hasman
    A
    ,  
    van der Linden
    CJ
    .  
    Protocol processing system (ProtoVIEW) to support residents at the emergency ward.
    Comput Methods Programs Biomed
    1995
    48
    53-8
     PubMed
    CrossRef
     PubMed
  153. Weir
    CJ
    ,  
    Lees
    KR
    ,  
    MacWalter
    RS
    ,  
    Muir
    KW
    ,  
    Wallesch
    CW
    ,  
    McLelland
    EV
    .  
    et al.
    PRISM Study Group
    Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke.
    QJM
    2003
    96
    143-53
     PubMed
    CrossRef
     PubMed
  154. White
    KS
    ,  
    Lindsay
    A
    ,  
    Pryor
    TA
    ,  
    Brown
    WF
    ,  
    Walsh
    K
    .  
    Application of a computerized medical decision-making process to the problem of digoxin intoxication.
    J Am Coll Cardiol
    1984
    4
    571-6
     PubMed
    CrossRef
     PubMed
  155. Alper
    BS
    ,  
    White
    DS
    ,  
    Ge
    B
    .  
    Physicians answer more clinical questions and change clinical decisions more often with synthesized evidence: a randomized trial in primary care.
    Ann Fam Med
    2005
    3
    507-13
     PubMed
    CrossRef
     PubMed
  156. Del Fiol
    G
    ,  
    Haug
    PJ
    ,  
    Cimino
    JJ
    ,  
    Narus
    SP
    ,  
    Norlin
    C
    ,  
    Mitchell
    JA
    .  
    Effectiveness of topic-specific infobuttons: a randomized controlled trial.
    J Am Med Inform Assoc
    2008
    15
    752-9
     PubMed
    CrossRef
     PubMed
  157. Etchells
    E
    ,  
    Adhikari
    NK
    ,  
    Cheung
    C
    ,  
    Fowler
    R
    ,  
    Kiss
    A
    ,  
    Quan
    S
    .  
    et al.
    Real-time clinical alerting: effect of an automated paging system on response time to critical laboratory values—a randomised controlled trial.
    Qual Saf Health Care
    2010
    19
    99-102
     PubMed
     PubMed
    CrossRef
  158. Bird
    JA
    ,  
    McPhee
    SJ
    ,  
    Jenkins
    C
    ,  
    Fordham
    D
    .  
    Three strategies to promote cancer screening. How feasible is wide-scale implementation?
    Med Care
    1990
    28
    1005-12
     PubMed
    CrossRef
     PubMed
  159. Frame
    PS
    ,  
    Zimmer
    JG
    ,  
    Werth
    PL
    ,  
    Hall
    WJ
    ,  
    Eberly
    SW
    .  
    Computer-based vs manual health maintenance tracking. A controlled trial.
    Arch Fam Med
    1994
    3
    581-8
     PubMed
    CrossRef
     PubMed
  160. Tierney
    WM
    ,  
    McDonald
    CJ
    ,  
    Hui
    SL
    ,  
    Martin
    DK
    .  
    Computer predictions of abnormal test results. Effects on outpatient testing.
    JAMA
    1988
    259
    1194-8
     PubMed
    CrossRef
     PubMed
  161. Cleveringa
    FG
    ,  
    Gorter
    KJ
    ,  
    van den Donk
    M
    ,  
    Rutten
    GE
    .  
    Combined task delegation, computerized decision support, and feedback improve cardiovascular risk for type 2 diabetic patients: a cluster randomized trial in primary care.
    Diabetes Care
    2008
    31
    2273-5
     PubMed
    CrossRef
     PubMed
  162. Cleveringa
    FG
    ,  
    Welsing
    PM
    ,  
    van den Donk
    M
    ,  
    Gorter
    KJ
    ,  
    Niessen
    LW
    ,  
    Rutten
    GE
    .  
    et al.
    Cost-effectiveness of the diabetes care protocol, a multifaceted computerized decision support diabetes management intervention that reduces cardiovascular risk.
    Diabetes Care
    2010
    33
    258-63
     PubMed
    CrossRef
     PubMed
  163. Smith
    DH
    ,  
    Feldstein
    AC
    ,  
    Perrin
    NA
    ,  
    Yang
    X
    ,  
    Rix
    MM
    ,  
    Raebel
    MA
    .  
    et al.
    Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial.
    Am J Manag Care
    2009
    15
    281-9
     PubMed
     PubMed
  164. Judge
    J
    ,  
    Field
    TS
    ,  
    DeFlorio
    M
    ,  
    Laprino
    J
    ,  
    Auger
    J
    ,  
    Rochon
    P
    .  
    et al.
    Prescribers' responses to alerts during medication ordering in the long term care setting.
    J Am Med Inform Assoc
    2006
    13
    385-90
     PubMed
    CrossRef
     PubMed
  165. Maviglia
    SM
    ,  
    Yoon
    CS
    ,  
    Bates
    DW
    ,  
    Kuperman
    G
    .  
    KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs.
    J Am Med Inform Assoc
    2006
    13
    67-73
     PubMed
    CrossRef
     PubMed
  166. Rollman
    BL
    ,  
    Hanusa
    BH
    ,  
    Gilbert
    T
    ,  
    Lowe
    HJ
    ,  
    Kapoor
    WN
    ,  
    Schulberg
    HC
    .  
    The electronic medical record. A randomized trial of its impact on primary care physicians' initial management of major depression [corrected].
    Arch Intern Med
    2001
    161
    189-97
     PubMed
    CrossRef
     PubMed
  167. McLaughlin
    D
    ,  
    Hayes
    JR
    ,  
    Kelleher
    K
    .  
    Office-based interventions for recognizing abnormal pediatric blood pressures.
    Clin Pediatr (Phila)
    2010
    49
    355-62
    CrossRef
     PubMed
  168. Hetlevik
    I
    ,  
    Holmen
    J
    ,  
    Krüger
    O
    .  
    Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system.
    Scand J Prim Health Care
    1999
    17
    35-40
     PubMed
     PubMed
  169. Hetlevik
    I
    ,  
    Holmen
    J
    ,  
    Krüger
    O
    ,  
    Kristensen
    P
    ,  
    Iversen
    H
    .  
    Implementing clinical guidelines in the treatment of hypertension in general practice.
    Blood Press
    1998
    7
    270-6
     PubMed
     PubMed
  170. Hetlevik
    I
    ,  
    Holmen
    J
    ,  
    Krüger
    O
    ,  
    Kristensen
    P
    ,  
    Iversen
    H
    ,  
    Furuseth
    K
    .  
    Implementing clinical guidelines in the treatment of diabetes mellitus in general practice. Evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system.
    Int J Technol Assess Health Care
    2000
    16
    210-27
     PubMed
    CrossRef
     PubMed
  171. Bosworth
    HB
    ,  
    Olsen
    MK
    ,  
    Dudley
    T
    ,  
    Orr
    M
    ,  
    Goldstein
    MK
    ,  
    Datta
    SK
    .  
    et al.
    Patient education and provider decision support to control blood pressure in primary care: a cluster randomized trial.
    Am Heart J
    2009
    157
    450-6
     PubMed
    CrossRef
     PubMed
  172. Bosworth
    HB
    ,  
    Olsen
    MK
    ,  
    Goldstein
    MK
    ,  
    Orr
    M
    ,  
    Dudley
    T
    ,  
    McCant
    F
    .  
    et al.
    The veterans' study to improve the control of hypertension (V-STITCH): design and methodology.
    Contemp Clin Trials
    2005
    26
    155-68
     PubMed
    CrossRef
     PubMed
  173. Buntin
    MB
    ,  
    Burke
    MF
    ,  
    Hoaglin
    MC
    ,  
    Blumenthal
    D
    .  
    The benefits of health information technology: a review of the recent literature shows predominantly positive results.
    Health Aff (Millwood)
    2011
    30
    464-71
     PubMed
    CrossRef
     PubMed
  174. Osheroff
    JA
    .  
    Healthcare Information and Management Systems Society
    Improving Outcomes with Clinical Decision Support: An Implementer's Guide. 2nd ed.
    Chicago
    Healthcare Information and Management Systems Society
    2005
This article was published at www.annals.org on 24 April 2012.
Appendix Figure.

Summary of evidence search and selection.

CDSS = clinical decision-support system; KQ = key question; RCT = randomized, controlled trial.

Figure 1.

Results of studies that examined whether recommended preventive care services were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or completing preventive care services of CDSS vs. control groups. In the 25 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to preventive care recommendations was 1.42 (95% CI, 1.27 to 1.58). CDSS = clinical decision-support system.

Figure 2.

Results of studies that examined whether recommended clinical studies were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or completing recommended clinical studies of CDSS vs. control groups. In the 20 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to clinical study recommendations was 1.72 (95% CI, 1.47 to 2.00). CDSS = clinical decision-support system.

Figure 3.

Results of studies that examined whether recommended treatments were ordered.

Studies reporting the odds ratio of adhering to recommendations for ordering or prescribing treatment of CDSS vs. control groups. In the 46 studies comparing CDSS with control groups, the random-effects–combined odds ratio of adherence to treatment recommendations was 1.57 (95% CI, 1.35 to 1.82). CDSS = clinical decision-support system.

Appendix Table 1.

Study Characteristics

Appendix Table 1.

Table.

Summary of Evidence, by Outcome

Table.

Appendix Table 2.

Examples of Clinical Decision-Support Interventions

Appendix Table 2.

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

This feature is available only to Registered Users

Subscribe/Learn More
Submit a Comment

1 Comment

Edward Hoffer MD, FACP

MD, FACP

July 9, 2012

To the Editor

The article by Bright et al on Clinical Decision-Support Systems (CDSS) (1) was timely and interesting, but had an important omission. No mention was made of CDSS that focused on helping doctors make the correct diagnosis. Such systems have been broadly available for some 25 years, including QMR/Internist-I, ILIAD and DXplain. Currently, at least two general-purpose Diagnostic Decision Support systems, DXplain and ISABEL, are widely available. Reviews (2,3) have shown that these systems suggest important diseases the clinician had not previously considered, and a study (4) from the Mayo Clinic found that use of DXplain lowered length-of-stay and hospital costs.

1. Bright TJ et al “Effect of Clinical Decision-Support Systems” Ann Intern Med 2012;157:29-432.

2. Berner ES et al “Performance of Four Computer-Based Diagnostic Systems” New Engl J Med 1994; 330:1792-63.

3. Bond WF et al “Differential Diagnosis Generators: an Evaluation of Currently Available Computer Programs” J Gen Intern Med 2011;27:213-94.

4. Elkin PL et al “The Introduction of a Diagnostic Decision Support System (DXplain) into the Workflow of a Teaching Hospital Service can Decrease the Cost of Service for Diagnostically Challenging DRGs” Internat Jl Med Informatics 2010; 79:772-

PDF
Not Available
Citations
Citation

Bright TJ, Wong A, Dhurjati R, et al. Effect of Clinical Decision-Support Systems: A Systematic Review. Ann Intern Med. 2012;157:29–43. doi: https://doi.org/10.7326/0003-4819-157-1-201207030-00450

Download citation file:

  • Ris (Zotero)
  • EndNote
  • BibTex
  • Medlars
  • ProCite
  • RefWorks
  • Reference Manager

© 2019

×
Permissions

Published: Ann Intern Med. 2012;157(1):29-43.

DOI: 10.7326/0003-4819-157-1-201207030-00450

362 Citations

See Also

Improving the Efficiency of Advanced Life Support Training
Improving the Efficiency of Advanced Life Support Training
View MoreView Less

Related Articles

Dimension of Time in Illness: An Objective View
Annals of Internal Medicine; 132 (1): 45-53
Developing and Implementing Computerized Protocols for Standardization of Clinical Decisions
Annals of Internal Medicine; 132 (5): 373-383
What Is Accountability in Health Care?
Annals of Internal Medicine; 124 (2): 229-239
Measuring Patients' Expectations and Requests
Annals of Internal Medicine; 134 (9_Part_2): 881-888
View MoreView Less

Journal Club

A clinical decision-support system with interactive alerts improved CD4 cell count in HIV
Annals of Internal Medicine; 158 (8): JC11
Accessing preappraised evidence: fine-tuning the 5S model into a 6S model
Annals of Internal Medicine; 151 (6): JC3-2
Guideline: USPSTF recommends against menopausal estrogen for primary prevention of chronic conditions
Annals of Internal Medicine; 168 (6): JC26
Review: In older patients with chronic disease, transitional care reduces mortality and readmissions
Annals of Internal Medicine; 167 (6): JC32
View MoreView Less

Related Topics

Prevention/Screening

Prevention/Screening.

PubMed Articles

Prospective comparison of diffusion-weighted MR enterography and contrast enhanced CT enterography for the detection of ileocolonic crohn's disease.
J Gastroenterol Hepatol 2019.
Increased systemic inflammation in children with Down syndrome.
Cytokine 2019;
View More

Results provided by: PubMed

CME/MOC Activity Requires Users to be Registered and Logged In.
Sign in below to access your subscription for full content
INDIVIDUAL SIGN IN
Sign In|Set Up Account
You will be directed to acponline.org to register and create your Annals account
Annals of Internal Medicine
CREATE YOUR FREE ACCOUNT
Create Your Free Account|Why?
To receive access to the full text of freely available articles, alerts, and more. You will be directed to acponline.org to complete your registration.
×
The Comments Feature Requires Users to be Registered and Logged In.
Sign in below to access your subscription for full content
INDIVIDUAL SIGN IN
Sign In|Set Up Account
You will be directed to acponline.org to register and create your Annals account
Annals of Internal Medicine
CREATE YOUR FREE ACCOUNT
Create Your Free Account|Why?
To receive access to the full text of freely available articles, alerts, and more. You will be directed to acponline.org to complete your registration.
×
link to top

Content

  • Home
  • Latest
  • Issues
  • Channels
  • CME/MOC
  • In the Clinic
  • Journal Club
  • Web Exclusives

Information For

  • Author Info
  • Reviewers
  • Press
  • Readers
  • Institutions / Libraries / Agencies
  • Advertisers

Services

  • Subscribe
  • Renew
  • Alerts
  • Current Issue RSS
  • Latest RSS
  • In the Clinic RSS
  • Reprints & Permissions
  • Contact Us
  • Help
  • About Annals
  • About Mobile
  • Patient Information
  • Teaching Tools
  • Annals in the News
  • Share Your Feedback

Awards and Cover

  • Personae (Cover Photo)
  • Junior Investigator Awards
  • Poetry Prize

Other Resources

  • ACP Online
  • Career Connection
  • ACP Advocate Blog
  • ACP Journal Wise

Follow Annals On

  • Twitter Link
  • Facebook Link
acp link acp
silverchair link silverchair

Copyright © 2019 American College of Physicians. All Rights Reserved.

Print ISSN: 0003-4819 | Online ISSN: 1539-3704

Privacy Policy

|

Conditions of Use

This site uses cookies. By continuing to use our website, you are agreeing to our privacy policy. | Accept
×

You need a subscription to this content to use this feature.

×
PDF Downloads Require Access to the Full Article.
Sign in below to access your subscription for full content
INDIVIDUAL SIGN IN
Sign In|Set Up Account
You will be directed to acponline.org to register and create your Annals account
INSTITUTIONAL SIGN IN
Open Athens|Shibboleth|Log In
Annals of Internal Medicine
PURCHASE OPTIONS
Buy This Article|Subscribe
You will be redirected to acponline.org to sign-in to Annals to complete your purchase.
CREATE YOUR FREE ACCOUNT
Create Your Free Account|Why?
To receive access to the full text of freely available articles, alerts, and more. You will be directed to acponline.org to complete your registration.
×
Access to this Free Content Requires Users to be Registered and Logged In. Please Choose One of the Following Options
Sign in below to access your subscription for full content
INDIVIDUAL SIGN IN
Sign In|Set Up Account
You will be directed to acponline.org to register and create your Annals account
Annals of Internal Medicine
CREATE YOUR FREE ACCOUNT
Create Your Free Account|Why?
To receive access to the full text of freely available articles, alerts, and more. You will be directed to acponline.org to complete your registration.
×