IMPROVING PATIENT CARE
Spencer S. Jones, PhD; Robert S. Rudin, PhD; Tanja Perry, BHM; Paul G. Shekelle, MD, PhD
Acknowledgment: The authors thank Aneesa Motala and Roberta Shanman for their research assistance and assistance with the literature searches and retrieval. They also thank the members of the Technical Expert Panel: David W. Bates, MD, MSc; George Hripsak, MD, MS; Philip J. Aponte, MD; Louise Liang, MD; and Paul Tang, MD, MS.
Grant Support: By the Office of the National Coordinator.
Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-1531.
Requests for Single Reprints: Spencer S. Jones, PhD, Southern California Evidence-Based Practice Center, RAND Health, 1776 Main Street, Santa Monica, CA 90401; e-mail: email@example.com.
Current Author Addresses: Drs. Jones and Shekelle and Ms. Perry: Southern California Evidence-Based Practice Center, RAND Health, 1776 Main Street, Santa Monica, CA 90401.
Dr. Rudin: RAND Corporation, 20 Park Plaza, Suite 920, Boston, MA 02116.
Author Contributions: Conception and design: S.S. Jones, R.S. Rudin, P.G. Shekelle.
Analysis and interpretation of the data: S.S. Jones, R.S. Rudin, P.G. Shekelle.
Drafting of the article: S.S. Jones, R.S. Rudin, T. Perry, P.G. Shekelle.
Critical revision of the article for important intellectual content: S.S. Jones, R.S. Rudin, P.G. Shekelle.
Final approval of the article: S.S. Jones, R.S. Rudin, T. Perry, P.G. Shekelle.
Provision of study materials or patients: S.S. Jones, P.G. Shekelle.
Statistical expertise: S.S. Jones.
Obtaining of funding: S.S. Jones, P.G. Shekelle.
Administrative, technical, or logistic support: T. Perry.
Collection and assembly of data: S.S. Jones, P.G. Shekelle.
Jones S., Rudin R., Perry T., Shekelle P.; Health Information Technology: An Updated Systematic Review With a Focus on Meaningful Use. Ann Intern Med. 2014;160:48-54. doi: 10.7326/M13-1531
Download citation file:
Published: Ann Intern Med. 2014;160(1):48-54.
Incentives offered by the U.S. government have spurred marked increases in use of health information technology (IT).
To update previous reviews and examine recent evidence that relates health IT functionalities prescribed in meaningful use regulations to key aspects of health care.
English-language articles in PubMed from January 2010 to August 2013.
236 studies, including pre–post and time-series designs and clinical trials that related the use of health IT to quality, safety, or efficiency.
Two independent reviewers extracted data on functionality, study outcomes, and context.
Fifty-seven percent of the 236 studies evaluated clinical decision support and computerized provider order entry, whereas other meaningful use functionalities were rarely evaluated. Fifty-six percent of studies reported uniformly positive results, and an additional 21% reported mixed-positive effects. Reporting of context and implementation details was poor, and 61% of studies did not report any contextual details beyond basic information.
Potential for publication bias, and evaluated health IT systems and outcomes were heterogeneous and incompletely described.
Strong evidence supports the use of clinical decision support and computerized provider order entry. However, insufficient reporting of implementation and context of use makes it impossible to determine why some health IT implementations are successful and others are not. The most important improvement that can be made in health IT evaluations is increased reporting of the effects of implementation and context.
Office of the National Coordinator.
In the United States, adoption of health information technology (IT) has been accelerated by the meaningful use incentive program, which provides financial incentives to individual health care providers and organizations that demonstrate that they use “certified” health IT to meet a set of several use criteria specified by the Centers for Medicare & Medicaid Services (1, 2). This increase in use has been accompanied by a concomitant increase in the number of published evaluations of health IT. Because of the rapidly expanding evidence base, the Office of the National Coordinator requested a systematic update of the literature (3). The objective of this review is to update previous reviews (4–6) and examine recent evidence that relates health IT functionalities prescribed in meaningful use regulations to health care quality (including process, health, and patient and provider satisfaction outcomes), safety (including medication safety and other manifestations of patient safety), and efficiency (including costs, utilization, timeliness, and time burden of health care).
Although we did not develop a formal protocol for this update, we based it on the search strategy, inclusion and exclusion criteria, data collection, and synthesis methods from previous reviews on health IT (4–6).
A 5-person technical expert panel, which included academic, health care delivery, and policy experts in health IT, guided the review process. Literature searches were based on the strategy initially used by Chaudhry and colleagues (4) and updated by Goldzweig (5) and Buntin (6) and their respective colleagues. This strategy uses broad-based search terms for the English-language literature indexed in PubMed. Our initial search covered the period of January 2010 to November 2011. We used a computer-aided screening method (7) to update that search to November 2012, and then updated searches again to August 2013 (Tables 1 and 2 of the Supplement). Our expert panel reviewed the search results and suggested additional articles that may have been missed.
Two expert reviewers used a Web-based system, DistillerSR (8), to independently select studies. Following the methods of our previous reviews, we considered “hypothesis-testing” studies of health IT effects and “descriptive quantitative” studies for inclusion. We classified articles as “hypothesis-testing” if the investigators compared data between groups or across periods and used statistical tests to assess differences. Hypothesis-testing studies were further classified by study design (such as randomized, controlled trials [RCTs]). To be included, a study needed to evaluate a health IT functionality encompassed by the meaningful use regulations. The meaningful use requirements specify 25 criteria for health IT functionality and use (such as “Use CPOE for medication orders”), of which providers must meet a portion to receive incentive payments (Tables 3 and 4 of the Supplement).
Using a structured form, we abstracted information about the following: study design; clinical setting, health care conditions, and aspects of care assessed; research sites; health IT type (commercial or “homegrown”); meaningful use functionality evaluated; and context and implementation details. We adapted criteria developed to assess health IT applications in patient safety to classify articles according to reported context (4 domains) and implementation details (7 specific components) (9). We used a modified version of the outcome result classification framework (positive, mixed-positive, neutral, or negative), originally used by Buntin and colleagues (6), to assess outcomes. Our adaptations made the classification framework more conservative than the original framework, thus increasing the likelihood that an article's findings would be classified as mixed-positive, neutral, or negative (Table 5 of the Supplement). The functionality evaluated, context domains, implementation components, and article outcomes were classified by dual-review, and conflicts were resolved by consensus.
This project was done under contract to the Office of the National Coordinator. Representatives of the Office of the National Coordinator were briefed on study findings and reviewed a draft manuscript but were not involved in the analysis or decision to submit the manuscript for publication.
We deemed 2482 of 12 678 titles identified in the searches potentially relevant. We excluded 2023 of these after abstract review and another 223 after full-text review. The 236 articles selected for review encompassed 278 outcomes because some articles addressed several aspects of care and outcomes (Figure). The full list of the included studies can be found in the reference list in the Supplement. Quality outcomes (n = 170) were evaluated more than safety (n = 46) and efficiency (n = 62) outcomes combined, and care processes (n = 103) outcomes were more than twice as common as health outcomes (n = 47). Studies with a simple pre–post design were most common (31%), followed by RCTs (25%) and time-series studies (11%). More than one half (53%) evaluated commercial health IT products, although approximately one quarter did not report whether the products that were evaluated were commercial or “homegrown.”
Evidence search and selection.
Most studies addressed clinical decision support (CDS)(n = 85 [36%]); computerized provider order entry (CPOE) (n = 49 [21%]); or multifunctional health IT interventions (n = 47 [20%]), which evaluated broad IT interventions, such as electronic health records (EHRs), that encompassed many of the functionalities required under meaningful use. Twelve of the 25 functionalities in the meaningful use regulations, such as “capacity to track vital signs” or “maintain medication allergy lists,” were not specifically evaluated in any studies. These features, however, were likely critical to the functionality of IT interventions (such as CDS) that were evaluated in many studies. Nevertheless, for some aspects of meaningful use, such as “implement systems to protect privacy and security,” no eligible studies were found.
Overall, 147 articles assessed the effect of health IT on 170 quality-related outcomes (Tables 6 and 7 of the Supplement). More than one half of these studies assessed CDS alerts and reminders; the most commonly studied outcomes were medication management, screening and preventive care, and process quality for diabetes and venous thromboembolism. More than three quarters of studies of alerts and reminders, including large quasi-experimental studies and RCTs, reported positive effects. Notable among these included a 1-year prospective, cluster randomized trial in 12 primary care pediatric practices that reported that EHR-embedded CDS substantially improved the use of asthma control medications, spirometry, and the maintenance of care plans (10); a study of nearly 20 000 surgical patients that found that CDS was associated with a 30% increase in adherence to infection prevention guidelines (11); a controlled before-and-after study compared 360 primary care physicians in New York, New York, with a matched set of control physicians and reported that EHR-sensitive measures of process quality (such as eye examinations and urine testing for patients with diabetes) improved substantially only in practices that received “high levels of technical assistance” in implementing their EHRs (12); and a before-and-after study showing that a CDS intervention in a commercial health IT system was associated with a significant increase in venous thromboembolism prophylaxis and a substantial decline in the rate of venous thromboembolism among nearly 40 000 patients admitted to a single academic medical center (13). Eighteen percent of studies reported no statistically significant improvements in quality or even negative effects. An illustrative example of these studies reported that on the basis of a nationally representative sample of physician office visits, EHRs were associated with poor depression care among patients diagnosed with multiple chronic conditions. The authors of this study hypothesized that EHR workflows gave precedence to the treatment of physical conditions, and therefore psychosocial problems were left unaddressed (14).
Forty-six studies investigated the effects of health IT functionalities on patient safety outcomes, focusing exclusively on medication safety (Tables 8 and 9 of the Supplement). Approximately 78% of these studies reported at least some positive effects. Of note, benefits were found for a wide range of medication safety outcomes in various care settings. Automated dose calculation features within CPOE systems were found to have significant relative reductions in errors in medication dosage ranging from 37% to 80% (15–17), and authors of 1 study reported that innovative features, such as incorporation of an order verification screen with a patient picture, was associated with complete elimination of incorrect patient orders (18). However, a few studies reported that, in some cases, health IT did not have the desired effect on medication safety, and “alert fatigue” and incongruent workflows were described as barriers to successful use of these systems.
We identified 58 articles that assessed the effect of health IT on 62 efficiency-related outcomes (Tables 10 and 11 of the Supplement). Cost effects ranged from a 75% decrease to a 69% increase in the targeted costs. A few of the studies clustered in the range of 6% to 12% increases in the targeted costs. Understanding the relationship between health IT and health care utilization is complicated by external factors, such as the payment environment in which the health care providers operate. We identified some large studies that reported that patient and provider access to health IT were associated with increased health care utilization. However, other studies found that health IT can have the opposite effect. Overall, 85% of the studies evaluating utilization concluded that the effects of health IT led to an appropriate increase or decrease in utilization. Finally, studies that assessed the time burden or timeliness of care reported mixed results. Positive findings included shorter emergency department length of stay, reduced diagnostic turnaround times, shorter time to the initiation of appropriate therapies, and more in-person time with patients (19–27). Nevertheless, some studies found that health IT was associated with increased documentation and that in some instances, even when providers were able to spend more time with their patients, much of that time was spent interacting with the computer.
Most studies reported the size, location, and academic status of the facility where the health IT system was being implemented. They also reported that staff are trained on the new system. Only 9% of studies reported on 2 or more contextual domains in addition to size, location, and academic status, and only 8% of studies reported on 4 or more implementation components (Tables 12 to 15 of the Supplement). Only 5 studies reported adequately on both context and implementation. Sixty-one percent and 64% of studies reported nothing beyond the basics for context and implementation, respectively, and 42% reported nothing beyond the basics for both context and implementation. Among 64 studies that assessed the effectiveness of a specific health IT intervention in multiple sites, only 5 studies quantitatively assessed for a difference in health IT effectiveness across sites, and 4 additional studies qualitatively reported on context sensitivity of the intervention.
The most commonly reported context domains were financial status in 24% of studies (however, 70% of these received credit because they were done in countries with government-financed health care or the Veterans Affairs health systems or Kaiser Permanente, leaving only 12 U.S. studies reporting financial status) and existing infrastructure in 16%. The most commonly reported specific implementation components were timeline of implementation in 28% and description of education and training in 26%.
This review was done using methods similar to 3 previous broad-based reviews of health IT (4–6) between 1995 and 2013. We found that the number of published health IT evaluation studies is increasing rapidly: Such studies increased by approximately 13% per year before 2007, and roughly 25% per year from 2008 to 2012. Table 1 shows the total number of article outcomes arrayed by the type of health IT functionalities studied and health care outcomes evaluated for all health IT studies identified in the 4 literature reviews. The summary data show that CDS and CPOE have been studied extensively, and other functionalities, such as health information exchange and functionalities that allow patients to access their own electronic records, are not as well-studied. Evaluations of the effect of health IT on health care quality predominate and make up more than one half of all studies included in all 4 systematic reviews.
Table 1. Health IT Evaluation Studies Between 1995 and 2013, by Study Outcome Type
Table 2 presents data for the 2 most recent broad-based health IT systematic reviews (since July 2007) arrayed by functionality and a broad measure of their effect. Combined CDS and CPOE has been the subject of more than 230 evaluations, most of which have reported positive results. Multifunctional health IT interventions have also been studied extensively. Although most of these latter studies report positive results, they found mixed results more often than did studies of CPOE or CDS. In addition, studies of other specific meaningful use functionalities, such as health information exchange and e-prescribing, are much less common but also reported positive findings more often than not.
Table 2. Health IT Evaluation Studies Between 2007 and 2013, by Study Outcome Result
Our update shows that the published literature on health IT is expanding rapidly and that much of this expansion is attributable to commercial health IT systems. In the original review by Chaudhry and colleagues, studies of commercial systems constituted a negligible proportion of the literature (less than 5%). In the more recent review by Goldzweig and colleagues, this proportion had increased to only approximately 8%. In the current review, more than 50% of the eligible studies were explicitly about commercial health IT systems. This increase is a welcome change because Chaudhry and colleagues identified evaluation of commercial systems as the top research priority in health IT.
Although the health IT evaluation literature base is expanding rapidly, we are concerned that there has not been a commensurate increase in our understanding of the effect of health IT or how it can be used to improve health and health care. Study questions, research methods, and reporting of study details have not sufficiently adapted to meet the needs of clinicians, health care administrators, and health policymakers and are falling short of addressing the future needs of the health care system.
Nevertheless, some broad conclusions can be drawn. Most studies of CDS report positive or mixed-positive results, and existing systematic reviews of specific CDS systems are mostly positive with respect to changes in processes of care (28–33). We conclude that CDS generally results in improvements in the processes targeted by the decision support. If neutral or negative results are reported in new studies of CDS, these results are more likely to be because of specifics of the particular intervention, context, or implementation than an indication that the general construct of computerized decision support is not a beneficial IT functionality for improving health care quality. The same is true for CPOE: Most evaluations have reported positive or mixed-positive effects, and most existing systematic reviews likewise conclude that CPOE reduces medication errors (34, 35). We can conclude that CPOE effectively decreases medication errors. Health care providers should be encouraged to adopt CDS and CPOE, and future studies of CDS and CPOE should concentrate on how to make them work better rather than testing the hypothesis of whether they work at all. In contrast to this, the evidence base on other functionalities, such as patient care reminders or patient specific education, have small numbers of studies, and any new studies add proportionately much more to our existing knowledge, both about the general construct of the functionality plus the potential for context and implementation sensitivity of the effects. The lack of reporting about key elements of context and implementation of health IT was noted in the review by Chaudhry and colleagues, and despite calls then and more recently for better reporting on context and implementation—and even suggestions for specific items to report on (36, 37)—we still find that crucial elements of context and implementation are missing from most published health IT studies.
This phenomenon of underreporting may be partially explained by the fact that early studies of health IT functionalities were trying to determine only whether a particular health IT functionality created value and to what extent. This was perhaps the most important research question when health IT was a novel phenomenon because it led to a demonstration of the potential of the new technology. With the increasing adoption of EHRs and other forms of health IT, it is no longer sufficient to ask whether health IT creates value; going forward, the most useful studies will help us understand how to realize value from health IT (38).
Stakeholders with interests in health IT, including policymakers, research funders, and journal editors, can help researchers shift their focus from if to how by promoting, soliciting, and publishing research that empirically studies the mediating effects of contextual and implementation factors on the relationship between health IT and key health care outcomes. Such a shift in focus will greatly advance the science of health IT evaluation and greatly increase our understanding of how the positive effects of health IT can be maximized and negative effects can be avoided or remediated. As a start, policymakers and other research funders could require that researchers funded by their organizations or reported in their journals explicitly analyze and report a basic set of contextual and implementation factors. Reporting frameworks for health IT intervention based on expert consensus are beginning to emerge (38), and other frameworks for similar health care interventions, such as patient safety interventions, could easily be adapted to health IT interventions (37).
The potential for publication bias is always a limitation of systematic reviews of this type. Researchers often do not evaluate the potential for negative effects, and even when identified, negative results are potentially less likely to be published. In addition, all of the included studies were weighted equally, regardless of study design or sample size. However, we considered factors that were related to the generalizability of the evidence, such as sample size, inclusion of several measures, and use of statistical methods, when drawing conclusions. Finally, health IT systems and the outcomes evaluated were heterogeneous and often incompletely described.
In sum, the health IT literature is expanding rapidly but failing to produce a commensurate amount of useful knowledge. Although most studies reported that health IT interventions had statistically and clinically significant benefits, sometimes these were not as large as the developers had expected, and there are also examples where benefits were not realized. Now that health IT is being widely adopted, researchers should refocus their efforts to show how health IT can be used to realize value. The most common characteristics of published studies are still pre–post studies of a CDS or CPOE at a single site that report nothing beyond the basics in terms of context and implementation. Such studies should be discouraged. The most important improvement that can be made in health IT evaluations is increased measurement of and reporting of context, implementation, and context-sensitivity of effectiveness.
The number of health information technology (IT) evaluation studies is rapidly increasing, driven primarily by increased evaluation of commercial health IT applications.
Most evaluations focus on clinical decision support and computerized provider order entry.
Most published health IT implementation studies report positive effects on quality, safety, and efficiency.
Insufficient reporting of contextual and implementation factors makes it impossible to determine why most health IT implementations are successful but some are not.
The most important improvement that can be made in health IT evaluations is increased measurement, analysis, and reporting of the effects of contextual and implementation factors.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Healthcare Delivery and Policy.
Results provided by:
Copyright © 2016 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use
This PDF is available to Subscribers Only