Enhancing the Potential of Quality Improvement Organizations To Improve Quality of Care FREE

Many agree that the quality of health care in the United States has improved in the past decade (1). Progress, however, has been slow, incremental, uneven, and far from what could have been realized with a more concerted effort, aligned incentives, improved performance measurement, wider adoption of electronic health records, and increased accountability (2). As the largest financial and programmatic sponsor of health care services in the United States, the Centers for Medicare & Medicaid Services (CMS) is responsible for leading the process of improving the quality of care for the nation's Medicare beneficiaries. Key components of this major responsibility are the Quality Improvement Organizations (QIOs), which contract with CMS to provide technical assistance and case reviews and investigate beneficiary complaints. These organizations have evolved over the past 40 years from an initial focus on hospital utilization review and identification of unnecessary care to a greater emphasis on measuring quality and outcomes of care and implementing interventions to promote quality improvement (3). Over time, the scope of these responsibilities has expanded beyond acute inpatient hospital care to include home health, skilled nursing facilities, and physician office practices. Health care provider involvement in the activities of the QIO Program is entirely voluntary.

A 1990 Institute of Medicine (IOM) report (3) called for the Health Care Financing Administration (now CMS) to systematically evaluate its efforts to improve the quality and outcome of care for Medicare beneficiaries. For the most part, CMS has not implemented this recommendation systematically, although it has made some largely isolated efforts to assess the QIO Program (4). The article by Rollow and colleagues in this issue (5) is one example. It is an ambitious undertaking in which the authors used available data to try to gauge the impact of QIO interventions across multiple health care provider sectors.

The basic evaluation strategy of the CMS was to compare measurements of quality before and after a QIO implemented a quality improvement intervention. The QIOs nonrandomly selected nursing homes, physician offices, and home health agencies to receive assistance, and some of them volunteered to receive more intense QIO assistance in various forms. The data suggest that organizations that received intensive assistance scored better on most quality indicators than those that did not receive such assistance. When the authors evaluated the effect of QIO quality improvements aimed at hospitals, it was not possible to have a control group because the QIOs implemented quality improvement efforts for hospitals at a statewide level rather than targeting intense efforts at selected hospitals. Using the same before–after study design, the authors found that hospitals improved in most categories over the study interval.

Interpreting these findings is difficult. The authors appropriately note the major limitations of their evaluation, including bias introduced by the nonrandom selection process, voluntary participation, the possible influence of regression to the mean in a before–after study, and the inability to distinguish the effects of QIO assistance from the impact of other concurrent quality improvement activities and exposures typically experienced by providers (an issue that is particularly relevant to hospitals, many of which do not participate in the QIO Program). An additional barrier to interpretation is the paucity of information about how the QIOs recruited provider organizations; the nature of the “general guidance” provided; and the content, timing, duration, and frequency of QIO interventions. We don't know which of the reported differences are statistically significant, although given the large number of organizations involved, one would assume that some were. Furthermore, the authors' analyses do not account for potentially relevant variables, such as organization size and ownership, availability of resources, degree of market competition, and the regulatory environment in the relevant state. The authors note that they tried to control for some of these factors, but they don't report the results.

So what is the reader to conclude from this analysis? In brief, although several performance indicators improved in desired directions, we do not know why and we cannot be sure that such improvement stemmed from the QIO interventions. Readers might also be concerned by the small degree of apparent improvements in many reported categories. Although the authors properly conclude that the “findings are consistent with an effect of the QIO Program and an effect of QIO technical assistance,” the results are also consistent with several alternative causes that the authors could not take into account given the study design and analysis. It should be stressed that this outcome is consistent with many reports in the quality improvement literature. For example, of 33 recent studies, 9 showed generally positive effects, 16 yielded ambiguous results, and 8 found no impact or negative effects (6). Most of these studies share the same limitations, in part or in whole, as this QIO evaluation.

We cannot exclude the possibility that QIOs and related organizations are having a positive impact. Accurate evaluation is of utmost importance and will require stronger studies. Given the presence of experienced QIOs in every state and territory, the program has enormous potential to assist in improving the quality of health care. Recognizing this potential and the need for definitive evaluations, a recent IOM committee has called for a markedly enhanced evaluation system to guide the Medicare program in the future (6). The committee recommended 4 types of evaluation: 1) evaluating the QIO Program as a whole; 2) evaluating processes of individual QIOs; 3) assessing the impact of selected quality improvement interventions; and 4) assessing the QIO Program over time by using an independent external evaluator. The committee also recommended that CMS form an outside technical review body to provide input into the evaluation process. The committee encouraged the CMS to use randomized, controlled designs involving a large number of providers, an appropriate time interval (perhaps with a “crossover” design component), and documentation of the exact nature of the interventions using qualitative methods. When such designs are not possible, the CMS should make creative use of carefully matched comparison groups and retrospective case–control studies.

The IOM committee also proposed several interrelated recommendations to enhance the QIO Program. These included 1) a primary, increased focus on direct technical assistance to providers (improved data collection and reporting, internal system redesign and process improvement); 2) a shift of the case review and beneficiary complaint responsibilities of QIOs to another organization or organizations selected by open competition; 3) improved QIO governance; 4) improved data processing, management, and communication between CMS and the QIOs; and 5) streamlined, strengthened oversight of the QIOs by CMS, coupled with extending the QIO core contracts from 3 years to 5 years to facilitate QIO Program improvement, monitoring, and evaluation.

All Medicare beneficiaries, and indeed all Americans, deserve access to high-quality health care. Therefore, Congress and the public at large must have a validated process to assess current and future investments in continuous quality-of-care improvements. Medicare has a special challenge because the baby boomer generation will markedly expand the Medicare-eligible population. The expected growth in Medicare is also an opportunity for CMS to lead the process of quality improvement for the entire U.S. population. For CMS to lead, it must show that it can not only measure quality of care accurately but also implement studies that can rule out plausible alternative explanations for results. As CMS incorporates more rigorous evaluations into future QIO scopes of work, Congress and the nation will be better able to determine the quality of health care received by its senior citizens and measure the return on the nation's investment in quality improvement initiatives.

Stephen M. Shortell, PhD, MPH

University of California, Berkeley

Berkeley, CA 94720-7360

William A. Peck, MD

Washington University School of Medicine

St. Louis, MO 63110