The full content of Annals is available to subscribers

Subscribe/Learn More  >
Improving Patient Care |

Effects of a Quality Improvement Collaborative on the Outcome of Care of Patients with HIV Infection: The EQHIV Study

Bruce E. Landon, MD, MBA; Ira B. Wilson, MD, MSc; Keith McInnes, MS; Mary Beth Landrum, PhD; Lisa Hirschhorn, MD, MPH; Peter V. Marsden, PhD; David Gustafson, PhD; and Paul D. Cleary, PhD
[+] Article, Author, and Disclosure Information

From Harvard Medical School, Beth Israel Deaconess Medical Center, New England Medical Center, and Dimock Community Health Center, Boston, Massachusetts; Harvard University, Cambridge, Massachusetts; and The University of Wisconsin, Madison, Wisconsin.

Acknowledgments: The authors thank their colleagues at the HIV AIDS Bureau of the Health Resources and Services Administration for their efforts and expertise. They also thank Lin Ding, PhD, for assistance with expert statistical programming; Deborah Collins for assistance with manuscript preparation; Carol Cosenza, MSW, and Patricia Gallagher, PhD, of the Center for Survey Research for assistance with instrument development and survey administration; and Barbara J. McNeil, MD, PhD, for comments on an earlier version of this manuscript.

Grant Support: By a grant from the Agency for Healthcare Research and Quality (R-01HS10227).

Potential Financial Conflicts of Interest:Other: D. Gustafson (board member of the Institute for Healthcare Improvement).

Requests for Single Reprints: Bruce E. Landon, MD, MBA, Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115; e-mail, landon@hcp.med.harvard.edu.

Current Author Addresses: Drs. Landon, McInnes, Landrum, and Cleary: Department of Health Care Policy, Harvard Medical School, 180 Longwood Avenue, Boston, MA 02115.

Dr. Wilson: The Health Institute, Division of Clinical Care Research, Tufts–New England Medical Center, 750 Washington Street, PO Box 345, Boston, MA 02111.

Dr. Hirschhorn: 26 Berkshire Road, Newton, MA 02460.

Dr. Marsden: William James Hall, 630, Harvard University, 33 Kirkland Street, Cambridge, MA 02138.

Dr. Gustafson: Center for Health Systems Research and Analysis, WARF Building, 11th Floor, 610 Walnut Street, Madison, WI 53705-2397.

Ann Intern Med. 2004;140(11):887-896. doi:10.7326/0003-4819-140-11-200406010-00010
Text Size: A A A

To our knowledge, this study is the first national controlled evaluation of a Breakthrough Series collaborative. In a prospective matched trial including almost 10 000 patients in 69 clinics, we found that the quality improvement collaborative did not significantly affect virologic outcomes or process measures, such as screening, prophylaxis, and access to care.

First Page Preview

View Large
First page PDF preview


Grahic Jump Location
Figure 1.
Theoretical construct of continuous quality improvement.

Reproduced from Berwick DM, Nolan TW (11) with permission from the American College of Physicians.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Design and time line of the Evaluation of Quality Improvement for HIV (EQHIV) Project.

Midway through the process, the collaborative was extended 3 months past its originally scheduled end date.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
In Response
Posted on July 28, 2004
Bruce E. Landon
Harvard Medical School
Conflict of Interest: None Declared

In response: Dr. Batalden suggests that more information than is provided in our article1 is needed to understand how to improve quality improvement (QI) interventions. We agree completely. A good evaluation study should first assess whether an intervention was successful.2 When there is no effect, a good study should provide information about how well the program was implemented, as well as information on potential moderating effects. Annals articles must be concise and we could not include all of these details in our article. However, we did assess virtually all of the factors mentioned by Dr. Batalden. We conducted detailed assessments of the almost 1,500 change initiatives attempted.1 We also surveyed clinicians and medical directors at each of the clinics both before and after the intervention to assess these factors, and we made site visits to a sample of clinics to collect qualitative data that might shed light on why the intervention was not more successful. Prior to publishing our evaluation, we conducted preliminary analyses of all that information to assure ourselves that we were not misrepresenting the results or missing a critical determinant of success. We hope that subsequent analyses will improve our understanding of why the intervention described was not more successful.

Dr. Agins is concerned that our evaluation might have been contaminated by the participation of control clinics in the HIV QUAL program or other quality improvement efforts. We had the same concern and asked each study clinic detailed questions about their participation in QI activities both before and after the collaborative. We did not find much evidence of ongoing quality improvement activities in the control clinics. The relatively small improvement in both the intervention and control sites suggests that our lack of results was not due to large improvements in the control clinics.

We did assess whether there were larger effects in the "population of focus" in the 11 clinics (out of 44 total) that restricted their efforts to a subset of the clinic population.1 In models that compared improvement in the population of focus to control clinics, only the increase in the number of visits in 3 or 4 quarters was significant (p=.04).

We hope that future studies continue to address these types of issues to enhance our understanding of the interventions that will lead to the greatest improvements.

Bruce E. Landon, M.D., M.B.A, M.Sc., Paul D. Cleary, Ph.D. Harvard Medical School Boston, MA 02115

Ira B. Wilson, M.D., M.Sc. Tufts-New England Medical Center Boston, MA 02111

1. Landon BE, Wilson IB, McInnes K, et al. Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study. Ann Intern Med. 2004;140:887-896. 2. Rossi PH, Freeman HE, Lipsey MW. Evaluation. A Systematic Approach. Sixth ed. Thousand Oaks: SAGE Publications; 1999.

Conflict of Interest:

None declared

Improving the Quality of Evaluations of Quality Improvement
Posted on August 3, 2004
Bruce Agins
NY State Dept. of Health
Conflict of Interest: None Declared

The landmark study by Landon et. al. is a rigorous evaluation of a major HIV quality improvement (QI) initiative and of large-scale QI collaboratives . Specific factors pertinent to the national network of HIV services in the US temper the findings of this important study, particularly with respect to comparison of results between intervention and control clinics.

Although the authors carefully mention that results may be contaminated because control clinics participated in another QI program, they suggest that this intervention occurred prior to the collaborative, when in fact, it continued during the collaborative, and is ongoing. The National HIVQUAL Project, also funded by the HIV/AIDS Bureau of HRSA, is another resource for QI grantees managed by the NYS Department of Health AIDS Institute providing onsite consultation to promote development of quality management programs for these clinics. In spite of acknowledging this potential "˜contamination' the analysis does not consider the effect of this parallel QI consultation among both control and intervention participants. Possibly, differences in results between those clinics would have differed from those which did not participate in either.

Another critical issue is the analysis of findings within a participating agency. Because selection of patients for testing changes in the collaborative included bias through the choice of a population of focus, the authors reviewed representing this group as well as those not in this group. They do not however describe differences in results between these groups, leaving open the question of whether interventions produced improvements in the testing group, as opposed to the entire group, which would support positive benefit for patients targeted by QI strategies, identifying instead a potential problem with spread of these changes throughout the care system from those achieved in the selected group. Targeting diffusion of change is a specific strategy that could become integrated into the collaborative framework.

The evaluation of QI collaboratives poses substantial challenges given the multitude of changes occurring simultaneously and the existence of concurrent external and internal stimuli to improve care. This study highlights important issues in the evaluation methodology of QI initiatives - which needs to be refined before we dismiss the collaborative model for improvement. With adherence to its own fundamental principles, the collaborative model can be improved to enhance its quality and our understanding to further the science of improvement.

Bruce Agins, MD, MPH

Medical Director, AIDS Institute New York State Dept. of Health Phone: 212.268.6142 Fax: 212.268.6098

Address: NYSDOH AIDS Institute

5 Penn Plaza; First Floor

New York, NY 10001

Conflict of Interest:

None declared

Improving Improvement
Posted on August 3, 2004
Paul Batalden
Dartmouth Medical School
Conflict of Interest: None Declared

Dear Editor, The recent articles by Landon, et al and Mittman are welcome encouragement in the effort to improve the improvement of clinical care. As they suggest, published studies with greater capacity to discern what has happened and greater ability to describe the context will be helpful. Let me extend their suggestions for future published reports:

1. About the populations. Patients with HIV AIDS fit into several subpopulations, e.g., drug-using/not, transfusion or blood products- acquired/not, etc. Improving care for subgroups may require different interventions. Were the efforts in either control or intervention group stratified and designed with the subgroup needs in mind? What knowledge about "˜desirable care' did the patients within a site have? What patterns of patient involvement in the continuing redesign and improvement of the care was the local habit/practice in the intervention and control settings?

2. About the changes introduced. Did the changes introduced match the known local barriers to better performance? Were they designed to fit the particular local situation? Was an ongoing process "˜independent of the intervention' of local problem identification and remediation underway in either group?

3. About the setting's change history. What was the change history in the settings? Did the changes arise from intact care units working together to make change, or were their changes done in response to a charismatic leader? Was there a clinical champion of the change? Did the intervention build on prior successful changes? If the practices didn't change, did the provider knowledge change? Was the length of the follow- up appropriate to assess the effects? Was there a curious local leader with demonstrated change skill in either setting?

4. About the use of measurement. What were the local habits about using outcome and process data for caregiving and for change? Did the interventions build on those patterns? Were the available data relevant for units that had to change? How did local practice environments encourage the linkage of data to care and its improvement by the clinical microsystems? Were local microsystems locally accountable for outcomes?

5. About the setting's self-awareness. Did the caregivers understand themselves to be part of an interdependent "˜system' and did they function that way as they considered changes? How did they understand their own work processes of HAART therapy and the other desired clinical outcomes such as immunization? Did they know their own processes well enough to connect measures of the variation to their work?

6. About the community surrounding the settings. What were the community HIV AIDS practices in the small area surrounding the settings? Did the sites intentionally adopt or differentiate themselves from those patterns? If so, why? Did the interventional design build on that knowledge? Was there geographic variation in the available health resources for the sites?

These and other evaluative studies with even greater detail can help us build a solid empirical base under efforts to change and improve clinical care.


Paul Batalden, M.D. Center for the Evaluative Clinical Sciences Dartmouth Medical School

Landon, et al. Effects of a Quality Improvement Collaborative on the Outcome of Care of Patients with HIV Infection:the EQHIV Study. Ann Int Med 2004,140:887-896.

Mittman,B.S. Creating the Evidence Base for Quality Improvement Collaboratives. Ann Int Med 2004,140:897-901

Conflict of Interest:

None declared

Submit a Comment/Letter

Summary for Patients

Do Quality Improvement Programs Improve Care for HIV-Infected Patients?

The summary below is from the full report titled “Effects of a Quality Improvement Collaborative on the Outcome of Care of Patients with HIV Infection: The EQHIV Study.” It is in the 1 June 2004 issue of Annals of Internal Medicine (volume 140, pages 887-896). The authors are B.E. Landon, I.B. Wilson, K. McInnes, M.B. Landrum, L. Hirschhorn, P.V. Marsden, D. Gustafson, and P.D. Cleary.


Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Healthcare Reform: Delivery Reform. Issue Brief Health Policy Track Serv 2015;():1-82.
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.