The full content of Annals is available to subscribers

Subscribe/Learn More  >
Improving Patient Care |

Public Reporting of Antibiotic Timing in Patients with Pneumonia: Lessons from a Flawed Performance Measure

Robert M. Wachter, MD; Scott A. Flanders, MD; Christopher Fee, MD; and Peter J. Pronovost, MD, PhD
[+] Article, Author, and Disclosure Information

From the University of California, San Francisco, San Francisco, California; University of Michigan School of Medicine, Ann Arbor, Michigan; and Johns Hopkins School of Medicine and School of Public Health, Baltimore, Maryland.

Potential Financial Conflicts of Interest:Consultancies: R.M. Wachter (American Board of Internal Medicine, Healthcare Advisory Board, Google). Honoraria: R.M. Wachter (Dr. Wachter has done more than 100 talks on issues of quality and patients safety to several health care organizations.). Grants received: R.M. Wachter (Agency for Healthcare Research and Quality).

Requests for Single Reprints: Robert M. Wachter, MD, Room M-994, Department of Medicine, 505 Parnassus Avenue, University of California, San Francisco, San Francisco, CA 94143-0120; e-mail, bobw@medicine.ucsf.edu.

Current Author Addresses: Dr. Wachter: Room M-994, Department of Medicine, 505 Parnassus, University of California, San Francisco, San Francisco, CA 94143-0120.

Dr. Flanders: University of Michigan, 3119F Taubman Center, 1500 East Medical Center Drive, Ann Arbor, MI 48109-03766.

Dr. Fee: University of California, Box 0208, 505 Parnassus, San Francisco, CA 94143-0208.

Dr. Pronovost: Departments of Anesthesiology and Critical Care, Surgery, and Health Policy and Management, The Johns Hopkins University School of Medicine, 1909 Thames Street, 2nd Floor, Baltimore, MD 21231.

Author Contributions: Conception and design: R.M. Wachter, S.A. Flanders, P.J. Pronovost.

Drafting of the article: R.M. Wachter, S.A. Flanders, C. Fee, P.J. Pronovost.

Critical revision of the article for important intellectual content: S.A. Flanders, C. Fee, P.J. Pronovost.

Final approval of the article: R.M. Wachter, S.A. Flanders, C. Fee, P.J. Pronovost.

Administrative, technical, or logistic support: R.M. Wachter.

Ann Intern Med. 2008;149(1):29-32. doi:10.7326/0003-4819-149-1-200807010-00007
Text Size: A A A

Improving health care quality depends on having valid ways to measure quality. Unfortunately, there are few validated quality outcome measurements, because valid and feasible case-mix adjustors are lacking and patients are difficult to follow over time for clinically important outcomes, such as death. Processes of care are easier to identify and measure, but some of these measures will be proven invalid or inappropriate because their scientific rationale was flawed from the start, unanticipated consequences emerge after implementation, or later studies undermine them.

First Page Preview

View Large
First page PDF preview





Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Improving Quality Metrics
Posted on July 24, 2008
Nadine Srouji
PinnacleHealth System
Conflict of Interest: None Declared
To the Editor: Your recent article reported that a widely-used performance measure, the administration of antibiotics within four hours to patients with pneumonia, may have led to increased inappropriate antibiotic use and less cost-effective care without decreasing mortality. This benchmark was used by Joint Commission, CMS, and insurance companies. CMS publicly reported hospital performance on this measure and some managed care companies tied reimbursements to it. From my own experience on a hospital committee charged with ensuring at least 90% compliance with this four hour requirement, I know that our organization allocated significant resources to accomplishing this goal. We surveyed hospitals that had successfully met the 4-hour benchmark by instituting a protocol whereby patients automatically received antibiotics if they met certain criteria in triage. The 4-hour benchmark changed patterns of care. The Joint Commission recently relaxed this window to six hours; seemingly in recognition of the metric's limitations. Yet no study has shown a benefit from a 6-hour rule. Isn't one of the main components of quality improvement a feedback loop? There has been much press recently about the Commonwealth Fund's report concluding that American health care, while expensive, does not rate accordingly high in quality when compared to other industrialized nations (1). Ironically, the report specifically identified pneumonia as one of the few diagnoses for which treatment has improved. This conclusion was based on three metrics, one of which was administration of antibiotics within four hours of patient presentation. Reporting on the Commonwealth findings, The New York Times quoted the president of the National Business Group on Health as saying that "it proves once again that if you have quantitative information and metrics and make people pay attention, they change (2)." How true and how concerning then when the metrics are wrong. Nadine Srouji, MD 1. The Commonwealth Fund Commission on a High Performance Health System, Why Not the Best? Results from the National Scorecard on U.S. Health System Performance, 2008, The Commonwealth Fund, July 2008. 2. Abelson, R. While the U.S. Spends Heavily on Health Care, a Study Faults the Quality. The New York Times. July 17, 2008. Conflict of Interest:

None declared

Timing of antibiotics and mortality in geriatric CAP patients
Posted on August 4, 2008
Manfred Gogol
Krankenhaus Lindenbrunn, Klinik fuer Geriatrie, Lindenbrunn 1, Coppenbruegge 31863, Germany
Conflict of Interest: None Declared
To the Editor: We are a tertiary community geriatric and neurologic hospital specialized for acute, sub-acute and rehab care. We fully agree with the paper by Wachter et al. (1) criticizing the flawed scientific basis of defining a time window for antibiotic administration for community-acquired pneumonia (CAP) in hospitalized patients. In 2006 and 2007 there were 40.428 CAP in-patients in Lower Saxony (Germany). In our hospital we treated 44 patients. Data derived from the quality measurement tool for hospitalized CAP patients we found no association between time to first antibiotic administration and mortality. Related to the fact that our patients have had a higher CRB-65-score, were older, coming more often from other hospitals or rehab units, were more bedridden, and cognitive impaired (see table), we add some more data to the growing evidence that questioned the sense of shorten door-to-antibiotics-time. In a time where some physicians believe in guidelines such as a bible it seems to be necessary to cite the last US guideline (2): "ž...the first antibiotic dose should be administered while still in the ED" is judged only as "žModerate recommendation; level III evidence.", p. S30), but "žFor these and other reasons, the committee did not feel that a specific time window for delivery of the first antibiotic dose should be recommended. However, the committee does feel that therapy should be administered as soon as possible after the diagnosis is considered likely" (p. S54). We will hope that prospective, randomized controlled trials would lead us to more reasonable quality measurement tools in future. 1. Wachter RM, Flanders SA, Fee C, Pronovost PJ. Public reporting of antibiotic timing in patients with pneumonia: lessons from a flawed performance measure. Ann Intern Med 2008;149:29-32. 2. Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell CG et al. Infectious Diseases Society of America/American Thoracic Society Consensus Guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis 2007;44 (Suppl 2):S27-S72. Conflict of Interest:

None declared

Consequences of Flawed Performance Measures on a Local Pneumonia Guideline
Posted on August 8, 2008
Nathan C. Dean
Intermountain Health Care and University of UtahE
Conflict of Interest: None Declared
I read with interest the recent "improving patient care" article by Dr. Wachter and colleagues (Ann Intern Med 149:29). An additional consequence of CMS policies has been to disrupt local pneumonia guideline processes. The 2007 American Thoracic Society-Infectious Disease Society of America pneumonia guideline recommended that "locally-adapted guidelines should be implemented to improve process of care variables and relevant clinical outcomes". (1) In addition, ""CAP guidelines should address a comprehensive set of elements in the process of care, rather than a single element in isolation". A guideline was implemented in Intermountain Healthcare System (Utah and Idaho) beginning in 1995, with demonstrated decrease in 30 day all cause mortality and decreased rate of pneumonia admission.(2,3) As part of multiple care elements, Intermountain's guideline recommends specific antibiotics (e.g. ceftriaxone plus azithromycin for admitted patients), and recommends that antibiotics be administered as soon as the diagnosis of pneumonia has been confirmed, at the site of initial care. With initiation of public reporting of CMS criteria, a competing focus developed within Intermountain Healthcare to achieve antibiotic administration within a 4, and now 6 hour window, instead of our prior standard. In addition, the CMS list of acceptable antibiotics is broader than our local guideline, leading physicians at Dixie Regional Medical Center (St. George, Utah) to rewrite the local guideline to include CMS accepted antibiotics such as ertapenem. Compliance with Intermountain's guideline as measured by initial antibiotic prescribed fell at Dixie Regional from 90% in January 2005 to only 55% in January 2008.(4) While Intermountain Healthcare as a system has over 90% compliance with the CMS antibiotic timing measure, the focus of internal processes has shifted to meeting these performance measures, instead of reinforcing the use of the local guideline. An earlier consequence of CMS payment policies was to reduce payments to Intermountain Healthcare for patients with pneumonia by approximately $500,000 yearly.(5)Revenue loss resulted from our guideline treating more pneumonia patients at home, since reimbursement for care of a hospitalized pneumonia patient was much higher than care of a similar patient at home. Unlike CMS pneumonia performance measures, our guideline has been shown to improve patient outcomes. Unlike reporting of specific care elements, severity adjusted measures of outcome such as survival to discharge home, time to return to usual activities, and 30 day mortality would support local guideline development and implementation. Real improvement in pneumonia outcomes would likely result, instead of the results we are seeing from well intended but flawed performance measures.


1) Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC,Dowell SF, File TM Jr, Musher DM, Niederman MS, Torres A, Whitney CG. (2007). Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis 44 Suppl 2, pp. S27-72.

2) Suchyta MR, Dean NC, Narus S, Hadlock CJ. (2001). Effects of a practice guideline for community-acquired pneumonia in an outpatient setting. Am J Med, 110(4), 306-9.

3) Dean NC, Bateman KA, Donnelly SM, Silver MP, Snow GL, Hale D. (2006). Improved clinical outcomes with utilization of a community-acquired pneumonia guideline. Chest, 130(3), 794-9.

4) Intermountain.net (internal system website) "Compliance rate for community acquired pneumonia" for Dixie Regional Medical Center accessed 8/08/2008.

5) Porter ME and Teisberg EO. "Redefining Health Care" page 263 Harvard Business School Press, Boston, MA 2006

Conflict of Interest:

None declared

TFAD- A measure flawed in many ways
Posted on August 15, 2008
P Dileep Kumar
Port Huron Hospital
Conflict of Interest: None Declared
Watcher and colleagues (1) convincingly argue about the futility of using the time to first antibiotic use (TAFD) in community acquired pneumonia as a core measure of quality. The authors seem to kid-glove the various organizations involved in the selection of this flawed measure. These organizations' abdication of responsibility after the administration of several million doses of probably unnecessary antibiotics is interesting. Instead of stretching the 4 hr rule, they should have either looked for other measures or planned prospective randomized trials. Another major problem with the pneumonia measure is that it is nondiscriminatory with regard to the etiology. Even though this measure was intended solely for community acquired pneumonia, other types of pneumonia are often clumped together. CMS website showing the comparison of different hospital performance on various quality measures does not discriminate between different types of pneumonia 1. Wachter RM, Flanders SA, Fee C, Pronovost PJ. Public reporting of antibiotic timing in patients with pneumonia: lessons from a flawed performance measure. Ann Intern Med. 2008 Jul 1;149(1):29-32. Conflict of Interest:

None declared

Authors' Response to Letters on Flawed Antibiotic Timing Measure
Posted on August 18, 2008
Robert M. Wachter
University of California, San Francisco
Conflict of Interest: None Declared
To the Editors: We appreciate the letters, which endorse our main premise while adding new data and insights. Although we agree with most of their points, we worry that the experience of a flawed measure such as door-to-antibiotics for pneumonia will lead some to throw out the baby (quality measurement and transparency) with the bathwater (the bad measure). For example, Dean argues that the imposition of a national standard for pneumonia care undermined their homegrown pneumonia strategy. We agree that national guidelines should provide enough flexibility to allow for individual institutional choice based on local factors such as cost, resistance, and ease of administration "“ as long as the choices are compatible with the best evidence. In fact, Intermountain's preferred antibiotics were on the list of recommended antibiotics. That said, we are concerned about the generalizability of the authors' example. Intermountain Healthcare is a large, highly evolved system with strong infrastructure, including world-class information technology (1). Most institutions around the country don't look like that. Substantial evidence supports the value of widespread adherence to evidence-based standards (2-3). We believe it would be a mistake to eschew thoughtful, evidence-based national guidelines because some organizations with the capacity to develop and study local guidelines might need to subsume their work to these national standards. Organizations with such capacity should become learning laboratories, testing existing guidelines and standards for effectiveness and engaged in the process of developing future evidence-based national guidelines. Gogol states that they were skeptical of the value of the door-to-antibiotic measure because antibiotic timing had no impact on outcomes in their population of 44 pneumonia patients. Here too, individual institutions may not be able to generate the statistical power to identify even significant effects of practice changes. That should not be cause to shun national measures. But it is yet another reason to be sure that national standards are based on strong evidence, have been field-tested, and are reviewed frequently for unintended consequences. Luckily, in part based on the door-to-antibiotic experience, both the National Quality Forum and the Centers for Medicare & Medicaid Services have begun to change their processes to ensure that future measures are less likely to lead to negative unintended consequences (4). This is particularly important since, as Srouji correctly observes, there is no question that the public reporting of quality measures does have the intended effect: to change clinical practice. For better or worse. 1. Hougaard J. Developing evidence-based interdisciplinary care standards and implications for improving patient safety. Int J Med Inform 2004;73:615-24. 2. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA. The quality of health care delivered to adults in the United States. N Engl J Med 2003; 348:2635-45. 3. Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ, Lucas FL, Pinder EL. The implications of regional variations in Medicare spending. Part 1: the content, quality, and accessibility of care. Ann Intern Med 2003;138:273-87. 4. Smith Moore T, Francis MD, Corrigan JM. Health care quality in the 21st century. Clin Exp Rheumatol 2007; 25:3-5. Conflict of Interest:

None declared

Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.