David W. Baker, MD, MPH; Amir Qaseem, MD, PhD, MHA; for the American College of Physicians' Performance Measurement Committee
Potential Conflicts of Interest: Dr. Qaseem was an author of the American College of Physicians' guideline on VTE prophylaxis. Forms can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-2065.
Requests for Single Reprints: David W. Baker, MD, MPH, Feinberg School of Medicine at Northwestern University, 750 North Lake Shore Drive, Chicago, IL 60611; e-mail, firstname.lastname@example.org.
Current Author Addresses: Dr. Baker: Feinberg School of Medicine at Northwestern University, 750 North Lake Shore Drive, Chicago, IL 60611.
Dr. Qaseem: American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.
Baker D., Qaseem A., ; Evidence-Based Performance Measures: Preventing Unintended Consequences of Quality Measurement. Ann Intern Med. 2011;155:638-640. doi: 10.7326/0003-4819-155-9-201111010-00015
Download citation file:
Published: Ann Intern Med. 2011;155(9):638-640.
Quality measurement and pay-for-performance programs can provide strong incentives to improve quality (1-3). However, performance measures can also have unintended consequences.
For example, consider performance measures related to antibiotic timing for patients presenting to hospitals with community-acquired pneumonia (4). The Medicare National Pneumonia Project adopted “first antibiotics within 8 hours of hospital arrival” as a national quality measure in 1998 and shortened the time window to 4 hours in 2002. The Centers for Medicare & Medicaid Services adopted this more strict measure as one of their initial core quality measures (PN-5b) for public reporting. In 2006, the PN-5b measure became part of a set used by several pilot pay-for-performance programs (5), and some hospitals gave bonuses to nonphysician staff who met a performance goal of 90% (6). However, the diagnosis of community-acquired pneumonia is often unclear during the initial evaluation, and the appropriate management of a stable patient is often to withhold antibiotics pending a more certain diagnosis. Yet, with public reporting and financial pressures in place, the PN-5b measure became the sword of Damocles hanging above physicians trying to choose between the best treatment for patients or following a national performance standard. Concerns arose that the measure was leading to unnecessary and inappropriate prescribing of antibiotics (7). Subsequent studies confirmed these fears (8-9) and failed to confirm that early administration of antibiotics decreased mortality in stable patients with community-acquired pneumonia (10). In response, the Joint Commission relaxed the time window to 6 hours and created a new data element of “diagnostic uncertainty” that can be used to exclude patients from the measure.
In this issue, an American College of Physicians clinical practice guideline (11) and background review (12) raise similar concerns about performance measurement for prophylaxis of venous thromboembolism (VTE) for patients hospitalized for medical conditions or stroke. The guideline concludes that evidence does not support routine VTE prophylaxis in all medical patients. Risks for VTE and bleeding vary substantially according to individual risk factors, and prophylaxis may actually harm patients who are at low risk for VTE or at high risk for bleeding. The guideline recommends that clinicians assess the individual patient's risk and benefit before initiating VTE prophylaxis. Moreover, it recommends against the use of performance measures that do not incorporate risk assessment. Similarly, current recommendations from the American College of Chest Physicians (ACCP) (13) and the National Quality Forum (14) specify that risk assessment guide the decision regarding prophylaxis.
There is a disturbing disconnect between the evidence-based guidelines and the Joint Commission's current performance measure for VTE prophylaxis (VTE-1) for surgical and hospitalized medical patients, which states, “This measure assesses the number of patients who received VTE prophylaxis or have documentation why no VTE prophylaxis was given the day of or the day after hospital admission” (15). Although very-low-risk patients are excluded (children and patients hospitalized for <2 days), the measure does not consider other patients who are unlikely to benefit from VTE prophylaxis or for whom the risk for harm exceeds the likely benefits. We believe that the measure implicitly encourages prophylaxis for all patients. Many hospitals have implemented it for medical patients as an internal quality measure. We believe that this well-intentioned mandate may be causing some low- to moderate-risk patients to receive prophylaxis, wasting resources, and perhaps harming some patients.
How can we mitigate unintended consequences from performance measurement?
First, it is essential that national performance measures be based on high-quality scientific evidence (16). The process for reviewing the evidence and developing measures must be protected from financial and nonfinancial conflicts of interest. Developers of performance measures and organizations that endorse such measures should use a rigorous system for grading the strength of evidence, such as the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) system (17). In addition, the methods used to develop a measure must be transparent to help stakeholders understand its validity and the expected health benefits of improving performance—merely citing references is not enough. The description of the Joint Commission's VTE prophylaxis measure references the ACCP guideline; however, as discussed, the ACCP guideline advocates VTE prophylaxis only for high-risk subgroups, whereas the Joint Commission measure includes most patients hospitalized with medical conditions.
Second, measure developers should carefully examine whether the benefits and risks of a test or treatment are homogeneous. Even if studies show that a clinical service is beneficial for an overall study population, the benefit may be large for some subgroups and negligible (or even harmful) for others. If the benefit–risk ratio is minimal or uncertain, use of a therapy should be left to the clinicians' judgment and not be encouraged through performance measurement. For example, the Joint Commission's VTE-1 measure could have been restricted to the subgroups of medical patients whom the ACCP designated as having “level A” evidence.
Third, all measures should routinely allow clinicians to report exceptions for not following a recommendation. The Joint Commission added the “diagnostic uncertainty” exception for the PN-5b measure for community-acquired pneumonia after concerns were raised about the measure, and the Joint Commission's VTE-1 measure can be satisfied if there is a documented reason why no VTE prophylaxis was given. When exception reporting is allowed, studies show that most exceptions reported by clinicians are valid (18-19). Ideally, exceptions should be recorded in electronic health records so that they can be easily audited to ensure accuracy, ensure that individual physicians or groups do not report very high rates of exceptions, and deter gaming. However, allowing documentation of a reason for not performing a recommendation in an unusual circumstance is not a substitute for accurate, evidence-based specification of the patient subgroups that should be targeted for measurement and quality improvement efforts.
Fourth, we must develop more sophisticated measures that can be programmed into flexible, electronic, clinical decision-support rules. These could take the form of more restrictive eligibility criteria, such as the patient subgroups described for the VTE prophylaxis measure. Corresponding point-of-care alerts or reminders designed to improve performance would only be triggered for patients in the same high-risk subgroups identified in the measure specifications. For some measures, more sophisticated multivariable risk rules can be applied, such as for identifying patients at high risk for cardiovascular events (for example, the Framingham risk score) who should be offered statin therapy. Several risk prediction methods have been proposed for VTE prophylaxis, but none has been endorsed for widespread use at this time. Further research is needed to refine and validate these VTE risk prediction rules, including assessment of the feasibility of implementing them into electronic health record systems.
Finally, to mitigate the risk for inducing unnecessary or harmful care, performance measure developers may need to allow multiple pathways to satisfy a measure. For example, when the Northwestern Medical Faculty Foundation sought to implement a performance measure for controlling low-density lipoprotein cholesterol levels (<2.59 mmol/L [<100 mg/dL]) in patients with diabetes, it recognized that some patients might not achieve this goal even if they were receiving a potent statin. Failure to recognize this in our performance measures and clinical decision-support tools could result in quality improvement efforts that encourage physicians to add additional medications to further reduce low-density lipoprotein cholesterol levels, even though such treatments have not been shown to improve outcomes. Therefore, the group decided that clinicians could satisfy the measure by prescribing a moderate to high dose of a potent statin, even if the low-density lipoprotein cholesterol level remains greater than 2.59 mmol/L (>100 mg/dL) and they used the same logic to suppress the electronic alert.
As we enter the age of personalized medicine and strive for high-value, cost-conscious care, we need flexible performance measures. Failure to develop and faithfully implement these more sophisticated measures will waste resources and harm some patients—the ultimate irony, as the rationale for performance measurement is to foster high-quality, high-value health care.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
AzeemMajeed, Professor of Primary Care
Imperial College London
November 5, 2011
Unintended Consequences of Quality Measurement
In recent years, pay for performance programs and public reporting of provider performance have become integral components of strategies to improve the quality of health care in many countries. However, as highlighted by David Baker and Amir Qaseem, quality measurement can also have unintended consequences. Among the measures that Baker and Qaseem advocate to reduce such unintended consequences is the ability of physicians to report exceptions for not following a recommendation. Although 'exception reporting' should be an essential component of quality measurement to ensure that patients who will not benefit from an intervention (such as those who are terminally ill) are not treated inappropriately, at the same time, it is important to monitor the use of exception reports at both patient and provider level.
In the United Kingdom, the Quality and Outcomes Framework, a major pay-for-performance scheme introduced into UK primary care in 2004, permits primary care physicians to 'exception report' patients from the quality targets in the framework. One potential disadvantage of allowing exception reporting is that physicians might exclude patients in whom they have difficulty reaching targets to increase their quality achievement scores. Excluded patients may then receive suboptimal care and, if so, pay-for-performance programs and quality measurement could worsen existing disparities in health care if already disadvantaged groups of patients are more likely to be excluded from measurement of achievement of quality targets.
Analysis of data on exception reporting rates at provider level in the UK Quality & Outcomes Framework shows a wide variation between primary care practices, particularly for indicators more closely linked to clinical outcomes, such as HBA1c control in people with diabetes. At patient level, when we examined exception reporting rates amongst people with diabetes from 23 primary care practices from a socio-economically and ethnically diverse part of London, we found that patients who were exception reported by their physicians were significantly less likely to achieve treatment targets for HbA1c, blood pressure and cholesterol control. There was considerable variation in exception reporting by patient characteristics (such as age, ethnicity, co-morbidity, and duration of illness). Highest rates of exception reporting occurred among already disadvantaged groups at greatest risk of diabetes complications and poorer health outcomes such as older patients; and patients from ethnic minority groups, with longer duration of diabetes and with increased levels of comorbidity.
The experience from the United Kingdom highlights the importance of monitoring exclusions from pay-for-performance and public reporting programmes. Failure to adequately monitor exception reporting and the reasons why physicians and providers exclude patients from public reporting may lead to the patients who are most at risk of adverse health outcomes being excluded from these schemes and thus from receiving evidence-based care. This in turn may exacerbate current health disparities and worsen health outcomes in these vulnerable groups of patients.
1. Baker DW, Qaseem A. Evidence-Based Performance Measures: Preventing Unintended Consequences of Quality Measurement. Annals of Internal Medicine 2011; 155: 638-640.
2. Ashworth M, Millett C. Quality improvement in UK primary care: the role of financial incentives. J Ambul Care Manage 2008; 31: 220-225.
3. Doran T, Fullwood C, Reeves D, Gravelle H, Roland M. Exclusion of patients from pay-for-performance targets by English physicians. N Engl J Med 2008; 359: 274-284.
4. Dalton ARH, Alshamsan R, Majeed A, Millett C. Exclusion of patients from quality measurement of diabetes care in the UK pay-for-performance programme. Diabetic Medicine 2011; 28: 525-531.
5. Millett C, Gray J, Saxena S, Netuveli G, Khunti K, Majeed A. Ethnic disparities in diabetes management and pay-for-performance in the UK: The Wandsworth Prospective Diabetes Study. PLoS Med 2007; 4: e191.
I am a general practitionerin the primary care practice of Dr Curran & Partners in Clapham, London. The Department of Primary Care & Public Health at Imperial College London has received funding from the NHS for research on pay for performance.
Dale W.Bratzler, DO, MPH, Jerod M. Loeb, PhD, Denise Krusenoski, MSN, RN, The Joint Commission, Oakbrook Terrance, IL
University of Oklahoma Health Sciences Center, College of Public Health, Oklahoma City, OK
November 16, 2011
VTE Prophylaxis Performance Measures
Venous thromboembolism (VTE) remains a common complication of hospital care (1, 2). Multiple medical record audits of high-risk medical and surgical patients have consistently shown underuse of VTE prophylaxis (2). Most patients who are sick enough to require hospitalization have risk factors for VTE. Pulmonary embolism is a common and potentially preventable cause of death from hospital care (1, 2).
In their editorial, Baker and Qaseem (3) criticize The Joint Commission's performance measure on VTE prophylaxis (VTE-1) for hospitalized patients because it purportedly encourages use of VTE prophylaxis for hospitalized patients unless there is documentation in the medical record showing that the patient is not at risk for VTE. We believe this measure construct, using a permissible exclusion remains appropriate.
In 2006 and 2007, the National Quality Forum (NQF) convened a steering committee and a technical advisory panel to evaluate existing and develop new measures to address prevention and treatment of VTE (4). These panels included representation from multiple stakeholders including two authors of the American College of Chest Physicians (ACCP) clinical practice guideline. The measures were tested by The Joint Commission in multiple hospitals and were posted for comment during the consensus development process at NQF.
The decision to promote routine VTE prophylaxis or require documentation that no prophylaxis was needed was based on careful consideration of the ACCP guideline recommendations. The ACCP did not advocate patient-specific assessment for VTE risk due to the lack of appropriately-derived and validated risk assessment tools, and stated that "there is sufficient evidence to recommend routine thromboprophylaxis for most hospitalized patient groups (2)." The ACCP also noted that "we are not able to confidently identify the small population of patients in the various groups who do not require thromboprophylaxis (2)."
While the ACCP highlighted the lack of clinically-validated risk- assessment models to guide thromboprophylaxis, a careful review of the performance measure specifications for VTE-1 shows that the measure construct allows documentation of the lack of need for VTE prophylaxis based on completed hospital risk assessment forms (5). The specifications also give complete discretion to the clinician at the bedside to decide if prophylaxis is indicated for their patient, and to chose the type of thromboprophylaxis (pharmacologic or mechanical) if indicated for their patients based on an assessment of the potential risks (5). This approach is consistent with the ACCP and NQF recommendations for each hospital to develop written, institution-wide thromboprophylaxis policies (2, 4).
1. Qaseem A, Chou R, Humphrey LL, Starkey M, Shekelle P; American College of Physicians. Venous thromboembolism prophylaxis in hospitalized patients: a clinical practice guideline from the American College of Physicians. Ann Intern Med. 2011;155:625-632. [PMID: 22041951]
2. Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al; American College of Chest Physicians. Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest. 2008;133:381S-453S. [PMID: 18574271]
3. Baker DW, Qaseem A. Evidence-based performance measures: preventing unintended consequences of quality measurement. Ann Intern Med. 2011;155:638-640. [PMID: 22041954]
4. National Quality Forum. National Voluntary Consensus Standards for Prevention and Care of Venous Thromboembolism: Phase II. Accessed at http://www.qualityforum.org/Projects/s- z/VTE_Phase_II_(2008)/VTE_Phase_II.aspx on 8 November 2011.
5. The Joint Commission. Specifications Manual for National Hospital Inpatient Quality Measures. Accessed at http://www.jointcommission.org/specifications_manual_for_national_hospital_inpatient_quality_measures/ on 8 November 2011.
Venous Thromboembolism, Healthcare Delivery and Policy, High Value Care, Prevention/Screening.
Results provided by:
Copyright © 2016 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use
This PDF is available to Subscribers Only