0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Original Research |

An Analysis of Calibration and Discrimination Among Multiple Cardiovascular Risk Scores in a Modern Multiethnic CohortCalibration and Discrimination Among CVD Risk Scores

Andrew P. DeFilippis, MD, MSc*; Rebekah Young, PhD*; Christopher J. Carrubba, MD; John W. McEvoy, MB, BCh, BAO; Matthew J. Budoff, MD; Roger S. Blumenthal, MD; Richard A. Kronmal, PhD; Robyn L. McClelland, PhD; Khurram Nasir, MD, MPH; and Michael J. Blaha, MD, MPH
[+] Article, Author, and Disclosure Information

* Drs. DeFilippis and Young contributed equally to this work.


From the University of Louisville, Louisville, Kentucky; University of Washington, Seattle, Washington; University of Colorado, Boulder, Colorado; Johns Hopkins Medicine Ciccarone Center for the Prevention of Heart Disease, Baltimore, Maryland; Los Angeles Biomedical Research Institute at Harbor–UCLA Medical Center, Torrance, California; and Baptist Medical Group, Miami, Florida.

Acknowledgment: The authors thank the other investigators, the staff, and the participants of MESA for their valuable contributions. A full list of participating MESA investigators and institutions can be found at www.mesa-nhlbi.org.

Grant Support: This research was supported by contracts N01-HC-95159 through N01-HC-95169 from the National Heart, Lung, and Blood Institute and by grants UL1-TR-000040 and UL1-TR-001079 from the National Center for Research Resources.

Disclosures: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M14-1281.

Reproducible Research Statement:Study protocol: Available at www.mesa-nhlbi.org. Statistical code: Requests will be considered by Dr. Rebekah Young on a case-by-case basis. Data set: Applications for use of MESA data are available to the public at www.mesa-nhlbi.org.

Requests for Single Reprints: Andrew P. DeFilippis, MD, MSc, University of Louisville, Division of Cardiovascular Medicine, 550 South Jackson Street, Louisville, KY 40202; e-mail, APDeFi01@louisville.edu.

Current Author Addresses: Dr. DeFilippis: University of Louisville, Division of Cardiovascular Medicine, Ambulatory Care Building, Cardiology, 550 South Jackson Street, Louisville, KY 40202.

Dr. Young: Collaborative Health Studies Coordinating Center, Building 29, Suite 210, University of Washington, Box 354922, 6200 Northeast 74th Street, Seattle, WA 98115.

Dr. Carrubba: University of Colorado, 12631 East 17th Avenue, B177, Academic Office 1, Aurora, CO 80045.

Mr. McEvoy and Dr. Blumenthal: The Johns Hopkins Hospital, Blalock 524C, 1800 Orleans Street, Baltimore, MD 21287.

Dr. Budoff: Los Angeles Biomedical Research Institute at Harbor–UCLA Medical Center, 1124 West Carson Street, Torrance, CA 90502.

Drs. Kronmal and McClelland: Collaborative Health Studies Coordinating Center, Building 29, Suite 210, University of Washington, Box 354922, 6200 Northeast 74th Street, Seattle, WA 98115.

Dr. Nasir: Baptist Health Medical Group, Center for Prevention and Wellness Research, 1691 Michigan Avenue, Suite 500, Miami, FL 33139.

Dr. Blaha: Johns Hopkins Ciccarone Center for the Prevention of Heart Disease, Carnegie 568A, Johns Hopkins Hospital, 600 North Wolfe Street, Baltimore, MD 21287.

Author Contributions: Conception and design: A.P. DeFilippis, R. Young, M.J. Blaha.

Analysis and interpretation of the data: A.P. DeFilippis, R. Young, C. Carrubba, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, M.J. Blaha.

Drafting of the article: A.P. DeFilippis, R. Young, C. Carrubba, R.A. Kronmal, M.J. Blaha.

Critical revision of the article for important intellectual content: A.P. DeFilippis, R. Young, J.W. McEvoy, M.J. Budoff, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, K. Nasir, M.J. Blaha.

Final approval of the article: A.P. DeFilippis, R. Young, J.W. McEvoy, M.J. Budoff, R.S. Blumenthal, R.A. Kronmal, R.L. McClelland, K. Nasir, M.J. Blaha.

Provision of study materials or patients: M.J. Budoff.

Statistical expertise: R. Young, R.A. Kronmal, R.L. McClelland, M.J. Blaha.

Administrative, technical, or logistic support: A.P. DeFilippis, R.L. McClelland, M.J. Blaha.

Collection and assembly of data: R. Young, R.A. Kronmal, R.L. McClelland, M.J. Blaha.


Ann Intern Med. 2015;162(4):266-275. doi:10.7326/M14-1281
Text Size: A A A

Background: Accurate risk assessment of atherosclerotic cardiovascular disease (ASCVD) is essential to effectively balance the risks and benefits of therapy for primary prevention.

Objective: To compare the calibration and discrimination of the new American Heart Association (AHA) and American College of Cardiology (ACC) ASCVD risk score with alternative risk scores and to explore preventive therapy as a cause of the reported risk overestimation using the AHA-ACC-ASCVD score.

Design: Prospective epidemiologic study of ASCVD.

Setting: MESA (Multi-Ethnic Study of Atherosclerosis), a community-based, sex-balanced, multiethnic cohort.

Patients: 4227 MESA participants aged 50 to 74 years and without diabetes at baseline.

Measurements: Observed and expected events for the AHA-ACC-ASCVD score were compared with 4 commonly used risk scores—and their respective end points—in MESA after a 10.2-year follow-up.

Results: The new AHA-ACC-ASCVD and 3 older Framingham-based risk scores overestimated cardiovascular events by 37% to 154% in men and 8% to 67% in women. Overestimation was noted throughout the continuum of risk. In contrast, the Reynolds Risk Score overestimated risk by 9% in men but underestimated risk by 21% in women. Aspirin, lipid-lowering or antihypertensive therapy, and interim revascularization did not explain the overestimation.

Limitation: Comparability of MESA with target populations for primary prevention and possibility of missed events in MESA.

Conclusion: Of the 5 risk scores, 4, including the new AHA-ACC-ASCVD score, showed overestimation of risk (25% to 115%) in a modern, multiethnic cohort without baseline clinical ASCVD. If validated, overestimation of ASCVD risk may have substantial implications for individual patients and the health care system.

Primary Funding Source: National Heart, Lung, and Blood Institute.

Figures

Grahic Jump Location
Figure 1.

Risk score–specific predicted and observed events in men, by decile of calculated risk.

Hosmer–Lemeshow calibration plots for men (n = 1961). ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Risk score–specific predicted and observed events in women, by decile of calculated risk.

Hosmer–Lemeshow calibration plots for women (n = 2266). ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score.

Grahic Jump Location
Grahic Jump Location
Appendix Figure.

Area under the curve for all 5 risk prediction models.

ACC = American College of Cardiology; AHA = American Heart Association; ASCVD = atherosclerotic cardiovascular disease; ATPIII = Adult Treatment Panel III; CHD = coronary heart disease; CVD = cardiovascular disease; FRS = Framingham risk score; RRS = Reynolds Risk Score. Left. Men (n = 1961). Right. Women (n = 2266).

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment/Letter
Comment
Posted on March 20, 2015
David C. Goff, Jr. MD PhD, Ralph B. D’Agostino, Sr., PhD, Michael Pencina, PhD, Donald M. Lloyd-Jones, MD ScM
Colorado School of Public, Boston University, Duke University, Northwestern University
Conflict of Interest: None Declared
DeFilippis et al.(1) evaluated the predictive utility of the 2013 ACC/AHA Pooled Cohort Equations in the MESA cohort, and reported what they perceive to be over-prediction. This potential issue was identified and thoroughly discussed in the guidelines.(2,3) DeFilippis et al. attempted to explore it in more detail, but there are substantial methodological problems with their analyses:
1. MESA participants have undergone coronary artery calcium (CAC) scoring at least twice during the study. Knowledge of the results was associated with greater initiation of preventive therapies, especially among those at higher risk or with higher CAC scores.(4) Thus, those more likely to have ASCVD events were also more likely to be treated, which undoubtedly contributed to “under-performance” of MESA event rates compared to the predicted natural history.
2. The use of preventive therapies in the MESA cohort is extraordinarily high compared with the US population (as shown in Table 2(1)). From baseline to follow up, aspirin use increased from roughly 25% to 55% (ever), blood pressure medications from 35% to 60%, lipid lowering drugs from 15% to 44%, and any drug use from 53% to over 80%. It is therefore impossible to use these data to evaluate the accuracy of estimated natural history risk.
3. DeFilippis attempted to account for use of preventive treatments with several methods, all of which are flawed. Simple statistical adjustments for use, or censoring at initiation of therapy, clearly do not account for treatment effects. The approach of excluding all participants who were ever treated yielded a highly biased analysis subject to major confounding by indication. Furthermore, only 14 (6%) of 218 observed events occurred in untreated individuals, leading to highly unstable estimates.
4. They appear to have included Chinese- and Hispanic-Americans, who comprised one-third of the MESA cohort. The ACC/AHA Pooled Cohort Equations were derived in and strongly recommended for Non-Hispanic Whites and African-Americans, with a weak recommendation for others. The guideline noted that the equations would overestimate risk in Hispanic- and East Asian-American groups. A more appropriate analysis would have looked at each race/ethnic group separately.

The concerns enumerated above raise important questions about the interpretation of DeFillipis’ findings. The new ACC/AHA guidelines represent a major advance that will result in reductions in the population burden of cardiovascular disease.(5,6) While future and better evidence will likely lead to even better guidelines, implementation of the current guidelines should be a national priority now.

REFERENCES
1. DeFilippis AP, Young R, Carrubba CJ, McEvoy JW, Budoff MJ, Blumenthal RS, et al. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med. 2015;162(4):266-75.
2. Goff DC, Jr., Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Sr., Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63(25 Pt B):2935-59.
3. Goff DC, Jr., Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25 Suppl 2):S49-73.
4. Nasir K, McClelland RL, Blumenthal RS, Goff DC, Jr., Hoffmann U, Psaty BM, et al. Coronary artery calcium in relation to initiation and continuation of cardiovascular preventive medications: The Multi-Ethnic Study of Atherosclerosis (MESA). Circ Cardiovasc Qual Outcomes. 2010;3(3):228-35.
5. Paixao AR, Ayers CR, Berry JD, de Lemos JA, Khera A. Atherosclerotic cardiovascular disease prevention: a comparison between the third adult treatment panel and the new 2013 Treatment of Blood Cholesterol Guidelines. Circ Cardiovasc Qual Outcomes. 2014;7(5):778-9.
6. Pencina MJ, Navar-Boggan AM, D'Agostino RB, Sr., Williams K, Neely B, Sniderman AD, et al. Application of new cholesterol guidelines to a population-based sample. N Engl J Med. 2014;370(15):1422-31.

Test sensitivity matters
Posted on April 1, 2015
Michel Romanens
Vascular Risk Foundation, Switzerland
Conflict of Interest: None Declared
It is with great interest that we read the work entitled “An Analysis of Calibration and Discrimination Among Multiple Cardiovascular Risk Scores in a Modern Multiethnic Cohort” (1). However, there are some issues about the role of such risk scores in clinical preventive medicine, especially with respect to the way, performance of risk scores are appreciated.
Risk scores are designed to assist the clinician to identify those subjects who benefit most from preventive medicine in terms of avoidance of cardiovascular events. This clinical work is being performed in a setting, where an “a priori” situation exists. The analyses De Filippis et al may be misleading, in that they look at the aggregated level of risk score performances in an “a posteriori” situation, where obsolescence for such risk scores may occur. Since prevention has to start early in life, many subjects who will develop cardiovascular events in 20 or 30 years may be classified as true negative cases during the shorter observation times. From a clinical point of view, the ratio of those with a future event (true positive cases) and those in whom a future event is erroneously missed (false negative) remains extremely important in this “a priori” setting, because – in contrast to the view of public health and economy – the clinician aims at identifying those “at risk” and not those “not a risk”.
Therefore, the classical relation between sensitivity and specificity – even in times where analysis of calibration, discrimination, and reclassification is highly prevalent – remains important. Sensitivity performance remains the benchmark for the clinician and specificity the one of the economy and public health people. As an example, applying risk scores to an individual patient would be senseless, if sensitivity is below 30%, because the number of missed cases (false negatives) creates a situation, where most subjects who will experience an event, are not identified. Therefore, efforts to identify those few true positive cases among all those having a true negative and a false negative situation, is frustrating and a real waste of resources. This is the case, where e.g. in 1000 subjects 140 will experience an event in 10 years (prevalence 14%), but only 20 of these 140 subjects are characterized as having a ten year risk of over 20% (sensitivity 14%). On the other hand, specificity in this example would be extremely high (95%: 820 true negatives and 40 false positives), giving an acceptable all over accuracy of 84%. Therefore, from a public health and economical view, where the performance of risk scores is judged from an “a posteriori” view over a quite limited observation time and the inherent problem of missed and therefore mislabeled cases, specificity driven aggregated calculations using calibration and discrimination analysis are justified. From a “subjective” clinical point of view, resources should be applied to risk scores calibrated at a sensitivity of at least 33%. In order to understand the clinical impact of the work presented, authors should recalculate their data and show clinicians the sensitivity and specificity of the different risk scores at the cutoff level of ≥20% coronary risk in 10 years (irrespective of the endpoints chosen) and recalibrate the risk score, where they use an exact sensitivity of 33% to detect a 10 year risk of ≥20%. From this, a recalibration factor for the different risk scores using the MESA population should be calculated in order to obtain a sensitivity of at least 33%. If risk scores then would show an unacceptably low specificity – where acceptability would have to be defined based upon a countries financial resources – the question emerges, if risk scores should be used at all or if risk scores should be used only in conjunction with emerging risk tools, e.g. those used in the BioImage Study (2), where informations about emerging test sensitivity and specificity could be integrated into an individuals clinical situation using a posttest coronary risk calculator, therefore creating a more individualized risk assessment (3).


References

1. DeFilippis AP, Young R, Carrubba CJ, et al. An Analysis of Calibration and Discrimination Among Multiple Cardiovascular Risk Scores in a Modern Multiethnic Cohort. Ann Intern Med. 2015;162:266-275.

2. Baber U, Mehran R, Sartori S, et al. Prevalence, Impact, and Predictive Value of Detecting Subclinical Coronary and Carotid Atherosclerosis in Asymptomatic Adults. J Am Coll Cardiol. 2015;65:1065-1074.

3. Romanens M, Ackermann F, Spence JD, et al. Improvement of cardiovascular risk prediction: time to review current knowledge, debates, and fundamentals on how to assess test characteristics. Eur J Cardiovasc Prev Rehab. 2010;17:18-23.
Author's Comment
Posted on April 10, 2015
Andrew P. DeFilippis, MD, MSc, Revekah Young, PhD, Michael J. Blaha, MD, MPH
University of Louisville, University of Washington, Johns Hopkins Medical
Conflict of Interest: None Declared
A distinguished author group, including several guideline writers, raised four issues related to cardiovascular risk prediction and our recent publication in the Annals of Internal Medicine.(1)
1. The study cited by Goff et al., demonstrates an association between coronary artery calcium (CAC) scores and medication use, not an association with decreased events. As the authors acknowledge in the guidelines (with the tepid IIB recommendation for CAC testing), there is limited evidence for a direct effect of CAC testing on hard outcomes.(2) For CAC testing to explain the ASCVD risk overestimation, the reporting of CAC results in MESA would have had to prevent almost half of all ASCVD events predicted by the AHA-ACC-ASCVD risk score.
2 and 3. The purpose of our sensitivity analysis was to evaluate whether eliminating the use of preventive therapies qualitatively changed the study’s main findings –it did not. Furthermore, available data does not support the correspondents claim that “The use of preventive therapies in the MESA cohort is extraordinarily high compared with the US population”. For example, NHANES (2001-2002) estimated that ~ 30% of the US population over 40 was treated for hypertension.(3) Using similar age-adjusted data, 31% of MESA participants were taking BP meds in 2000-2002. Cholesterol lowering medication use was reported in 20% of the US population over 40 in NHANES 2003-2004, and in 19% of age-adjusted MESA participants in 2002-2004. These percentages increased to an estimated 28% of US adults in 2011-2012 and 34% of subjects in our study over the same time period.(4)
4. In simple stratified analysis, the AHA-ACC-ASCVD score overestimated risk among all ethnicities studied in MESA: 103% for African Americans, 311% for Chinese, 61% for Hispanics and 49% for whites. A full multi-variable adjusted analysis of the factors most associated with risk overestimation in MESA is currently in peer review.
Any new therapy, diagnostic, or prognostic test should be compared to existing standards of care. When evaluated by us and others(5), the AHA-ACC-ASCVD risk calculator did not demonstrate improved calibration or discrimination compared to four other scores used in clinical practice. The concerns raised by Goff et al. also fail to explain why the Reynolds Risk Score was well calibrated.
We speculate that risk scores derived from cohorts assembled decades ago, either exclude pertinent factors or are poorly calibrated for modern populations. Diet, exercise, air pollution, ethnic diversity, intensity of tobacco use, content of tobacco products, and effectiveness of anti-hypertensive therapy are among cardiovascular risk factors that have changed over the last quarter century. Risk prediction is an evolving science and will require continual updating via the study of well-characterized, contemporary primary prevention cohorts.

Andrew P. DeFilippis, MD, MSc
University of Louisville, Louisville, Kentucky

Rebekah Young, PhD
University of Washington, Seattle, Washington

Michael J. Blaha, MD, MPH
Johns Hopkins Medicine Ciccarone Center for the Prevention of Heart Disease, Baltimore, Maryland

1. DeFilippis AP, Young R, Carrubba CJ, McEvoy JW, Budoff MJ, Blumenthal RS, et al. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann Intern Med. 2015;162(4):266-75.
2. Goff DC, Jr., Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation. 2014;129(25 Suppl 2):S49-73.
3. Ong KL, Cheung BM, Man YB, Lau CP, Lam KS. Prevalence, awareness, treatment, and control of hypertension among United States adults 1999-2004. Hypertension. 2007;49(1):69-75.
4. Qiuping G, Paulose-Ram R, Burt VL, and Kit BK. Prescription Cholesterol-lowering Medication Use in Adults Aged 40 and Over: United States, 2003–2012. NCHS Data Brief Number 177. Available at: http://www.cdc.gov/nchs/data/databriefs/db177.htm#ref10. 2014.
5. Kavousi M, Leening MJ, Nanchen D, Greenland P, Graham IM, Steyerberg EW, et al. Comparison of Application of the ACC/AHA Guidelines, Adult Treatment Panel III Guidelines, and European Society of Cardiology Guidelines for Cardiovascular Disease Prevention in a European Cohort. JAMA. 2014;311(14):1416-23.

Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)