David M. Kent, MD, MS; Jessica K. Paulus, ScD; David van Klaveren, PhD; Ralph D'Agostino, PhD; Steve Goodman, MD, MHS, PhD; Rodney Hayward, MD; John P.A. Ioannidis, MD, DSc; Bray Patrick-Lake, MFS; Sally Morton, PhD; Michael Pencina, PhD; Gowri Raman, MBBS, MS; Joseph S. Ross, MD, MHS; Harry P. Selker, MD, MSPH; Ravi Varadhan, PhD; Andrew Vickers, PhD; John B. Wong, MD; Ewout W. Steyerberg, PhD
Disclaimer: The views, statements, and opinions presented in this work are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or its Methodology Committee.
Acknowledgment: The authors thank Mark Adkins, Teddy Balan, and Dan Sjoberg for excellent technical support in analyses included in the figures and supporting appendix tables. They also thank the Annals of Internal Medicine editors and reviewers, whose thoughtful feedback greatly improved this work. They thank Jennifer Lutz and Christine Lundquist for assistance with copyediting and creating exhibits.
Financial Support: Development of the PATH Statement was supported through contract SA.Tufts.PARC.OSCO.2018.01.25 from the PCORI Predictive Analytics Resource Center. This work was also informed by a 2018 conference (“Evidence and the Individual Patient: Understanding Heterogeneous Treatment Effects for Patient-Centered Care”) convened by the National Academy of Medicine and funded through a PCORI Eugene Washington Engagement Award (1900-TMC).
Disclosures: Dr. Kent reports grants from PCORI during the conduct of the study. Dr. Hayward reports grants from the National Institute of Diabetes and Digestive and Kidney Diseases and the Veterans Affairs Health Services Research and Development Service during the conduct of the study. Dr. Pencina reports grants from PCORI (Tufts Subaward) during the conduct of the study; grants from Sanofi/Regeneron, Amgen, and Bristol-Myers Squibb outside the submitted work; and personal fees from Boehringer Ingelheim and Merck outside the submitted work. Dr. Ross reports personal fees from PCORI during the conduct of the study and grants from the U.S. Food and Drug Administration, Medtronic, Johnson & Johnson, the Centers for Medicare & Medicaid Services, Blue Cross Blue Shield Association, the Agency for Healthcare Research and Quality, the National Institutes of Health (National Heart, Lung, and Blood Institute), and Laura and John Arnold Foundation outside the submitted work. Dr. Varadhan reports personal fees from Tufts University during the conduct of the study. Dr. Vickers reports grants from the National Institutes of Health during the conduct of the study. Dr. Wong reports grants from PCORI during the conduct of the study. Dr. Steyerberg reports royalties from Springer for his book Clinical Prediction Models. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-3667.
Corresponding Author: David M. Kent, MD, MS, Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, 800 Washington Street, Box 63, Boston, MA 02111; e-mail, email@example.com.
Current Author Addresses: Drs. Kent, Paulus, Raman, and Selker: Predictive Analytics and Comparative Effectiveness (PACE) Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, 800 Washington Street, Box 63, Boston, MA 02111.
Dr. van Klaveren: Erasmus University Medical Center, Doctor Molewaterplein 40, 3015 GD Rotterdam, the Netherlands.
Dr. D'Agostino: Boston University Mathematics and Statistics Department, 111 Cummington Street, Boston, MA 02215.
Dr. Goodman: Stanford University School of Medicine, 150 Governor's Lane, Room T265, Stanford, CA 94305.
Dr. Hayward: VA Ann Arbor Health Services Research and Development, 2800 Plymouth Road, Building 14, G100-36, Ann Arbor, MI 48109.
Dr. Ioannidis: Stanford Prevention Research Center, 1265 Welch Road, Stanford, CA 94305.
Ms. Patrick-Lake: Evidation Health, 167 2nd Avenue, San Mateo, CA 94401.
Dr. Morton: Virginia Tech, North End Center Suite 4300, 300 Turner Street NW, Blacksburg, VA 24061.
Dr. Pencina: Duke Clinical Research Institute, 200 Trent Street, Durham, NC 27710.
Dr. Ross: Yale University School of Medicine, PO Box 208093, New Haven, CT 06520.
Dr. Varadhan: Johns Hopkins University, Division of Biostatistics and Bioinformatics, 550 North Broadway, Suite 1103-A, Baltimore, MD 21205.
Dr. Vickers: Memorial Sloan Kettering Cancer Center, 485 Lexington Avenue, 2nd Floor, New York, NY 10017.
Dr. Wong: Tufts Medical Center, 800 Washington Street #302, Boston, MA 02111.
Dr. Steyerberg: Erasmus University Medical Center, PO Box 2040, 3055 PC Rotterdam, the Netherlands.
Author Contributions: Conception and design: D.M. Kent, J.K. Paulus, R. Hayward, J.P.A. Ioannidis, J.S. Ross, A. Vickers, J.B. Wong, E.W. Steyerberg.
Analysis and interpretation of the data: D.M. Kent, J.K. Paulus, R. D'Agostino, R. Hayward, J.P.A. Ioannidis, J.B. Wong, E.W. Steyerberg.
Drafting of the article: D.M. Kent, J.K. Paulus, R. D'Agostino, A. Vickers, J.B. Wong.
Critical revision of the article for important intellectual content: D.M. Kent, J.K. Paulus, D. van Klaveren, R. D'Agostino, R. Hayward, J.P.A. Ioannidis, S. Morton, M. Pencina, G. Raman, J.S. Ross, R. Varadhan, A. Vickers, J.B. Wong, E.W. Steyerberg.
Final approval of the article: D.M. Kent, J.K. Paulus, D. van Klaveren, R. D'Agostino, S. Goodman, R. Hayward, J.P.A. Ioannidis, B. Patrick-Lake, S. Morton, M. Pencina, G. Raman, J.S. Ross, H.P. Selker, R. Varadhan, A. Vickers, J.B. Wong, E.W. Steyerberg.
Provision of study materials or patients: D.M. Kent, J.B. Wong.
Statistical expertise: D.M. Kent, D. van Klaveren, R. D'Agostino, R. Hayward, J.P.A. Ioannidis, S. Morton, R. Varadhan, A. Vickers, J.B. Wong, E.W. Steyerberg.
Obtaining of funding: D.M. Kent, J.K. Paulus, J.B. Wong.
Administrative, technical, or logistic support: D.M. Kent, J.K. Paulus, G. Raman, H.P. Selker, J.B. Wong.
Collection and assembly of data: D.M. Kent, J.K. Paulus, G. Raman, J.B. Wong.
Heterogeneity of treatment effect (HTE) refers to the nonrandom variation in the magnitude or direction of a treatment effect across levels of a covariate, as measured on a selected scale, against a clinical outcome. In randomized controlled trials (RCTs), HTE is typically examined through a subgroup analysis that contrasts effects in groups of patients defined “1 variable at a time” (for example, male vs. female or old vs. young). The authors of this statement present guidance on an alternative approach to HTE analysis, “predictive HTE analysis.” The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risks with versus without the intervention, taking into account all relevant patient attributes simultaneously. The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed using a multidisciplinary technical expert panel, targeted literature reviews, simulations to characterize potential problems with predictive approaches, and a deliberative process engaging the expert panel. The authors distinguish 2 categories of predictive HTE approaches: a “risk-modeling” approach, wherein a multivariable model predicts the risk for an outcome and is applied to disaggregate patients within RCTs to define risk-based variation in benefit, and an “effect-modeling” approach, wherein a model is developed on RCT data by incorporating a term for treatment assignment and interactions between treatment and baseline covariates. Both approaches can be used to predict differential absolute treatment effects, the most relevant scale for clinical decision making. The authors developed 4 sets of guidance: criteria to determine when risk-modeling approaches are likely to identify clinically important HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. The PATH Statement, together with its explanation and elaboration document, may guide future analyses and reporting of RCTs.
Table 1. Mathematical Dependence of Treatment Effect on CER
Schematized (left) and actual (right) risk-based heterogeneous treatment effects.
Q1 = first risk quarter (lowest); Q2 = second risk quarter; Q3 = third risk quarter; Q4 = fourth risk quarter (highest). A. Schematic results in a trial intervention that lowers the odds of an outcome by 25% (odds ratio, 0.75) but has an absolute treatment-related harm of 1%. Outcome risks (top), observed odds ratios (middle), and risk differences (bottom) are shown. Overall trial results are dependent on the average risk for the enrolled trial population. When the average risk is about 7% (as in this example), a well-powered study would detect a positive overall treatment benefit (shown by the horizontal dashed line in the middle and bottom panels). However, a prediction model with a c-statistic of 0.75 generates the risk distribution in the top panel of the figure. A treatment-by-risk interaction emerges (middle). Regardless of whether this interaction is statistically significant, examination of treatment effects on the absolute risk difference scale (bottom) shows harm in the low-risk group and very substantial benefit in the high-risk group, both of which are obscured by the overall summary results. Conventional “1-variable-at-a-time” subgroup analyses are typically inadequate to disaggregate patients into groups that are sufficiently heterogeneous for risk, so benefit–harm tradeoffs can misleadingly seem to be consistent across the trial population. Although this figure shows idealized relationships between risk and treatment effects, these relationships will be sensitive to how risk is described (i.e., what variables are in the risk model). Baseline risk has a logit-normal distribution, with µ = −3 and σ = 1 (the log odds are normally distributed). Adapted from reference 3. Q1 = first risk quarter (lowest); Q2 = second risk quarter; Q3 = third risk quarter; Q4 = fourth risk quarter (highest). B. Stratified results of RITA-3 (Randomized Intervention Trial of unstable Angina 3) (33). The RITA-3 trial (n = 1810) tested early intervention vs. conservative management of non–ST-segment elevation acute coronary syndrome. Results for the outcome of death or nonfatal myocardial infarction at 5 y are shown, stratified into equal-sized risk quarters using an internally derived risk model; the highest-risk quarter is substratified into halves (groups 4a and 4b). Event rates with 95% CIs (top), odds ratios (middle), and risk differences (bottom) are shown. The risk model comprises the following easily obtainable clinical characteristics: age, sex, diabetes, prior myocardial infarction, smoking status, heart rate, ST-segment depression, angina severity, left bundle branch block, and treatment strategy. As in the schematic diagram to the left, the average treatment effect seen in the summary results (horizontal dashed line in middle and bottom panels) closely reflects the effect in patients in risk group 3, whereas half of patients (risk groups 1 and 2) receive no treatment benefit from early intervention. Absolute benefit (bottom) in the primary outcome was very pronounced in the eighth of patients at highest risk (risk group 4b). A statistically significant risk-by-treatment interaction can be seen when results are expressed in the odds ratio scale (middle) (the interaction P value is from a likelihood ratio test for adding an interaction between the linear predictor of risk and treatment assignment). Such a pattern can emerge if early intervention is associated with some procedure-related risks that are evenly distributed over all risk groups, eroding benefit in low-risk but not high-risk patients, as illustrated schematically in the left panel.
Table 2. Equations Corresponding to Risk-Modeling and Effect-Modeling Approaches
Appendix Table 1. PATH Technical Expert Panel
Appendix Table 2. TEP Votes on PATH Statement Revisions Held 1 May 2019 and 17 July 2019 (7 Voters)
Appendix Table 3. Results of the Final TEP Vote on Criteria to Identify When a Risk-Modeling Approach to RCT Analysis Is Likely to Be of Most Value (11 Voters)*
Appendix Table 4. Results of Final Consensus Vote on Risk Modeling Guidance to Identify HTE (13 Voters)*
Appendix Table 5. Results of Final TEP Vote on Caveats of Treatment Effect–Modeling Approaches to Identify HTE (9 voters)*
Consensus criteria: when is a risk-modeling approach to RCT analysis likely to be most valuable?
RCT = randomized controlled trial.
Consensus guidance on risk-modeling approaches to identify HTE.
HTE = heterogeneous treatment effects; RCT = randomized controlled trial.
Consensus statements on caveats and considerations before moving to clinical practice.
HTE = heterogeneous treatment effects.
Consensus statements on considerations and caveats in effect modeling for HTE.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Stuart G. Baker
National Cancer Institute
November 22, 2019
Treatment effect Heterogeneity and Benefit Functions
In discussing effect modeling for heterogeneity of treatment effect, Kent et al. (1) restricted their prediction models to include multiplicative treatment-by-covariate interaction terms. An informative alternative is the subpopulation treatment effect pattern plot (STEPP) using a benefit function involving multiple predictive markers (2). To implement this STEPP approach, investigators split the data in a randomized trial into training and test samples. Using the training sample, investigators fit a benefit function based on the predictive markers. Two possible benefit functions are risk difference, for standard treatment trials, and responders-only, for prevention trials with a rare outcome. The risk difference benefit function is the estimated difference, between randomization groups, in the probabilities of a favorable outcome as a function of the predictive markers. The responders-only benefit function is the estimated probability of assignment to the new treatment given a favorable outcome and the predictive markers. For each participant in the test sample, investigators compute a benefit score from the benefit function. The multivariate STEPP plots, for the test sample, the estimated difference in the probability of favorable outcome between randomization groups among participants with a benefit score greater than a cutpoint versus the cutpoint. The lower bound of the 95% confidence intervals for STEPP can identify a promising subgroup to receive treatment (if the subgroup treatment effect is sufficiently large), even if the randomized trial shows no overall beneficial effect of the receipt of new versus old treatment. References1. Kent DM, Paulus JK, van Klaveren D, D'Agostino R, Goodman S, Hayward R, et al. The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement. Ann Intern Med. 2019 Nov 12. doi: 10.7326/M18-3667. [Epub ahead of print]2. Baker SG and Bonetti M. Evaluating markers for guiding treatment. Journal of the National Cancer Institute 2016; 108:djw101
Kent DM, Paulus JK, van Klaveren D, et al. The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement. Ann Intern Med. 2019;:. [Epub ahead of print 12 November 2019]. doi: https://doi.org/10.7326/M18-3667
Download citation file:
Published: Ann Intern Med. 2019.
Research and Reporting Methods.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use