David M. Phillippo, MSc, BSc; Sofia Dias, PhD; Nicky J. Welton, PhD; Deborah M. Caldwell, PhD; Nichole Taske, PhD; A.E. Ades, PhD
Disclaimer: Any views expressed in this work are those of the authors and not the funders.
Acknowledgment: The authors thank Dr. Kathryn Hopkins, Technical Adviser at NICE Centre for Guidelines (CfG), for providing the risk-of-bias assessments for the headaches example; members of the U.K. GRADE Network Steering Group and the NICE NMA Working Group; and all NICE developers who commented on earlier versions of this work.
Financial Support: This work was partly funded by the NICE CfG through the NICE Guidelines Technical Support Unit, University of Bristol. Prof. Dias received partial funding from the U.K. Medical Research Council (MRC) (grant MR/M005232/1). Prof. Welton received partial funding from the MRC ConDuCT-II Hub for Trials Methodology Research (grant MR/K025643/1).
Disclosures: Mr. Phillippo and Prof. Ades report grants from NICE CfG during the conduct of the study. Prof. Dias reports grants from NICE CfG and MRC during the conduct of the study. Prof. Welton is principal investigator on a methodology grant jointly funded by the MRC and Pfizer (Pfizer partly funds a junior researcher) for a project that is purely methodological, uses historical data on pain relief, and is unrelated to this work. Dr. Caldwell is Deputy Director of the NICE Guidelines Technical Support Unit. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-3542.
Corresponding Author: David M. Phillippo, MSc, BSc, Bristol Medical School (Population Health Sciences), University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, United Kingdom; e-mail, email@example.com.
Current Author Addresses: Mr. Phillippo, Profs. Welton and Ades, and Dr. Caldwell: Bristol Medical School (Population Health Sciences), University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, United Kingdom.
Prof. Dias: Centre for Reviews and Dissemination, University of York, Heslington, York, YO10 5DD, United Kingdom.
Dr. Taske: National Institute for Health and Care Excellence, 10 Spring Gardens, London, SW1A 2BU, United Kingdom.
Author Contributions: Conception and design: D.M. Phillippo, S. Dias, N.J. Welton, D.M. Caldwell, N. Taske, A.E. Ades.
Analysis and interpretation of the data: D.M. Phillippo.
Drafting of the article: D.M. Phillippo.
Critical revision for important intellectual content: D.M. Phillippo, S. Dias, N.J. Welton, D.M. Caldwell, N. Taske, A.E. Ades.
Final approval of the article: D.M. Phillippo, S. Dias, N.J. Welton, D.M. Caldwell, N. Taske, A.E. Ades.
Statistical expertise: D.M. Phillippo, S. Dias, N.J. Welton, A.E. Ades.
Obtaining of funding: S. Dias.
Guideline development requires the synthesis of evidence on several treatments of interest, typically by using network meta-analysis (NMA). Because treatment effects may be estimated imprecisely or be based on evidence lacking internal or external validity, guideline developers must assess the robustness of recommendations made on the basis of the NMA to potential limitations in the evidence. Such limitations arise because the observed estimates differ from the true effects of interest, for example, because of study biases, sampling variation, or issues of relevance. The widely used GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework aims to assess the quality of evidence supporting a recommendation by using a structured series of qualitative judgments. This article argues that GRADE approaches proposed for NMA are insufficient for the purposes of guideline development, because the influence of the evidence on the final recommendation is not taken into account. It outlines threshold analysis as an alternative approach, demonstrating the method with 2 examples of clinical guidelines from the National Institute for Health and Care Excellence (NICE) in the United Kingdom. Threshold analysis quantifies precisely how much the evidence could change (for any reason, such as potential biases, or simply sampling variation) before the recommendation changes, and what the revised recommendation would be. If it is judged that the evidence could not plausibly change by more than this amount, then the recommendation is considered robust; otherwise, it is sensitive to plausible changes in the evidence. In this manner, threshold analysis directly informs decision makers and guideline developers of the robustness of treatment recommendations.
Phillippo DM, Dias S, Welton NJ, Caldwell DM, Taske N, Ades A. Threshold Analysis as an Alternative to GRADE for Assessing Confidence in Guideline Recommendations Based on Network Meta-analyses. Ann Intern Med. [Epub ahead of print 26 March 2019]170:538–546. doi: 10.7326/M18-3542
Download citation file:
Published: Ann Intern Med. 2019;170(8):538-546.
Published at www.annals.org on 26 March 2019
Research and Reporting Methods.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use