Robert F. Wolff, MD *; Karel G.M. Moons, PhD *; Richard D. Riley, PhD; Penny F. Whiting, PhD; Marie Westwood, PhD; Gary S. Collins, PhD; Johannes B. Reitsma, MD, PhD; Jos Kleijnen, MD, PhD; Sue Mallett, DPhil; for the PROBAST Group†
Disclaimer: This report presents independent research supported by the National Institute for Health Research (NIHR). The views and opinions expressed in this publication are those of the authors and do not necessarily reflect those of the National Health Service (NHS), the NIHR, or the Department of Health and Social Care.
Acknowledgment: The authors thank the members of the Delphi panel (Appendix) for their valuable input and all testers, especially Cordula Braun, Johanna A.A.G. Damen, Paul Glasziou, Pauline Heus, Lotty Hooft, and Romin Pajouheshnia, for providing feedback on PROBAST. They also thank Janine Ross and Steven Duffy for support in managing the references.
Financial Support: Drs. Moons and Reitsma received financial support from the Netherlands Organisation for Scientific Research (ZONMW 918.10.615 and 91208004). Dr. Riley is a member of the Evidence Synthesis Working Group funded by the NIHR School for Primary Care Research (project 390). Dr. Whiting (time) was supported by the NIHR Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust. Dr. Collins was supported by the NIHR Biomedical Research Centre, Oxford. Dr. Mallett is supported by NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Disclosures: Dr. Wolff reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Westwood reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Kleijnen reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-1376.
Corresponding Author: Robert F. Wolff, MD, Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, United Kingdom; e-mail, firstname.lastname@example.org.
Current Author Addresses: Drs. Wolff, Westwood, and Kleijnen: Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, United Kingdom.
Drs. Moons and Reitsma: Julius Centre for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, PO Box 85500, 3508 GA Utrecht, the Netherlands.
Dr. Riley: Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire ST5 5BG, United Kingdom.
Dr. Whiting: NIHR CLAHRC West, University Hospitals Bristol NHS Foundation Trust and School of Social and Community Medicine, University of Bristol, Bristol BS1 2NT, United Kingdom.
Dr. Collins: Centre for Statistics in Medicine, NDORMS, University of Oxford, Botnar Research Centre, Windmill Road, Oxford OX3 7LD, United Kingdom.
Dr. Mallett: Institute of Applied Health Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, United Kingdom.
Author Contributions: Conception and design: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Analysis and interpretation of the data: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Drafting of the article: R.F. Wolff, K.G.M. Moons, P.F. Whiting, M. Westwood, S. Mallett.
Critical revision of the article for important intellectual content: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Final approval of the article: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Statistical expertise: K.G.M. Moons, R.D. Riley, G.S. Collins, J.B. Reitsma, S. Mallett.
Obtaining of funding: K.G.M. Moons, R.D. Riley, P.F. Whiting, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Administrative, technical, or logistic support: R.F. Wolff, K.G.M. Moons, J. Kleijnen, S. Mallett.
Collection and assembly of data: R.F. Wolff, K.G.M. Moons, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models).
PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting.
PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions.
Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.
Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST.
Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (7, 26). PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Differences between diagnostic and prognostic prediction model studies.
PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Table 1. Four Steps in PROBAST
Table 2. PROBAST: Summary of Step 3—Assessment of Risk of Bias and Concerns Regarding Applicability*
Table 3. Suggested Tabular Presentation for PROBAST Results*
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Silvan Licher; MD – email@example.com, Pinar Yilmaz; MD – firstname.lastname@example.org, M. Kamran Ikram; MD;PhD – email@example.com, M. Arfan Ikram;MD;PhD – firstname.lastname@example.org, Maarten J.G. Leening;MD;PhD – email@example.com
Departments of Epidemiology, Radiology & Nuclear Medicine, Neurology and Cardiology, Erasmus MC – University Medical Center Rotterdam, Rotterdam, the Netherlands
January 30, 2019
Conflict of Interest:
We declare no potential financial conflicts of interest. The literature search that was done for this work also identified one model that that has been developed and validated by the authors themselves.
Feasibility of using PROBAST to assess bias and applicability of dementia prediction models
With the recent publication of the Prediction model Risk Of Bias ASsessment Tool (PROBAST) in this journal, a framework for critical and systematic evaluation of prediction models has been established (1). We aimed to assess the feasibility of PROBAST and whether its results can facilitate model selection for clinical practice. We used dementia prediction models as an illustration. Numerous prediction models for dementia have been developed (2), yet none of these has been recognized as an established model to facilitate targeted preventive efforts or select high risk individuals for inclusion in clinical trials. Several systematic reviews on dementia risk prediction models have been published, but the internal validity (i.e. bias), and applicability of these models have not been systematically evaluated (2-4).Following PROBAST, we selected dementia prediction models that have been developed and externally validated to identify individuals at high risk in the general population in order to facilitate targeted preventive efforts (Steps 1 and 2). We systematically examined the risk of bias and concerns regarding the applicability for each of these models (Step 3). Based on a previously published literature search on dementia models (updated until Jan 1st, 2019) (4, 5), we identified five validated models (Cardiovascular Risk Factors, Aging, and Incidence of Dementia; ANU-Alzheimer's Disease Risk Index; Brief Dementia Screening Indicator; Dementia Risk Score; and The Rotterdam Study model). PROBAST identified high risk of bias in three of these models that could compromise their internal validity but raised low concern regarding applicability (Step 4). For one model, we identified both high risk of bias and high concerns regarding applicability. A single model met all criteria, indicating that there were low concerns regarding risk of bias and applicability. Of the four domains in PROBAST (i.e. participants, predictors, outcome, and analysis), a high risk of bias in the analysis domain was common, being present in four of the five models. Detailed PROBAST assessments for each of these models are available upon request.We conclude that PROBAST facilitates systematic examination and summarizing of the quality and applicability of prediction models, and thereby facilitates selection of prediction models for clinical use. Importantly, our application of PROBAST revealed methodological shortcomings for the majority of dementia prediction models, which may compromise reliable risk estimation in clinical practice. These findings highlight that rigorous external validation of prediction models is not sufficient to guarantee internal validity. References1. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51-8.2. Tang EY, Harrison SL, Errington L, Gordon MF, Visser PJ, Novak G, et al. Current developments in dementia risk prediction modelling: an updated systematic review. Plos One. 2015;10(9):e0136181.3. Hou XH, Feng L, Zhang C, Cao XP, Tan L, Yu JT. Models for predicting risk of dementia: a systematic review. J Neurol Neurosurg Psychiatry. 2018. Epub Ahead of Print.4. Licher S, Yilmaz P, Leening MJG, Wolters FJ, Vernooij MW, Stephan BCM, et al. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study. European Journal of Epidemiology. 2018;33(7):645-55.5. Licher S, Leening MJG, Yilmaz P, Wolters FJ, Heeringa J, Bindels PJE, et al. Development and validation of a dementia risk prediction model in the general population: an analysis of three longitudinal studies. Am J Psychiatry. 2018. Epub Ahead of Print.
Robert F. Wolff, Karel G.M. Moons, Sue Mallett
Kleijnen Systematic Reviews, York, UK (RFW); Julius Center for Health Sciences and Primary Care, Utrecht University (KGMM); Institute of Applied Health Research, University of Birmingham, UK (SM)
February 1, 2019
Conflict of Interest:
See disclosure information in the PROBAST article
Response to: Feasibility of using PROBAST to assess bias and applicability of dementia prediction models
We would like to congratulate Silvan Licher and his colleagues on their work on dementia prediction models(1,2) and would like to thank them for their positive feedback on the use of PROBAST, a tool to assess the risk of bias and applicability of prediction model studies.In fact, we are pleasantly surprised to see that PROBAST, which has only been published recently, is already been used.(3,4) Furthermore, we are pleased to see that the comment highlighted the usefulness of two key elements of PROBAST:1. Four domains (participants, predictors, outcome, and analysis) are rated as part of a PROBAST assessment, allowing users to pinpoint shortcomings in the methodology of the underlying study reporting the development and/or validation of a prediction model2. PROBAST allows the assessment of the risk of bias and applicability of a study. The comment illustrated that PROBAST appears to have been a useful tool in assessing the five models investigated by Licher et al., i.e. identifying a prediction model study which was rated as low risk of bias and low concerns regarding applicability.On www.probast.org, we will list this study as an example of the use of PROBAST. The website also presents other details relevant to prediction research, e.g. information on workshops, details on relevant research initiatives as well as the current version of the PROBAST tool.References:1. Licher S, Yilmaz P, Leening MJG, Wolters FJ, Vernooij MW, Stephan BCM, et al. External validation of four dementia prediction models for use in the general community-dwelling population: a comparative analysis from the Rotterdam Study. European Journal of Epidemiology. 2018;33(7):645-55.2. Licher S, Leening MJG, Yilmaz P, Wolters FJ, Heeringa J, Bindels PJE, et al. Development and validation of a dementia risk prediction model in the general population: an analysis of three longitudinal studies. Am J Psychiatry. 2018. Epub Ahead of Print.3. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S, PROBAST Group. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med 2019;170(1):51-58. Freely available from: http://annals.org/aim/fullarticle/2719961/probast-tool-assess-risk-bias-applicability-prediction-model-studies4. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med 2019;170(1):W1-W33. Freely available from: http://annals.org/aim/fullarticle/2719962/probast-tool-assess-risk-bias-applicability-prediction-model-studies-explanation
Wolff RF, Moons KG, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med. 2019;170:51–58. doi: 10.7326/M18-1376
Download citation file:
Published: Ann Intern Med. 2019;170(1):51-58.
Research and Reporting Methods.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use