Karel G.M. Moons, PhD *; Robert F. Wolff, MD *; Richard D. Riley, PhD; Penny F. Whiting, PhD; Marie Westwood, PhD; Gary S. Collins, PhD; Johannes B. Reitsma, MD, PhD; Jos Kleijnen, MD, PhD; Sue Mallett, DPhil
Disclaimer: This report presents independent research supported by the National Institute for Health Research (NIHR). The views and opinions expressed in this publication are those of the authors and do not necessarily reflect those of the National Health Service (NHS), the NIHR, or the Department of Health and Social Care.
Acknowledgment: The authors thank the members of the PROBAST Delphi panel (38) for their valuable input and all testers, especially Cordula Braun, Johanna A.A.G. Damen, Paul Glasziou, Pauline Heus, Lotty Hooft, and Romin Pajouheshnia, for providing feedback on PROBAST. They also thank Janine Ross and Steven Duffy for support in managing the references.
Financial Support: Drs. Moons and Reitsma received financial support from the Netherlands Organisation for Scientific Research (ZONMW 918.10.615 and 91208004). Dr. Riley is a member of the Evidence Synthesis Working Group funded by the NIHR School for Primary Care Research (project 390). Dr. Whiting (time) was supported by the NIHR Collaboration for Leadership in Applied Health Research and Care West at University Hospitals Bristol NHS Foundation Trust. Dr. Collins was supported by the NIHR Biomedical Research Centre, Oxford. Dr. Mallett is supported by NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Disclosures: Dr. Wolff reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Westwood reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Dr. Kleijnen reports grants from Bayer, Biogen, Pfizer, UCB, Amgen, BioMarin, Grünenthal, Chiesi, and TESARO outside the submitted work. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-1376.
Corresponding Author: Karel G.M. Moons, PhD, Julius Centre for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, PO Box 85500, 3508 GA Utrecht, the Netherlands; e-mail, K.G.M.Moons@umcutrecht.nl.
Current Author Addresses: Drs. Moons and Reitsma: Julius Centre for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, PO Box 85500, 3508 GA Utrecht, the Netherlands.
Drs. Wolff, Westwood, and Kleijnen: Kleijnen Systematic Reviews Ltd, Unit 6, Escrick Business Park, Riccall Road, Escrick, York YO19 6FD, United Kingdom.
Dr. Riley: Centre for Prognosis Research, Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire ST5 5BG, United Kingdom.
Dr. Whiting: NIHR CLAHRC West, University Hospitals Bristol NHS Foundation Trust and School of Social and Community Medicine, University of Bristol, Bristol BS1 2NT, United Kingdom.
Dr. Collins: Centre for Statistics in Medicine, NDORMS, University of Oxford, Botnar Research Centre, Windmill Road, Oxford OX3 7LD, United Kingdom.
Dr. Mallett: Institute of Applied Health Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, United Kingdom.
Author Contributions: Conception and design: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Analysis and interpretation of the data: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Drafting of the article: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, S. Mallett.
Critical revision of the article for important intellectual content: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Final approval of the article: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Statistical expertise: K.G.M. Moons, R.D. Riley, G.S. Collins, J.B. Reitsma, S. Mallett.
Obtaining of funding: K.G.M. Moons, R.D. Riley, P.F. Whiting, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Administrative, technical, or logistic support: K.G.M. Moons, R.F. Wolff, J. Kleijnen, S. Mallett.
Collection and assembly of data: K.G.M. Moons, R.F. Wolff, R.D. Riley, P.F. Whiting, M. Westwood, G.S. Collins, J.B. Reitsma, J. Kleijnen, S. Mallett.
Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model).
Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed.
A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic.
PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.
Table 1. Guidance on Conducting Systematic Reviews of Prediction Model Studies
Types of diagnostic and prognostic modeling studies or reports addressed by PROBAST.
Adopted from the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) and CHARMS (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) guidance (8, 16).
Differences between diagnostic and prognostic prediction model studies.
PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Examples of systematic review questions for which PROBAST is suitable.
There are various different questions that systematic reviews of prediction models may address. The following are examples of different types of reviews in which PROBAST can be applied. EuroSCORE = European System for Cardiac Operative Risk Evaluation; PROBAST = Prediction model Risk Of Bias ASsessment Tool.
Table 2. PICOTS*
Prediction model performance measures.
Table 3. Four Steps in PROBAST
Table 4. Example Papers
Table 5. Example Step 1 Applied to the Perel Example Study*
Table 6. Example Step 2 Applied to the Perel Example Study*
Table 7. Domain 1: Participants—Guidance Notes for Rating Risk of Bias and Applicability
Table 8. Domain 2: Predictors—Guidance Notes for Rating Risk of Bias and Applicability
Table 9. Domain 3: Outcome—Guidance Notes for Rating Risk of Bias and Applicability
Table 10. Domain 4: Analysis—Guidance Notes for Rating Risk of Bias
Table 11. Overall Assessment of Risk of Bias and Concerns for Applicability
Table 12. Suggested Tabular Presentation for PROBAST Results*
Suggested graphical presentation for PROBAST results.
PROBAST = Prediction model Risk Of Bias ASsessment Tool.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Moons KG, Wolff RF, Riley RD, et al. PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Ann Intern Med. 2019;170:W1–W33. doi: https://doi.org/10.7326/M18-1377
Download citation file:
Published: Ann Intern Med. 2019;170(1):W1-W33.
Research and Reporting Methods.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use