J. Jane Jue, MD, MSc; Sarah Cunningham, MA; Kathleen Lohr, PhD; Paul Shekelle, MD, PhD; Richard Shiffman, MD, MCIS; Craig Robbins, MD, MPH; Mary Nix, MS; Vivian Coates, MBA; Karen Schoelles, MD, SM
Disclaimer: The views expressed here are solely those of the authors and do not necessarily reflect those of the AHRQ, the Department of Health and Human Services, or the federal government.
Acknowledgment: The authors thank the following persons for their contributions to this project: Eileen Erinoff, MSLIS; Linnea Hermanson, MA; Kariann Hudson, MEd; Amber Moran, MA; Connie Martin, BA; Edmond Baganizi, MBBS, MPH, MBA; Lisa Haskell, MSOT; Anne Wert, BS; Kelly Petersen, MA; Asim Mujahid, BS; Helen Dunn, BA; Jennifer Maslin; James Reston, PhD, MPH; Jonathan Treadwell, PhD; Kristen D'Anci, PhD (ECRI Institute); and Erin Vidic, MA (NCCN).
Financial Support: By AHRQ under contract 290-2013-00006C to ECRI Institute for NGC.
Disclosures: Dr. Jue reports funding for this work from AHRQ during the conduct of the study. Ms. Cunningham reports funding for this work from AHRQ during the conduct of the study. Dr. Lohr reports that she is retired but at the time of this work she received a small honorarium from ECRI Institute (the home institution of the main author) or used an RTI International stipend awarded to Distinguished Fellows (of which she is now “Emeritus”). Dr. Shekelle reports payments from ECRI Institute to chair their NGC editorial board during the conduct of the study. Dr. Robbins reports reimbursement to his employer from ECRI Institute during the conduct of the study and nonfinancial support from Guidelines International Network Board of Trustees outside the submitted work. In Kaiser Permanente (KP), he has served as the Medical Director of the National Guideline Program since January 2009. They have submitted some of their KP National Guidelines to NGC for publishing on its Web site. Had NGC continued, its use of the NEATS instrument would have affected whether to and which guidelines they would have submitted to NGC for consideration in the future. Ms. Coates reports that the work was performed as part of ECRI Institute's contract to create and maintain NGC. Dr. Schoelles reports funding for this work from AHRQ during the conduct of the study. Authors not named here have disclosed no conflicts of interest. Disclosures can also be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M18-2950.
Corresponding Author: J. Jane Jue, MD, MSc, Center for Clinical Evidence and Guidelines, ECRI Institute, 5200 Butler Pike, Plymouth Meeting, PA 19462; e-mail, firstname.lastname@example.org.
Current Author Addresses: Drs. Jue and Schoelles and Ms. Cunningham: Center for Clinical Evidence and Guidelines, ECRI Institute, 5200 Butler Pike, Plymouth Meeting, PA 19462.
Dr. Lohr: 30086 Britt, Chapel Hill, NC 27517.
Dr. Shekelle: West Los Angeles VA Medical Center, Room 111G, 11301 Wilshire Boulevard, Los Angeles, CA 90073.
Dr. Shiffman: Yale Center for Medical Informatics, 300 George Street, Suite 501, New Haven, CT 06510.
Dr. Robbins: 2500 South Havana Street, Aurora, CO 80014.
Ms. Nix: Agency for Healthcare Research and Quality, 5600 Fishers Lane, Room 06E14/Mailstop 06E53A, Rockville, MD 20857.
Ms. Coates: Silver Spring, MD.
Author Contributions: Conception and design: J.J. Jue, P. Shekelle, R. Shiffman, C. Robbins, M. Nix, V. Coates.
Analysis and interpretation of the data: J.J. Jue, S. Cunningham, K. Lohr.
Drafting of the article: J.J. Jue, K. Lohr, R. Shiffman, C. Robbins, K. Schoelles.
Critical revision of the article for important intellectual content: J.J. Jue, K. Lohr, P. Shekelle, R. Shiffman, C. Robbins, V. Coates, K. Schoelles.
Final approval of the article: J.J. Jue, S. Cunningham, K. Lohr, P. Shekelle, R. Shiffman, C. Robbins, M. Nix, V. Coates, K. Schoelles.
Statistical expertise: J.J. Jue.
Obtaining of funding: V. Coates.
Administrative, technical, or logistic support: J.J. Jue, K. Schoelles.
Collection and assembly of data: J.J. Jue, S. Cunningham.
In 2011, the Institute of Medicine (IOM) (now the National Academy of Medicine) published standards for trustworthy guidelines and recommended that the National Guideline Clearinghouse (NGC) of the Agency for Healthcare Research and Quality clearly indicate the extent to which guidelines adhere to these standards. To accomplish this, the authors developed and tested the NGC Extent of Adherence to Trustworthy Standards (NEATS) instrument. The standards were operationalized as an instrument containing 15 items that cover disclosure of the funding source; disclosure and management of conflicts of interest; multidisciplinary input; incorporation of patient perspectives; rigorous systematic review; recommendations accompanied by rationale, assessment of benefits and harms, clear linkage to the evidence, and assessment of strength of evidence and strength of recommendation; clear articulation of recommendations; external review by diverse stakeholders; and plans for updating. After multiple rounds of feedback from experts on clinical practice guideline development, the external validity and interrater reliability of the instrument were evaluated. For each item, 80% to 100% of survey respondents judged it to be a good measure of the IOM standards. All external stakeholders stated that NEATS was suitable for its intended goal. Interrater reliability for the final NEATS instrument had a weighted κ of 0.73. The NEATS instrument is a focused tool that provides a concise evaluation of a guideline's adherence to the IOM standards for trustworthy guidelines. It has good external validity among guideline developers and good interrater reliability across trained reviewers.
Jue JJ, Cunningham S, Lohr K, Shekelle P, Shiffman R, Robbins C, et al. Developing and Testing the Agency for Healthcare Research and Quality's National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument. Ann Intern Med. 2019;170:480–487. doi: 10.7326/M18-2950
Download citation file:
Published: Ann Intern Med. 2019;170(7):480-487.
Published at www.annals.org on 19 March 2019
Research and Reporting Methods.
Results provided by:
Copyright © 2019 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use