The full content of Annals is available to subscribers

Subscribe/Learn More  >
Research and Reporting Methods |

Evaluating Diagnostic Accuracy in the Face of Multiple Reference Standards

Christiana A. Naaktgeboren, MPH; Joris A.H. de Groot, PhD; Maarten van Smeden, MSc; Karel G.M. Moons, PhD; and Johannes B. Reitsma, MD, PhD
[+] Article, Author, and Disclosure Information

From University Medical Center Utrecht, Utrecht, the Netherlands.

Financial Support: By the Netherlands Organization for Scientific Research (project 918.10.615).

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-0322.

Requests for Single Reprints: Christiana A. Naaktgeboren, MPH, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, PO Box 85500, 3508 GA Utrecht, the Netherlands.

Current Author Addresses: Ms. Naaktgeboren; Drs. de Groot, Moons, and Reitsma; and Mr. van Smeden: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, PO Box 85500, 3508 GA Utrecht, the Netherlands.

Author Contributions: Conception and design: C.A. Naaktgeboren, J.A.H. de Groot, M. van Smeden, K.G.M. Moons, J.B. Reitsma.

Analysis and interpretation of the data: K.G.M. Moons, J.B. Reitsma.

Drafting of the article: C.A. Naaktgeboren, J.A.H. de Groot, K.G.M. Moons.

Critical revision of the article for important intellectual content: C.A. Naaktgeboren, J.A.H. de Groot, M. van Smeden, K.G.M. Moons, J.B. Reitsma.

Final approval of the article: J.A.H. de Groot, M. van Smeden, K.G.M. Moons, J.B. Reitsma.

Statistical expertise: J.A.H. de Groot, M. van Smeden, K.G.M. Moons, J.B. Reitsma.

Obtaining of funding: K.G.M. Moons.

Administrative, technical, or logistic support: J.A.H. de Groot.

Collection and assembly of data: K.G.M. Moons.

Ann Intern Med. 2013;159(3):195-202. doi:10.7326/0003-4819-159-3-201308060-00009
Text Size: A A A

A universal challenge in studies that quantify the accuracy of diagnostic tests is establishing whether each participant has the disease of interest. Ideally, the same preferred reference standard would be used for all participants; however, for practical or ethical reasons, alternative reference standards that are often less accurate are frequently used instead. The use of different reference standards across participants in a single study is known as differential verification.

Differential verification can cause severely biased accuracy estimates of the test or model being studied. Many variations of differential verification exist, but not all introduce the same risk of bias. A risk-of-bias assessment requires detailed information about which participants receive which reference standards and an estimate of the accuracy of the alternative reference standard. This article classifies types of differential verification and explores how they can lead to bias. It also provides guidance on how to report results and assess the risk of bias when differential verification occurs and highlights potential ways to correct for the bias.


Grahic Jump Location
Figure 1.

An example of bias due to differential verification.

The example is loosely inspired by a study on the accuracy of VIA in screening for cervical cancer (15). The preferred reference standard is colposcopy plus biopsy when a lesion is detected. Because the preferred standard is invasive, one might use an alternative, less invasive reference standard, the Pap smear, for participants with a normal VIA result. If one assumed that the Pap smear had perfect accuracy, the naive (biased) estimates of sensitivity and specificity for the VIA would be 0.68 and 0.81, respectively. If one recognized the sensitivity of the Pap smear as only 0.70, the true estimate of the sensitivity for the VIA would be 0.60. Pap = Papanicolaou; VIA = visual inspection using acetic acid.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Main verification patterns and terminology.

Grahic Jump Location
Grahic Jump Location
Figure 3.

Classification of differential verification patterns.

In pattern A, the test a participant receives depends solely on the index test result, whereas in pattern B, other variables may influence reference standard assignment.

Grahic Jump Location
Grahic Jump Location
Figure 4.

An example of how to report complete index test–dependent differential verification.

The example is inspired by AMUSE-1 (Amsterdam Maastricht Utrecht Study on Thromboembolism) (7), which looked at the safety of a diagnostic rule (the Wells rule) for excluding DVT. When the rule predicted a high risk for DVT, the patients were referred for further testing (ultrasonography). Those with a low predicted risk were sent home and followed instead. In this example, the probability that a participant truly has DVT given that they had a diagnostic score ≥4 (that is, the PPV) is 25%. Likewise, the probability that a person who was sent home because they had a diagnostic score ≤3 would soon thereafter be diagnosed with DVT (and was therefore erroneously sent home) was 1%. DVT = deep venous thrombosis; NPV = negative predictive value; PPV = positive predictive value.

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.