0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Research and Reporting Methods |

Empirical Evidence of the Importance of Comparative Studies of Diagnostic Test Accuracy

Yemisi Takwoingi, DVM; Mariska M.G. Leeflang, PhD; and Jonathan J. Deeks, PhD
[+] Article and Author Information

From University of Birmingham, Birmingham, United Kingdom, and Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands.

Acknowledgment: The authors thank Professor Chris Hyde, MBBS (University of Exeter), for providing clinical input; Mary Pennant, PhD (University of Birmingham), for assistance with data extraction; Sue Bayliss, BA (University of Birmingham), for assistance with searching; Georgina MacKenzie, MSc (Centre for Reviews and Dissemination, University of York), for providing the list of diagnostic reviews in DARE; and Richard Riley, PhD (University of Birmingham), for comments and suggestions on improving an earlier draft.

Grant Support: Dr. Takwoingi is funded through a United Kingdom National Institute for Health Research Award (DRF-2011-04-135). Dr. Leeflang is supported by the Netherlands Organization for Scientific Research (project 916.10.034). Dr. Deeks is partially supported by the Medical Research Council Midland Hub for Trials Methodology Research (grant number G0800808).

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M12-2051.

Requests for Single Reprints: Yemisi Takwoingi, DVM, Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham B15 2TT, United Kingdom; e-mail, y.takwoingi@bham.ac.uk.

Current Author Addresses: Dr. Takwoingi and Dr. Deeks: Public Health, Epidemiology and Biostatistics, School of Health and Population Sciences, University of Birmingham, Birmingham B15 2TT, United Kingdom.

Dr. Leeflang: Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Academic Medical Center, University of Amsterdam, Amsterdam, PO Box 22700, Amsterdam 1100, the Netherlands.

Author Contributions: Conception and design: Y. Takwoingi, M.M.G. Leeflang, J.J. Deeks.

Analysis and interpretation of the data: Y. Takwoingi, J.J. Deeks.

Drafting of the article: Y. Takwoingi, M.M.G. Leeflang, J.J. Deeks.

Critical revision of the article for important intellectual content: Y. Takwoingi, M.M.G. Leeflang, J.J. Deeks.

Final approval of the article: Y. Takwoingi, M.M.G. Leeflang, J.J. Deeks.

Provision of study materials or patients: Y. Takwoingi.

Statistical expertise: Y. Takwoingi, J.J. Deeks.

Obtaining of funding: Y. Takwoingi, M.M.G. Leeflang.

Administrative, technical, or logistic support: Y. Takwoingi.

Collection and assembly of data: Y. Takwoingi, M.M.G. Leeflang.


Ann Intern Med. 2013;158(7):544-554. doi:10.7326/0003-4819-158-7-201304020-00006
Text Size: A A A

Background: Systematic reviews that “compare” the accuracy of 2 or more tests often include different sets of studies for each test.

Purpose: To investigate the availability of direct comparative studies of test accuracy and to assess whether summary estimates of accuracy differ between meta-analyses of noncomparative and comparative studies.

Data Sources: Systematic reviews in any language from the Database of Abstracts of Reviews of Effects and the Cochrane Database of Systematic Reviews from 1994 to October 2012.

Study Selection: 1 of 2 assessors selected reviews that evaluated at least 2 tests and identified meta-analyses that included both noncomparative studies and comparative studies.

Data Extraction: 1 of 3 assessors extracted data about review and study characteristics and test performance.

Data Synthesis: 248 reviews compared test accuracy; of the 6915 studies, 2113 (31%) were comparative. Thirty-six reviews (with 52 meta-analyses) had adequate studies to compare results of noncomparative and comparative studies by using a hierarchical summary receiver-operating characteristic meta-regression model for each test comparison. In 10 meta-analyses, noncomparative studies ranked tests in the opposite order of comparative studies. A total of 25 meta-analyses showed more than a 2-fold discrepancy in the relative diagnostic odds ratio between noncomparative and comparative studies. Differences in accuracy estimates between noncomparative and comparative studies were greater than expected by chance (P < 0.001).

Limitation: A paucity of comparative studies limited exploration of direction in bias.

Conclusion: Evidence derived from noncomparative studies often differs from that derived from comparative studies. Robustly designed studies in which all patients receive all tests or are randomly assigned to receive one or other of the tests should be more routinely undertaken and are preferred for evidence to guide test selection.

Primary Funding Source: National Institute for Health Research (United Kingdom).

Figures

Grahic Jump Location
Appendix Figure 1.

Flowchart of review and meta-analysis selection.

DARE = Database of Abstracts of Reviews of Effects; rDOR = relative diagnostic odds ratio.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Ratio of rDORs (with 95% CIs).

Test comparisons are grouped according to meta-analytic findings—whether rDOR was greater than or less than 1 and ratio of rDOR was greater than or less than 1, or whether noncomparative studies ranked tests in opposite order of the comparative studies. The P value is the probability of a ratio of rDORs at least as extreme as the observed ratio, assuming no difference in comparative accuracy between noncomparative and comparative studies. The numbers in parentheses in the first column are the reference citations for the reviews from which the meta-analysis was obtained. AFP = α-fetoprotein; anti-CCP = anti–cyclic citrullinated peptide; βhCG = β human chorionic gonadotrophin; BNP = B-type natriuretic peptide; BS = bone scintigraphy; BTA = bladder tumor antigen; CA-125 = cancer antigen 125; CEA = carcinoembryonic antigen; CEUS = contrast-enhanced ultrasonography; CXR = chest radiography; CK = creatinine kinase; CK-MB = creatine kinase–MB; CT = computed tomography; CYFRA-21-1 = cytokeratin fragment 19; DOR = diagnostic odds ratio; DST = dexamethasone suppression test; DUS = Doppler ultrasonography; ECG = electrocardiography; echo = echocardiography; ELFA = enzyme-linked fluorescent assay; ELISA = enzyme-linked immunosorbent assay; EmA = IgA antiendomysial antibodies; EUS = endoscopic ultrasonography; FDG-PET = 18-fluorodeoxyglucose positron emission tomography; FISH = fluorescence in situ hybridization; FS = femur shortening; GDS15 = Geriatric Depression Scale (15-item questionnaire); GDS30 = Geriatric Depression Scale (30-item questionnaire); hCG = human chorionic gonadotrophin; HS = humerus shortening; IgA-tTG = IgA antitissue transglutaminase antibodies; LE = leukocyte esterase; MA = maternal age; MPI = myocardial perfusion imaging; MR = magnetic resonance; MRI = magnetic resonance imaging; NFT = nuchal fold translucency; NMP22 = nuclear matrix protein; NT-proBNP = N-terminal pro–B-type natriuretic peptide; PET = positron emission tomography; PET-CT = positron emission tomography–computed tomography; QCSC = quantitative catheter segment culture; RDT = rapid diagnostic test; rDOR = relative diagnostic odds ratio; RF = rheumatoid factor; RUT = rapid urease test; SE = stress echocardiography; SPECT = single photon emission computed tomography; SQCSC = semi-quantitative catheter segment culture; UBT = urea breath test; uE3 = unconjugated estriol; UFC = urinary free cortisol; US = ultrasonography; USS = duplex ultrasonography.

Grahic Jump Location
Grahic Jump Location
Figure 3.

One-sided contour-enhanced funnel plot of the ratio of rDORs.

Thirteen (25%) points are to the right of the 5% contour line, indicating statistically significant differences between estimates from comparative and noncomparative studies. This observed proportion was significantly higher than the 5% expected by chance (P < 0.001). rDOR = relative diagnostic odds ratio.

Grahic Jump Location
Grahic Jump Location
Figure 4.

Absolute differences in sensitivity and specificity (with 95% CIs).

The numbers in parentheses in the first column are the reference citations for the reviews from which the meta-analysis was obtained. BS = bone scintigraphy; BTA = bladder tumor antigen; CEUS = contrast-enhanced ultrasonography; CXR = chest radiography; CS = comparative study; CT = computed tomography; DUS = Doppler ultrasonography; ECG = electrocardiography; echo = echocardiography; EUS = endoscopic ultrasonography; FDG-PET = 18-fluorodeoxyglucose positron emission tomography; FS = femur shortening; HS = humerus shortening; LE = leukocyte esterase; MPI = myocardial perfusion imaging; MR = magnetic resonance; MRI = magnetic resonance imaging; NS = noncomparative study; NFT = nuchal fold translucency; PET = positron emission tomography; PET-CT = positron emission tomography–computed tomography; QCSC = quantitative catheter segment culture; RDT = rapid diagnostic test; RUT = rapid urease test; SE = stress echocardiography; SPECT = single photon emission computed tomography; SQCSC = semi-quantitative catheter segment culture; UBT = urea breath test; US = ultrasonography; USS = duplex ultrasonography.

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)