0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Research and Reporting Methods |

Uncertainty in Treatment Rankings: Reanalysis of Network Meta-analyses of Randomized TrialsUncertainty in Treatment Rankings From Network Meta-analyses

Ludovic Trinquart, PhD; Nassima Attiche, MSc; Aïida Bafeta, PhD; Raphaël Porcher, PhD; and Philippe Ravaud, MD, PhD
[+] Article, Author, and Disclosure Information

This article was published at www.annals.org on 19 April 2016.


From INSERM U1153, Université Paris Descartes–Sorbonne Paris Cité, Assistance Publique-Hôpitaux de Paris, Hôpital Hôtel-Dieu, Centre d'Epidémiologie Clinique, and Cochrane France, Paris, France, and Columbia University Mailman School of Public Health, New York, New York.

Acknowledgment: The authors thank Laura Smales (BioMedEditing, Toronto, Ontario, Canada) for proofreading this manuscript.

Disclosures: Authors have disclosed no conflicts of interest. Forms can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M15-2521.

Editors' Disclosures: Christine Laine, MD, MPH, Editor in Chief, reports that she has no financial relationships or interests to disclose. Darren B. Taichman, MD, PhD, Executive Deputy Editor, reports that he has no financial relationships or interests to disclose. Cynthia D. Mulrow, MD, MSc, Senior Deputy Editor, reports that she has no relationships or interests to disclose. Deborah Cotton, MD, MPH, Deputy Editor, reports that she has no financial relationships or interest to disclose. Jaya K. Rao, MD, MHS, Deputy Editor, reports that she has stock holdings/options in Eli Lilly and Pfizer. Sankey V. Williams, MD, Deputy Editor, reports that he has no financial relationships or interests to disclose. Catharine B. Stack, PhD, MS, Deputy Editor for Statistics, reports that she has stock holdings in Pfizer and Johnson & Johnson.

Grant Support: Drs. Trinquart, Attiche, and Bafeta were supported by Cochrane France.

Reproducible Research Statement:Study protocol: Available in Supplement 1. Statistical code: See Supplement 2; for more detail, contact Dr. Trinquart (e-mail, ludovic.trinquart@aphp.fr). Data set: Available at http://clinicaltrialnetworks.com.

Requests for Single Reprints: Ludovic Trinquart, PhD, Hôpital Hôtel-Dieu, Centre d'Epidémiologie Clinique, 1 place du Parvis Notre-Dame, 75181 Paris Cedex 04, France; e-mail, ludovic.trinquart@aphp.fr.

Current Author Addresses: Drs. Trinquart, Attiche, Bafeta, Porcher, and Ravaud: Hôpital Hôtel-Dieu, Centre d'Epidémiologie Clinique, 1 place du Parvis Notre-Dame, 75181 Paris Cedex 04, France.

Author Contributions: Conception and design: P. Ravaud, L. Trinquart.

Analysis and interpretation of the data: N. Attiche, R. Porcher, P. Ravaud, L. Trinquart.

Drafting of the article: P. Ravaud, L. Trinquart.

Critical revision for important intellectual content: N. Attiche, R. Porcher, P. Ravaud, L. Trinquart.

Final approval of the article: N. Attiche, A. Bafeta, R. Porcher, P. Ravaud, L. Trinquart.

Obtaining of funding: P. Ravaud.

Administrative, technical, or logistic support: P. Ravaud.

Collection and assembly of data: N. Attiche, A. Bafeta, P. Ravaud, L. Trinquart.


Ann Intern Med. 2016;164(10):666-673. doi:10.7326/M15-2521
© 2016 American College of Physicians
Text Size: A A A

Background: Ranking of interventions is one of the most appealing elements of network meta-analysis. There is, however, little evidence about the reliability of these rankings.

Purpose: To empirically evaluate the extent of uncertainty in intervention rankings from network meta-analysis.

Data Sources: Two previous systematic reviews that involved searches of the Cochrane Library, MEDLINE, and Embase up to July 2012 for articles that included networks of at least 3 interventions.

Study Selection: 58 network meta-analyses involving 1308 randomized trials and 404 interventions with available aggregated outcome data.

Data Analysis: Each network was analyzed with a Bayesian approach. For each intervention, the surface under the cumulative ranking curve (SUCRA) and its 95% credible interval (95% CrI) were estimated. Through use of the SUCRA values, the interventions were then rank-ordered between 0% (worst) and 100% (best).

Data Synthesis: The median width of the 95% CrIs of the SUCRA was 65% (first to third quartile, 38% to 80%). In 28% of networks, there was a 50% or greater probability that the best-ranked treatment was actually not the best. No evidence showed a difference between the best-ranked intervention and the second and third best-ranked interventions in 90% and 71% of comparisons, respectively. In 39 networks with 6 or more interventions, the median probability that 1 of the top 2 interventions was among the bottom 2 was 35% (first to third quartile, 14% to 59%).

Limitation: This analysis did not consider such factors as the risk of bias within trials or small-study effects that may affect the reliability of rankings.

Conclusion: Treatment rankings derived from network meta-analyses have a substantial degree of imprecision. Authors and readers should interpret such rankings with great caution.

Primary Funding Source: Cochrane France.

Figures

Grahic Jump Location
Figure 1.

Flow diagram showing the selection of published network meta-analyses.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Mean ranks and 95% CrIs of the top 3 treatments in 5 network meta-analyses.

Values along the x-axis refer to study identification numbers. CrI = credible interval.

Grahic Jump Location
Grahic Jump Location
Figure 3.

SUCRA values and 95% CrIs of the top 3 treatments in 5 network meta-analyses.

Values along the x-axis refer to study identification numbers. In each network, the SUCRA and the 95% Crl for each intervention were estimated. The median width of the 95% CrIs for the SUCRA was 66.7% (quartile 1 to quartile 3, 37.5% to 80.0%). CrI = credible interval; SUCRA = surface under the cumulative ranking curve.

Grahic Jump Location
Grahic Jump Location
Appendix Figure 1.

95% CrIs for the mean ranks of the top 3 treatments across 58 network meta-analyses.

In each network, the mean ranks and the 95% CrIs for each intervention were estimated. CrI = credible interval; SUCRA = surface under the cumulative ranking curve.

Grahic Jump Location
Grahic Jump Location
Appendix Figure 2.

95% CrIs for the SUCRA of the top 3 treatments across 58 network meta-analyses.

In each network, the SUCRA and the 95% CrI for each intervention were estimated. The median width of the 95% CrIs for the SUCRA was 66.7% (quartile 1 to quartile 3, 37.5% to 80.0%). CrI = credible interval; SUCRA = surface under the cumulative ranking curve.

Grahic Jump Location

Tables

References

Letters

CME Activities are only available to ACP members and Individual Annals subscribers. If you are a member or a subscriber please sign in. Otherwise please become a member or subscribe to Annals.
NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment/Letter
In Response
Posted on May 9, 2016
Haitian Zhang MD
Unknown
Conflict of Interest: None Declared
I'm surprised that I should read the artical "Uncertainty in Treatment Rankings Reanalysis of Network Meta-analyses of Randomized Trials" after the ASA's statement on p-values about context, process, and purpose was made a short time ago. The conclusion from the artical "Uncertainty in Treatment Rankings Reanalysis of Network Meta-analyses of Randomized Trials" states that treatment rankings derived from network meta analyses have a substantial degree of imprecision. Authors and readers should interpret such rankings with great caution. I do not agree with the conclusion. First of all, 95%CrI does not provide a good measure of evidence regarding a model or hypothesis according to the ASA's statement, and you can not say that no evidence of difference between the best-ranked intervention and the second, third and fourth best-ranked interventions according to the 95%CrI overlapping. Secondly?although the surface under the cumulative ranking curve (SUCRA) has its shortcomings?the 95%CrI overlapping between the best-ranked intervention and the second, third and fourth best-ranked interventions is not the fault of the SUCRA, and it comes from the data themselves. It means that the uncertainty in treatment rankings comes from the randomized trials, not the network meta-analyses. thirdly, the 95%CrI does not tell us that the observed difference between the groups is real or could be a chance finding? interpretation of the clinical importance of the results requires clinical/biological judgement, and cannot be assessed solely through statistical testing. The results from the simulation data tell us that the SUCRA is perfect.
Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Journal Club
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)