0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Research and Reporting Methods |

Influence of Reported Study Design Characteristics on Intervention Effect Estimates From Randomized, Controlled Trials

Jelena Savović, PhD; Hayley E. Jones, PhD; Douglas G. Altman, DSc; Ross J. Harris, MSc; Peter Jüni, MD; Julie Pildal, MD, PhD; Bodil Als-Nielsen, MD, PhD; Ethan M. Balk, MD, MPH; Christian Gluud, DrSciMed; Lise Lotte Gluud, DrSciMed; John P.A. Ioannidis, MD, DSc; Kenneth F. Schulz, PhD, MBA; Rebecca Beynon, MA; Nicky J. Welton, PhD; Lesley Wood, PhD; David Moher, PhD; Jonathan J. Deeks, PhD; and Jonathan A.C. Sterne, PhD
[+] Article and Author Information

From the School of Social and Community Medicine, University of Bristol, Bristol, United Kingdom; Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom; Health Protection Agency, London, United Kingdom; Institute of Social and Preventive Medicine, University of Bern, Switzerland; The Nordic Cochrane Centre and Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen, Denmark; Hilleroed Hospital, Hilleroed, Denmark; Tufts Clinical and Translational Science Institute, Tufts Medical Center, Boston, Massachusetts; Copenhagen University Hospital Gentofte, Hellerup, Denmark; Stanford Prevention Research Center, Stanford University School of Medicine, and Stanford University School of Humanities and Sciences, Stanford, California; FHI 360, Durham, North Carolina; Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; The Children's Food Trust, Sheffield, United Kingdom; and Public Health, Epidemiology and Biostatistics, School of Health and Population Sciences, College of Medical and Dental Sciences, University of Birmingham, Birmingham, United Kingdom.

Note: Drs. Savović, Jones, and Sterne had full access to all study data and take responsibility for the integrity of the data and accuracy of the data analysis.

Acknowledgment: The authors thank Matthias Egger for providing data and contributing to the initial stages of this project.

Grant Support: This project was funded by the National Institute for Health Research Health Technology Assessment programme in the United Kingdom (project 06/91/10) and will be published in full (13). See the Health Technology Assessment programme Web site for further project information (www.hta.ac.uk). Dr. Savović was partially funded by the Medical Research Council UK (grant G0701659/1). Data contributed by Dr. Als-Nielsen and colleagues were from a study that was partially funded by the Danish Centre for Evaluation and Health Technology Assessment.

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-2327.

Requests for Single Reprints: Jonathan A.C. Sterne, PhD, School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, United Kingdom.

Current Author Addresses: Drs. Savović, Jones, Welton, and Sterne and Ms. Beynon: University of Bristol, School of Social and Community Medicine, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, United Kingdom.

Professor Altman: Centre for Statistics in Medicine, University of Oxford, Wolfson College Annexe, Linton Road, Oxford OX2 6UD, United Kingdom.

Mr. Harris: Health Protection Services, Health Protection Agency, 61 Colindale Avenue, London NW9 5EQ, United Kingdom.

Dr. Jüni: Division of Clinical Epidemiology & Biostatistics, Institut für Sozial- und Präventivmedizin, Universität Bern, Finkenhubelweg 11, CH-3012 Bern, Switzerland.

Dr. Pildal: The Nordic Cochrane Centre, Rigshospitalet, 7811, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.

Dr. Als-Nielsen: Department of Pediatrics, Hilleroed Hospital, Dyrehavevej 29, DK-3400 Hilleroed, Denmark.

Dr. Balk: Tufts Clinical and Translational Science Institute, Tufts Medical Center, 800 Washington Street, Box 63, Boston, MA 02111.

Dr. C. Gluud: Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Blegdamsvej 9, DK-2100 Copenhagen, Denmark.

Dr. L.L. Gluud: Department of Internal Medicine, Copenhagen University Hospital Gentofte, Niels Andersens Vej 65, 2900 Hellerup, Denmark.

Dr. Ioannidis: Stanford University School of Medicine, Room X306, 1265 Welch Road, Stanford, CA 94305.

Dr. Schulz: Quantitative Sciences, FHI 360, Research Triangle Park, 2224 East NC Highway 54, Durham, NC 27713.

Dr. Wood: Children's Food Trust, Regus Temple Quay, 1 Friary, Temple Quay, Bristol BS1 1EA, United Kingdom.

Dr. Moher: Ottawa Methods Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Box 208, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada.

Dr. Deeks: College of Medical and Dental Sciences, University of Birmingham, Edgbaston, Birmingham B15 2TT, United Kingdom.

Author Contributions: Conception and design: J. Savović, D.G. Altman, P. Jüni, J. Pildal, J.P.A. Ioannidis, K. F. Schulz, D. Moher, J.J. Deeks, J.A.C. Sterne.

Analysis and interpretation of the data: J. Savović, H.E. Jones, D.G. Altman, R.J. Harris, B. Als-Nielsen, E.M. Balk, C. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, D. Moher, J.J. Deeks, J.A.C. Sterne.

Drafting of the article: J. Savović, H.E. Jones, J.A.C. Sterne.

Critical revision of the article for important intellectual content: J. Savović, H.E. Jones, D.G. Altman, R.J. Harris, P. Jüni, J. Pildal, B. Als-Nielsen, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, L. Wood, D. Moher, J.J. Deeks, J.A.C. Sterne.

Final approval of the article: J. Savović, D.G. Altman, R.J. Harris, P. Jüni, J. Pildal, B. Als-Nielsen, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, L. Wood, D. Moher, J.J. Deeks, J.A.C. Sterne.

Provision of study materials or patients: J. Pildal, B. Als-Nielsen, E.M. Balk, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz.

Statistical expertise: H.E. Jones, D.G. Altman, R.J. Harris, P. Jüni, J.P.A. Ioannidis, K.F. Schulz, N.J. Welton, J.J. Deeks, J.A.C. Sterne.

Obtaining of funding: D.G. Altman, C. Gluud, J.J. Deeks, J.A.C. Sterne.

Collection and assembly of data: J. Savović, J. Pildal, B. Als-Nielsen, E.M. Balk, C. Gluud, L.L. Gluud, J.P.A. Ioannidis, K.F. Schulz, R. Beynon, L. Wood, J.A.C. Sterne.


Ann Intern Med. 2012;157(6):429-438. doi:10.7326/0003-4819-157-6-201209180-00537
Text Size: A A A

Published evidence suggests that aspects of trial design lead to biased intervention effect estimates, but findings from different studies are inconsistent. This study combined data from 7 meta-epidemiologic studies and removed overlaps to derive a final data set of 234 unique meta-analyses containing 1973 trials. Outcome measures were classified as “mortality,” “other objective,” “or subjective,” and Bayesian hierarchical models were used to estimate associations of trial characteristics with average bias and between-trial heterogeneity. Intervention effect estimates seemed to be exaggerated in trials with inadequate or unclear (vs. adequate) random-sequence generation (ratio of odds ratios, 0.89 [95% credible interval {CrI}, 0.82 to 0.96]) and with inadequate or unclear (vs. adequate) allocation concealment (ratio of odds ratios, 0.93 [CrI, 0.87 to 0.99]). Lack of or unclear double-blinding (vs. double-blinding) was associated with an average of 13% exaggeration of intervention effects (ratio of odds ratios, 0.87 [CrI, 0.79 to 0.96]), and between-trial heterogeneity was increased for such studies (SD increase in heterogeneity, 0.14 [CrI, 0.02 to 0.30]). For each characteristic, average bias and increases in between-trial heterogeneity were driven primarily by trials with subjective outcomes, with little evidence of bias in trials with objective and mortality outcomes. This study is limited by incomplete trial reporting, and findings may be confounded by other study design characteristics. Bias associated with study design characteristics may lead to exaggeration of intervention effect estimates and increases in between-trial heterogeneity in trials reporting subjectively assessed outcomes.

Figures

Grahic Jump Location
Figure 1.

Study flow diagram.

BRANDO = Bias in Randomized and Observational Studies; OR = odds ratio.

* Removal of trials in these 2 steps resulted in the removal of 4 entire meta-analyses.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Estimated RORs and effects on heterogeneity associated with reported study design characteristics.

Univariable analyses were based on all available data. CrI = credible interval; ROR = ratio of odds ratios.

Grahic Jump Location
Grahic Jump Location
Appendix Figure.

Estimated RORs and effects on heterogeneity associated with combinations of study design characteristics.

Univariable analyses were based on subsets of trials with assessments of all 3 study design characteristics (top and middle) and allocation concealment and blinding (bottom). CrI = credible interval; ROR = ratio of odds ratios.

Grahic Jump Location
Grahic Jump Location
Figure 3.

Estimated RORs and effects on heterogeneity from multivariable analyses in which effects of each characteristic are adjusted for the effect of the other 2.

CrI = credible interval; ROR = ratio of odds ratios.

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
PubMed Articles

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)