0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Articles |

Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov

Florence T. Bourgeois, MD, MPH; Srinivas Murthy, MD; and Kenneth D. Mandl, MD, MPH
[+] Article and Author Information

From Children's Hospital Boston, Harvard Medical School, and Children's Hospital Informatics Program, Harvard-MIT Division of Health Sciences and Technology, Boston, Massachusetts, and Hospital for Sick Children, Toronto, Ontario, Canada.


Grant Support: By training grant 5T32HD040128 (Dr. Bourgeois) from the National Institute of Child Health and Human Development and grant 5G08LM009778 (Dr. Mandl) from the National Library of Medicine.

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M10-0087.

Reproducible Research Statement:Study protocol: Not available. Statistical code: Available from Dr. Bourgeois (e-mail, florence.bourgeois@childrens.harvard.edu). Data set: Primary data are available at ClinicalTrials.gov and PubMed.

Requests for Single Reprints: Florence T. Bourgeois, MD, MPH, Division of Emergency Medicine, Children's Hospital Boston, 300 Longwood Avenue, Boston, MA 02115; e-mail, florence.bourgeois@childrens.harvard.edu.

Current Author Addresses: Dr. Bourgeois: Division of Emergency Medicine, Children's Hospital Boston, 300 Longwood Avenue, Boston, MA 02115.

Dr. Murthy: Division of Infectious Diseases, Hospital for Sick Children, 555 University Avenue, Toronto, Ontario M5G 1X9, Canada.

Dr. Mandl: Children's Hospital Informatics Program, Harvard-MIT Division of Health Sciences and Technology, Children's Hospital Boston, 300 Longwood Avenue, Boston, MA 02115.

Author Contributions: Conception and design: F.T. Bourgeois, S. Murthy, K.D. Mandl.

Analysis and interpretation of the data: F.T. Bourgeois, S. Murthy, K.D. Mandl.

Drafting of the article: F.T. Bourgeois, S. Murthy, K.D. Mandl.

Critical revision of the article for important intellectual content: F.T. Bourgeois, S. Murthy, K.D. Mandl.

Final approval of the article: F.T. Bourgeois, S. Murthy, K.D. Mandl.

Statistical expertise: F.T. Bourgeois.

Obtaining of funding: F.T. Bourgeois, K.D. Mandl.

Administrative, technical, or logistic support: S. Murthy.

Collection and assembly of data: F.T. Bourgeois, S. Murthy.


Ann Intern Med. 2010;153(3):158-166. doi:10.7326/0003-4819-153-3-201008030-00006
Text Size: A A A

Background: Clinical trial registries are in widespread use to promote transparency around trials and their results.

Objective: To describe characteristics of drug trials listed in ClinicalTrials.gov and examine whether the funding source of these trials is associated with favorable published outcomes.

Design: An observational study of safety and efficacy trials for anticholesteremics, antidepressants, antipsychotics, proton-pump inhibitors, and vasodilators conducted between 2000 and 2006.

Setting: ClinicalTrials.gov, a Web-based registry of clinical trials launched in 1999.

Measurements: Publications resulting from the trials for the 5 drug categories of interest were identified, and data were abstracted on the trial record and publication, including timing of registration, elements of the study design, funding source, publication date, and study outcomes. Assessments were based on the primary funding categories of industry, government agencies, and nonprofit or nonfederal organizations.

Results: Among 546 drug trials, 346 (63%) were primarily funded by industry, 74 (14%) by government sources, and 126 (23%) by nonprofit or nonfederal organizations. Trials funded by industry were more likely to be phase 3 or 4 trials (88.7%; P < 0.001 across groups), to use an active comparator in controlled trials (36.8%; P = 0.010 across groups), to be multicenter (89.0%; P < 0.001 across groups), and to enroll more participants (median sample size, 306 participants; P < 0.001 across groups). Overall, 362 (66.3%) trials had published results. Industry-funded trials reported positive outcomes in 85.4% of publications, compared with 50.0% for government-funded trials and 71.9% for nonprofit or nonfederal organization–funded trials (P < 0.001). Trials funded by nonprofit or nonfederal sources with industry contributions were also more likely to report positive outcomes than those without industry funding (85.0% vs. 61.2%; P = 0.013). Rates of trial publication within 24 months of study completion ranged from 32.4% among industry-funded trials to 56.2% among nonprofit or nonfederal organization–funded trials without industry contributions (P = 0.005 across groups).

Limitations: The publication status of a trial could not always be confirmed, which could result in misclassification. Additional information on study protocols and comprehensive trial results were not available to further explore underlying factors for the association between funding source and outcome reporting.

Conclusion: In this sample of registered drug trials, those funded by industry were less likely to be published within 2 years of study completion and were more likely to report positive outcomes than were trials funded by other sources.

Primary Funding Source: National Library of Medicine and National Institute of Child Health and Human Development, National Institutes of Health.

Figures

Grahic Jump Location
Figure 1.
Search and selection.

* 5 trials in the final trial group studied both an antidepressant and an antipsychotic.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Percentage of trials, by funding source within drug classes.

Industry was the primary funding source for all drug classes and funded more than two thirds of trials among 4 of the classes. Five trials studied both an antidepressant and an antipsychotic and are included in each class.

Grahic Jump Location
Grahic Jump Location
Figure 3.
Percentage of trials with reported favorable outcomes, by trial phase within funding source.

Trials funded by industry sources reported the highest percentage of favorable outcomes across all trial phases. Trial phase was not a significant determinant of outcome (P = 0.70, Cochran–Mantel–Haenszel test controlling for primary funding source). Phase 1 and 2 trials were combined because of the small number of phase 1 trials (n = 13).

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Outcome reporting among drug economic evaluations: the role of the funding source
Posted on August 9, 2010
Steven Simoens
Katholieke Universiteit Leuven
Conflict of Interest: None Declared

Bourgeois et al. (1) reported that industry-funded clinical trials of drugs were more likely to report positive outcomes than trials funded by other sources. My research shows that this association also applies to economic evaluations of drugs. Economic evaluation provides a tool to assess the value for money of a drug and such evidence can be used to inform pharmaceutical pricing/reimbursement decisions.

An observational study identified drug economic evaluations in the Tufts Medical Center Cost-Effectiveness Analysis Registry between 2000 and 2007 (2). This registry includes economic evaluations identified through an extensive MEDLINE search. Data were extracted on publication year, target population, type of drug, country of patient sample, disease classification, funding source, and study outcomes. The source of funding was classified as industry, government, not specified, and not funded. As an economic evaluation may report multiple outcomes pertaining to different drugs, comparators or patient populations, each economic evaluation contributed one or more outcomes to the analysis. Associations between outcomes and funding source were examined by means of the Mann- Whitney U-test.

Among 231 economic evaluations, 145 (63%) economic evaluations were funded by industry, 45 (20%) by government sources, 34 (15%) by unspecified sources, and 7 (2%) were not funded. An analysis of the 608 outcomes reported by these economic evaluations showed that evaluations sponsored by industry generated more favorable outcomes than studies sponsored from other sources (p < 0.001).

One possible explanation of the association between outcomes and funding source is that industry influences the design of economic evaluations with a view to improving the value for money of a drug. This may be exemplified by the observation that economic evaluations funded by industry are more likely to be of lower methodological quality and to be published in scientific journals with lower impact factors (3). Alternative explanations include selection bias, implying that industry only markets and sponsors economic evaluations of those drugs that are likely to generate value for money; and publication bias, as drugs which are not likely to provide value for money are unlikely to be studied or those studies published.

These results highlight the need for authors to disclose all elements of an economic evaluation in accordance with guidelines for the conduct and reporting of economic evaluations issued by professional societies such as the International Society for Pharmacoeconomics and Outcomes Research (4).

References

1. Bourgeois FT, Murthy S, Mandl KD. Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov. Ann.Intern.Med. 2010 Aug 3;153(3):158-66.

2. Centre for the Evaluation of Value and Risk in Health. The Tufts Cost-Effectiveness Analysis Registry.[electronic mail system]. 2009.

3. Bell CM, Urbach DR, Ray JG, Bayoumi A, Rosen AB, Greenberg D, Neumann PJ. Bias in published cost effectiveness studies: systematic review. BMJ 2006 Mar 25;332(7543):699-703.

4. Drummond M, Brown R, Fendrick AM, Fullerton P, Neumann P, Taylor R, Barbieri M. Use of pharmacoeconomics information--report of the ISPOR Task Force on use of pharmacoeconomic/health economic information in health-care decision making. Value Health 2003 Jul;6(4):407-16.

Conflict of Interest:

Financial support for this research project was received from Pharma.be, the Belgian association of the innovative pharmaceutical industry. The sponsor was not involved in the study design; the collection, analysis and interpretation of data; the writing of the manuscript; and the decision to submit the paper for publication.

Effect of actual sample size versus planned sample size
Posted on August 9, 2010
Adam Jacobs
Dianthus Medical Limited
Conflict of Interest: None Declared

First of all, I'd like to congratulate Bourgeois et al for a fascinating piece of research that provides some useful data.

The finding that industry-sponsored trials are more likely than independent trials to report favourable results now seems well established, as this paper adds to a growing literature with similar findings. However, the reasons for that difference are still unclear.

One possible reason occurs to me on reading the paper, which doesn't seem to have been investigated. I see that industry-sponsored trials were more likely to recruit the planned sample size than independent studies. Could that be at least partly responsible for the difference in the proportion of favourable outcomes? Clearly, a trial that succeeds in recruiting its planned sample size is more likely to reach a favourable result than one that doesn't.

However, I couldn't find a description in the paper of whether that had been investigated. Did Bourgeois et al attempt a multivariate analysis including both funding source and whether the final sample size was as planned? I suspect such an analysis might be informative.

Conflict of Interest:

My company provides publication writing services both to pharmaceutical companies and academic researchers, but much more often to the former.

Study size and time to publication
Posted on August 10, 2010
Adam Jacobs
Dianthus Medical Limited
Conflict of Interest: None Declared

Further to the response I wrote yesterday, more questions have occurred to me, this time about the analysis of how many trials were published within 2 years.

First, what was the rationale for choosing 2 years as a cutoff point? The choice of cutoff point clearly affects the conclusions here, as table 3 of the paper shows that the publication rate from industry was no longer lagging behind other categories if the outcome "ever published" is considered (in fact, if you include posting of electronic results as a form of publication, then industry funded studies had the highest publication rate). It seems that it would have been possible to choose 3 years, as that much time had elapsed after all studies before your search. For large, complex, multicentre studies, it can easily take longer than 2 years after study completion for a publication to appear even in the absence of unreasonable delays, as I have explained in more detail here. Could Bourgeois et al also clarify whether the cutoff of 24 months was chosen prospectively or after seeing the data?

Second, it seems possible that the finding of fewer industry-sponsored trials being published within 2 years could be a result of confounding by study size. I note that industry trials were more likely to be multicentre and to have larger sample sizes. I would expect large, multicentre trials to take longer to publish than smaller trials. Is this something that was investigated? Is source of funding still a significant predictor of publication within 2 years in a multivariate analysis after controlling for study size?

Conflict of Interest:

My company provides publication writing services both to pharmaceutical companies and academic researchers, but much more often to the former.

Influence of sample size on results
Posted on August 16, 2010
Erick H. Turner
Oregon Health & Science University
Conflict of Interest: None Declared

I agree with the point made in earlier responses about the importance of the differences in sample size. Although this confounding variable was not mentioned in the discussion section, it could well have played a major role in generating the outcome observed.

As can be seen in Table 1, the median final sample size for the industry-sponsored trials was 3 times larger than that of the nonprofit/nonfederal trials and 8 times larger than that of the government -sponsored trials. Let us assume that three hypothetical trials were performed on the same drug and that the trials were identical in all respects except for sample size, such that each trial resulted in a "medium" effect size of 0.5 (one-half of a standard deviation).(1) Using the median final sample size of 191 for industry-sponsored trials, one calculates a highly "favorable" P=0.001. Using the median final sample size of 63 for the nonprofit/nonfederal-sponsored trial, one calculates P=0.05, right at the threshold of statistical significance, as it is typically defined. And using a sample size of 23 for the government- sponsored trial, one calculates a clearly "unfavorable" P=0.24, which represents a Type II error.

Therefore, the reason why more favorable outcomes were reported for industry-sponsored trials may not be directly related to sponsorship. Rather, the reason might simply be that (a) large, adequately powered trials are more likely to yield "favorable", statistically significant, outcomes and (b) industry sponsorship is associated with larger sample size (as seen in Table 1), possibly because industry has the financial resources and infrastructure to recruit more patients.

The authors might examine this question by adjusting for sample size as they did for the effect of trial phase. However, such an analysis should ideally adjust for all potential confounding variables simultaneously rather than in separate analyses.

On a separate topic, it was somewhat troubling that the authors found (Table 3) more inconsistencies between registered and published primary outcomes within non-industry-sponsored, compared to industry-sponsored, trials. So that journal reviewers and editors can identify post hoc attempts at "spin", more journals should require that manuscripts on clinical trials be accompanied by the corresponding original trial protocols.(2)

Finally, the authors stated that FDA reviews will be made public as a result of proposed changes to ClinicalTrials.gov. In fact, FDA reviews have been publicly available for decades through the Freedom of Information Act and available for download from the FDA website since 1997.(3)

References:

1. J. Cohen, Statistical Power Analysis for the Behavioral Sciences. (Lawrence Erlbaum Associates, Inc., New York, ed. Second Edition, 1988).

2. A.-W. Chan et al., BMJ 332, 1086 (May 6, 2006).

3. E. H. Turner, PLoS Medicine 1, e60 (December 1, 2004, 2004).

Conflict of Interest:

I previously was on speakers' bureaus for three pharmaceutical companies, but these activities ceased five years ago.

No "Good" Explanations for High Rate of Favorable Results
Posted on August 18, 2010
Deborah A. Zarin
National Library of Medicine, NIH
Conflict of Interest: None Declared

Bourgeois et al.(1) report that published industry and nonprofit/nonfederal trials reported "favorable" results 85% and 72% of the time. Although features of their sample and the high rates of censored data (unpublished trials) may have contributed to differences across sponsor types, their results for industry raise concerns. The conclusion that selective publication likely accounts for much of this high rate is well supported. Some commenters have focused on alternative explanations, such as industry may be "good" at designing trials(2). In our view, there are no "good" explanations.

Three possible factors may explain this rate of favorable publications:

1. Selective publication of positive outcomes occurs when trials with positive results are more likely to be published than those with negative results.

2. Strategic study design aimed at producing positive outcomes(3), such as choosing a study population, comparator, or specific outcome measure to increase the likelihood of desired results.

3. Study conduct and analysis that bias the results towards positive outcomes -- ranging from small changes to the prespecified outcome measure and choice of analysis population, to bias in adjudicating events of interest -- undermine the validity of each trial's results.

For example, "good" study designs that produce positive results 85% of the time would raise scientific concerns. To achieve an overall favorability rate of 85%, studies would have to be designed to obtain desirable answers, rather than to answer compelling, scientific questions. Obtaining one particular outcome with such a high probability across studies also raises ethical issues. For example, if the investigators could predict, with 85% probability, which arm was best, was this information conveyed to the IRB and potential study participants? How many volunteers would agree to be randomized into such a study?

As noted by the authors, the trials in this study completed by 2006 and were not subject to the 2007 FDA Amendments Act(4). This law was designed, in part, to address the types of problems revealed by this study; e.g., sponsors will have to enter summary results data to ClinicalTrials.gov based on a legally defined timeline, thus mitigating the effect of publication bias. However, its impact will depend on the efforts of all trial sponsors, whether industry, university, government or other, to ensure the timely entry of complete, accurate, and informative data into ClinicalTrials.gov.

References

1. Bourgeois FT, Murthy S, Mandl KD. Outcome reporting among drug trials registered in ClinicalTrials.gov. Ann Intern Med. 2010;153(3):158- 66.

2. Begley S. Some pharmaceutical clinical trial results are buried, study shows. Newsweek. 2010. August 5, 2010. Available at http://www.newsweek.com/2010/08/05/some-pharmaceutical-clinical-trial- results-are-buried-study-shows.html (accessed on 18 Aug 2010)

3. Safer DJ. Design and reporting modifications in industry-sponsored comparative psychopharmacology trials. J Nerv Ment Dis. 2002;190(9):583- 92.

4. Tse T, Williams RJ, Zarin DA. Reporting "basic results" in ClinicalTrials.gov. Chest. 2009;136(1):295-303.

Conflict of Interest:

Disclaimer: The views expressed are the authors' own. They do not reflect any position or policy of the National Library of Medicine, National Institutes of Health, US Public Health Service, or the Department of Health and Human Services.

Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Drug-induced cardiovascular diseases. Drugs 1977;14(1):29-40.

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)