The full content of Annals is available to subscribers

Subscribe/Learn More  >
Academia and the Profession |

Clinical Trials: Discerning Hype From Substance

Thomas R. Fleming, PhD
[+] Article, Author, and Disclosure Information

From the University of Washington, Seattle, Washington.

Acknowledgment: The author thanks Williamson Bradford and Steven Porter for their supportive role in dissemination of scientifically reliable insights about clinical trials evaluating Actimmune in IPF.

Grant Support: By the National Institutes of Health, National Institute of Allergy and Infectious Diseases (R37 AI 29168).

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M10-1264.

Requests for Single Reprints: Thomas R. Fleming, PhD, Department of Biostatistics, Box 357232, University of Washington, Seattle, WA 98195-7232; e-mail, tfleming@u.washington.edu.

Ann Intern Med. 2010;153(6):400-406. doi:10.7326/0003-4819-153-6-201009210-00008
Text Size: A A A

This article has been corrected. For original version, click “Original Version (PDF)” in column 2.

The interest in being able to interpret and report results in clinical trials as being favorable is pervasive throughout health care research. This important source of bias needs to be recognized, and approaches need to be implemented to effectively address it. The prespecified primary analyses of the primary and secondary end points of a clinical trial should be clearly specified when disseminating results in press releases and journal publications. There should be a focus on these analyses when interpreting the results. A substantial risk for biased conclusions is produced by conducting exploratory analyses with an intention to establish that the benefit-to-risk profile of the experimental intervention is favorable, rather than to determine whether it is. In exploratory analyses, P values will be misleading when the actual sampling context is not presented to allow for proper interpretation, and the effect sizes of outcomes having particularly favorable estimates are probably overestimated because of “random high” bias. Performing exploratory analyses should be viewed as generating hypotheses that usually require reassessment in prospectively conducted confirmatory trials. Awareness of these issues will meaningfully improve our ability to be guided by substance, not hype, in making evidence-based decisions about medical care.


Grahic Jump Location
Figure 1.
Survival effects in colorectal cancer of surgical adjuvant therapy when using LEV alone or the combination of 5-FU and LEV in the NCCTG trial and the Cancer Intergroup Trial.

Adapted from reference (34). 5-FU = 5-fluorouracil; LEV = levamisole; NCCTG = North Central Cancer Treatment Group.

Grahic Jump Location
Grahic Jump Location
Figure 2.
Using pretrial likelihood regarding whether a treatment is truly effective in an initial trial and a confirmatory trial.

When a trial is positive, the probability that the treatment is truly effective depends on the pretrial likelihood that the treatment is effective. In both scenarios, the false-positive error rate is 0.025 (corresponding to requiring a 2-sided P value ≤ 0.05 in the favorable direction in order for the trial to be positive) and the false-negative error rate is 0.10 (corresponding to having 90% statistical power).

* For the initial trial, suppose the pretrial likelihood that the treatment is effective is 4% (i.e., 40/1000). For the confirmatory trial, suppose the pretrial likelihood that the treatment is effective is 60% (i.e., 600/1000).

Grahic Jump Location




Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).


Submit a Comment/Letter
Is it time to say goodbye to ' Intent to Treat' analysis?
Posted on September 22, 2010
Chakrapani Prakash, MD FACP
Community Medical Center, Toms River, NJ
Conflict of Interest: None Declared

Controlled double blind trials involving two or three different arms in chronic conditions like inflammatory bowel disease or hepatitis C, are notoriously prolonged, sometimes lasting close to 96 weeks, with end points being clinical improvement or viral suppression with no absolute cure.

Drop out rates are high and fewer than 60% complete the trial as designed. For example in the SONIC trial comparing Infliximab, Infliximab with Azathioprine and Azathioprine with placebo, had a high rate of drop out in the Azathioprine arm [ 84 out of 170 dropped out]. Since the study was designed to include 'intent to treat patients', the denominator 'N' remained the same in all three arms, but patients who completed the trial were disproportionally high in the other two arms (1).

Similarly, in the IDEAL study using PEG Interferon Alfa-2b or 2a with Ribavarin for the treatment of Hepatitis C infection, the dropout rate was even higher, 1416 out of 3070 did not complete the protocol for various reasons (2).

As a clinician, I am more interested in patients who complete the study to compare the responses to treatment. I would also like to know the reasons why patients dropped out, not necessarily include them in the final analysis except for mortality rate and medication side-effects. When everyone is looking for a p values of < 0.05, the clinician is anxious to know whether the results were drawn from patients who completed the trial.

Statistically, it may make sense to include all the dropouts, but doesn't make a whole lot of sense to the clinician.

Time to dump 'intent to treat' analysis and come up with an alternative [ statiticians can be of help here], which will make sense to the practicing doctor.

Chakrapani Prakash, MD, FACP


1. SONIC study group: Infliximab, Azathioprine or combination therapy for Crohn's disease NEJM 2010; 362:1383-1395, April 15, 2010

2. IDEAL Study group: PEG Interferon Alfa-2a or Alfa 2a with Ribavarin for treatment of Hepatitis C NEJM 2009; 361:580-593, August 6, 2010

Conflict of Interest:

None declared

Submit a Comment/Letter

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.


Buy Now for $32.00

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Related Articles
Topic Collections
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.