Paul S. Mueller, MD, MPH; Victor M. Montori, MD, MSc; Dirk Bassler, MD, MSc; Barbara A. Koenig, PhD; Gordon H. Guyatt, MD, MSc
Potential Financial Conflicts of Interest: Consultancies: B.A. Koenig (Affymetrix Corporation).
Requests for Single Reprints: Paul S. Mueller, MD, MPH, Program in Professionalism and Bioethics, Department of Medicine, Mayo Clinic College of Medicine, 200 First Street SW, Rochester, MN 55905; e-mail, firstname.lastname@example.org.
Current Author Addresses: Drs. Mueller, Montori, and Koenig: Mayo Clinic College of Medicine, 200 First Street SW, Rochester, MN 55905.
Dr. Bassler: University Children's Hospital, Department of Neonatology, Calwerstrasse 7, 72076 Tuebingen, Germany.
Dr. Guyatt: McMaster University, 1200 Main Street West, Room 2C12, Hamilton, L8N 3Z5 Ontario, Canada.
Mueller PS, Montori VM, Bassler D, Koenig BA, Guyatt GH. Ethical Issues in Stopping Randomized Trials Early Because of Apparent Benefit. Ann Intern Med. 2007;146:878-881. doi: 10.7326/0003-4819-146-12-200706190-00009
Download citation file:
Published: Ann Intern Med. 2007;146(12):878-881.
Stopping randomized trials early because of an apparent benefit is becoming more common. To protect and promote the interests of trial participants, investigators may feel obligated to stop a trial early because of the apparent benefit of a study treatment (compared with placebo or other treatment). There are, however, serious ethical problems with doing so. Truncated trials systematically overestimate treatment effects; in cases where the number of accrued outcome events is small, the overestimation may be very large. Generating seriously inflated estimates of treatment effect violates the ethical research requirement of scientific validity. Subsequent use of inflated estimates to inform clinical decision making and practice guidelines violates the ethical requirements of social value and a favorable riskâ€“benefit ratio. Researchers should ensure that a large number of outcome events accrues before stopping a trial and then continue recruitment to assess whether positive trends continue. This can balance the need to protect research participants with the ethical requirements of scientific validity, social value, and a favorable riskâ€“benefit ratio.
Stopping randomized trials early because of an apparent benefit is a growing phenomenon. A recent systematic review found that the number of randomized trials stopped early for benefit has more than doubled since 1990 (1). To protect and promote the interests of trial participants, investigators may feel ethically obligated to stop a trial early because of the unexpected harm or apparent benefit of a study treatment. If a study treatment's benefit far outweighs its adverse effects, is it not unethical to continue enrolling patients in a trial in which, as is typically the case, patients have a 50% chance of receiving a placebo or an inferior treatment?
In this article, we argue that stopping a randomized trial early for apparent benefit is often unethical and can be justified only under restricted circumstances. If the scientific community were to accept our arguments, then the approach that investigators, institutional review boards, and data monitoring committees take to the practice of stopping trials early for apparent benefit would substantially change.
Emanuel and colleagues (2) describe a framework of 7 requirements for determining whether clinical research is ethical. We use this framework to identify and assess the ethical issues raised by stopping trials early because of apparent benefit (Table).
The purpose of a trial of alternative interventions is to generate an estimate of treatment effect that closely approximates the true effect and is not misleading. This requires application of scientific procedures that yield valid and reliable data and thus minimize both systematic and random error.
A systematic review of randomized trials stopped early for apparent benefit (1) found that many of the trials yielded implausibly large treatment effects; the median relative risk was 0.53. Apparent large treatment effects occurred much more frequently when trials accrued only a small number of events. The odds of a treatment effect larger than the overall median relative risk of 0.53 was 28 times greater (95% CI, 11 to 73) among trials in which fewer than the median of 66 events accrued than among trials in which more events accrued. These results, which are consistent with predictions from statistical theory (3), suggest that stopping trials early for apparent benefit will systematically overestimate treatment effects.
The scientific validity of trials that are stopped early is further compromised when trials yield inconclusive data about outcomes that did not influence trial truncation but are nonetheless important to patients, such as disease-free survival, symptom control, quality of life, and adverse effects of treatment. For example, a trial of vitamin E supplementation in premature newborns that was stopped early because of an apparent reduction in intracranial hemorrhage (4) failed to detect the increase in sepsis associated with vitamin E supplementation that subsequent trials identified (5).
It is understandable that investigators focus their ethical obligations on research participants. Such focus, however, risks neglecting obligations to society. The tendency of truncated trials to overestimate the effect of a treatment on the end point that resulted in trial truncation and to yield insufficient data about other important outcomes endangers the wider community to whom the results will be applied (6). On reviewing the results of a truncated trial, astute clinicians might appropriately conclude that the benefits of the intervention remain uncertain. However, less skeptical clinicians might assume that the results are true and inappropriately expose patients to the intervention and its unknown harms.
Consider the results of a trial in which the investigators continued to enroll patients even though prespecified criteria for early stopping were met. Two interim analyses of a randomized trial of 5 versus 4 courses of chemotherapy in patients with acute myeloid leukemia (7) found apparent large benefits to the 5-course regimen (relative odds reduction of 53% [CI, 23% to 71%; P = 0.003] in the first analysis and 45% [CI, 20% to 62%; P = 0.0002] in the second analysis). Finding these results too good to be true, the data monitoring committee recommended continuing the trial, which ultimately showed a trend in favor of the 4-course regimen. Had the investigators terminated the trial in accordance with their stopping rule, subsequent patients with leukemia may have experienced the toxicity of an additional course of chemotherapy without benefit.
Harm resulting from the misleading findings of truncated trials can be compounded if the findings influence the recommendations of clinical practice guideline panels. Investigators conducting a trial that involved patients undergoing vascular surgery (8) stopped the trial early when 2 of 53 patients randomly assigned to receive the β-blocker bisoprolol and 18 of 59 control patients had major cardiovascular events (relative risk reduction, 90% [CI, 59% to 98%]). These results contributed to recommendations by the American Heart Association and the American College of Cardiology favoring administration of β-blockers to patients with cardiac risk factors who were undergoing noncardiac surgery (9). However, these results contradict those of 2 much larger subsequently published trials, neither of which suggested that β-blockers reduce cardiac risk in patients undergoing noncardiac surgery (10, 11).
Further social detriment may occur when clinicians compromise the ability of others to conduct more definitive studies by placing undue confidence in the results of a truncated trial. Investigators (including 2 contributors to this article) who obtained funding for a trial of β-blockers in noncardiac surgery with an enrollment target of 10 000 patients (12) faced challenges in persuading clinicians that the question remained unanswered.
Key prerequisites for informed consent include the participant's decision-making capacity and voluntariness and whether he or she had received adequate information to decide that participation in the research was in alignment with his or her values and goals. However, informed consent is not a single event, but it is an ongoing collaboration between participants and investigators. When important changes occur during a trial, investigators should inform participants of the changes.
One justification for stopping a trial early for benefit is to inform study participants of the preliminary results and offer them the superior treatment. According to this argument, uncertainty about the relative merits of alternative interventions (equipoise) has been lost and informed clinicians and patients will overwhelmingly choose the superior treatment (13). However, as we have pointed out, the astute clinician or patient may remain skeptical about a treatment's apparent benefits if the findings come from a truncated trial. Unfortunately, many clinicians and even more patients probably will not have the knowledge and understanding to appropriately interpret the results. Disclosing interim results to study participants may therefore prove misleading.
Furthermore, if investigators were to continue a trial after informing patients of the interim results, patients would be unblinded and may cross over or leave the trial. These behaviors create problems in interpreting trial results by further weakening inferences about the efficacy and safety of the intervention and compromising the ethical requirement of scientific validity.
Finally, stopping a trial early does not guarantee that current and potential trial participants will receive the apparently beneficial treatment (assuming that one believes they should). Studies of dissemination of new treatments reveal that long delays, such as those between reports of randomized trials and recommendations of experts in review articles and textbooks, are common (14). Continuing a 2-group trial gives participants at least a 50% chance of receiving the experimental treatment, whereas if the trial is stopped early, the probability that participants will receive the treatment due to rapid dissemination is likely to be considerably less than 50%.
Trials may have stopping rules that allow early termination because of genuine (although misguided) ethical concerns. However, investigators, trial sponsors, journals, and patients may all have additional motives for stopping trials early for apparent benefit. For example, truncated trials that report a large treatment effect tend to be published in the most prestigious medical journals (1), which enhances the careers of the investigators and increases the likelihood that they will receive grants. Funding agencies have an interest in stopping trials early to minimize research costs. Pharmaceutical and for-profit sources that financially support trials are interested not only in controlling costs but also in the publicity and market share that result from reporting a trial stopped early for apparent benefit. Medical journals are interested in these trials because of publicity and citations, which result in increased journal impact factor, prestige, and advertising revenue. And patients and their advocates are motivated to stop a trial early when the experimental intervention is promising in order to hasten delivery of the intervention to clinical practice. All of these motives may affect investigators' decisions and encourage an inappropriately early stop to a trial. These considerations mandate that institutional review boards and data monitoring committees understand the principles outlined in this article and insist on appropriate standards for stopping a trial early for apparent benefit to maintain the ethical integrity of clinical trials.
We have described the dangers of stopping trials early for apparent benefit and the ethical arguments for continuing trials to their planned conclusion. This issue, however, is not 1-sided. If a treatment truly has large benefits, one would want rapid dissemination of knowledge of that treatment. Indeed, if the treatment is associated with minimal harm, there is a legitimate ethical argument for terminating a trial early and thus facilitating rapid dissemination of the results.
Consider a randomized trial that tested the effect of intensive lifestyle modification on the incidence of diabetes (15). The trial initially planned a sample size of 3000 participants. After enrolling 3234 participants and observing a total of 436 events, the investigators found a plausible relative risk reduction of 42% (CI, 34% to 52%) and terminated follow-up of the enrolled trial participants 1 year earlier than planned. Other large trials (16, 17) confirmed the accuracy of this estimate of benefit.
Is it appropriate to stop a trial early if the apparent treatment benefit is unlikely to largely overestimate the true benefit? How can one be certain that is the case? Application of currently published stopping rules do not achieve this goal (3). Investigators may set a relatively rigorous P-value threshold (say, 0.001) but may examine their findings early when few events have accrued. If their trial crosses the prespecified threshold at this early point, the apparent treatment benefit almost certainly represents a very large overestimate. This was the case in the randomized trial of bisoprolol in patients undergoing noncardiac surgery (8): The investigators took a single look at their data according to an O'Brien–Fleming stopping rule with a P-value threshold of 0.001. However, if β-blockers do reduce vascular events in patients undergoing noncardiac surgery, the effect is far more modest than the 90% relative risk reduction that the trial results suggested.
The solution is to demand that a large number of events accrue before investigators or data monitoring committees examine interim data. However, just how many events are required to avoid large overestimates of effect remains to be determined. Without this knowledge, we have outlined the ethical considerations that mandate a conservative approach. Requiring that 200 to 400 events accrue before invoking a stopping rule associated with a low P value (1), such as 0.001, and then continuing enrollment and follow-up for a further period to ensure that the trend continues should minimize seriously misleading results.
The approach we describe should be included in trial protocols, reported when registering a trial, and conveyed to trial participants. Furthermore, data monitoring committees should not deviate from these rules (18).
In addition to more stringent early stopping rules, the ethical issues that we have discussed suggest further recommendations. First, journals should require that reports of truncated trials describe the rationale for early stopping, including the statistics and rules applied. These descriptions are not currently routine (1). Second, investigators and data monitoring committees should consider the context of the illness and the treatment being studied when deciding to stop a trial early. The adverse consequences of overestimating a treatment effect are greater if the treatment is toxic and expensive, making the decision to stop early grave. Even when the treatment seems to incur minimal toxicity and cost, investigators and data monitoring committees should bear in mind that approximately 25% of new drugs eventually prove to have serious adverse effects that were unexpected when they entered the market (19).
In summary, understanding the biases associated with stopping randomized trials early, adopting conservative stopping rules, and considering the potential harms to subsequent patients of overestimating treatment effects will reduce the ethical problems resulting from stopping trials early.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Istanbul University, Cerrahpasa Medical Faculty, Istanbul, Turkey
June 24, 2007
Transparency in the informed consent process
We read with interest the opposing views of Drs. Mueller et al and Goodman (1, 2) about ending randomized control trials (RCT) "before term". We suggest there is yet another issue about terminating RCTs early, not discussed by either of our colleagues. We propose the information about what the patient will exactly know or knew about trial duration at the time he/she signed the related informed consent at the very start should be transparent to the reviewers of grants, journal submissions, to the readers if the trial report is accepted for publication and, in fact, to the general public. Such transparency, we reason, not only is simply fair to the trial patient, both seeking remedy and helping science as he/she enters a drug trial, but also is an essential starting point whenever we decide to discuss the related ethical and practical issues of a RCT in the optimal way. The transparency we propose also is also desirable when the issue is not early termination but, just the reverse, undue prolongation of RCTs. The interested reader might find the related debate (3, 4) about one such RCT explanatory. This was where we had brought up the point that we as readers, were not optimally informed about the actual intended duration of the RCT in question when it commenced and we had concerns about what the trial patients knew at the beginning and at various time points. It is important here to note that one potential scientific problem in both the extended and the early terminated RCT is an, what some might call an overzealous, attempt of proving one's hypothesis "“ that the drug understudy is working - and shying away from the time honored method of good science, falsification (5). We propose a simple solution to these concerns. Clinical trial registration was initiated in 2005 and has been welcome. We now suggest that effective from a certain point in time in the near future, all manuscripts about clinical trials submitted and published, perhaps even all studies needing informed consent, should submit their informed consents to these registries where the peer reviewers, interested readers, in fact interested anybody, can see what the initial informed consent form (or any amendments thereof) related to that trial actually reads. We envisage our proposal, should it find general support, will surely be a further step in the right direction for transparency which we all seek.
Hasan Yazici, MD 1
Yusuf Yazici, MD 2
1 Istanbul University Cerrahpasa Medical Faculty, Istanbul, Turkey
2 New York University, Hospital for Joint Diseases, New York, New York, USA
Corresponding author: Yusuf Yazici, MD 246 East 20th Street, New York, NY 10003 email@example.com
1- Mueller PS, Montori VM, Bassler D, Koenig BA, Guyatt GH. Ethical issues in stopping randomized trials early because of apparent benefit. Ann Intern Med 2007; 146: 878-81
2- Goodman SN. Stopping at nothing? Some dilemmas of data monitoring in clinical trials. Ann Intern Med 2007; 146: 882-7
3- Yazici Y, Yazici H. Trial of etanercept and methotrexate with radiographic and patient outcomes two-year clinical and radiographic results: Comment on the article by van der Heijde et al. Arthritis Rheum. 2006; 54: 3061-2
4- Van der Heijde D, Klareskog L. Reply to letter by Yazici and Yazici commenting on the two-year report on the trial of etanercept and methotrexate with radiographic and patient outcomes. Arthritis Rheum. 2007.;56:1031-2.
5- Yazici H. Use and Abuse of the Controlled Clinical Trial. Bull NYU Hosp Jt Dis 2007; 65:132-4
Results provided by:
Copyright © 2016 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use
This PDF is available to Subscribers Only