Disclosures: Christine Laine, MD, MPH, Editor in Chief, reports that she has no financial relationships or interests to disclose. Darren B. Taichman, MD, PhD, Executive Deputy Editor, reports that he has no financial relationships or interests to disclose. Cynthia D. Mulrow, MD, MSc, Senior Deputy Editor, reports that she has no relationships or interests to disclose. Deborah Cotton, MD, MPH, Deputy Editor, reports that she has no financial relationships or interest to disclose. Jaya K. Rao, MD, MHS, Deputy Editor, reports that she has stock holdings/options in Eli Lilly and Pfizer. Sankey V. Williams, MD, Deputy Editor, reports that he has no financial relationships or interests to disclose. Catharine B. Stack, PhD, MS, Deputy Editor for Statistics, reports that she has stock holdings in Pfizer.
This article was published at www.annals.org on 22 December 2016.
; Discrepancies Between Prespecified and Reported Outcomes. Ann Intern Med. 2016;164:374-375. doi: 10.7326/L15-0615
Download citation file:
Published: Ann Intern Med. 2016;164(5):374-375.
Published at www.annals.org on 22 December 2015
The Centre for Evidence-Based Medicine Outcome Monitoring Project (1) recently commented on Gepner and colleagues' article (2) and mentioned MacPherson and associates' trial (3) on their Web site. This watchdog group aims to monitor the reporting of prespecified outcomes in all clinical trials published in 5 top-tier medical journals, including Annals. Because these comments express concerns about “switched” or incompletely reported outcomes, we would like to describe our editorial process and potential reasons for the discrepancies noted by COMPare. We attempted to post a public comment on the group's Web site to express our concerns about their assessments, but there were no means for doing so.
We share COMPare's overarching goals to ensure the validity and reporting quality of biomedical studies, but we differ on how best to achieve those aims. We routinely ask authors of clinical trials to submit their protocols with their manuscripts, and we examine trial registries for the initial and final information entered about trials. We review both items because registries include only extracted information, do not routinely monitor whether the data in the registry match the protocol, and may not be updated when the protocol changes. We therefore rely primarily on the protocol for details about prespecified primary and secondary outcomes, study interventions and procedures, and statistical analysis. To be consistent with CONSORT recommendations, we ask authors to describe, either in the manuscript or in an appendix, any major differences between the trial registry and protocol, including changes to trial end points or procedures.
According to COMPare's protocol (4), abstractors are to look first for a protocol that has been published before a trial's start date. If they find no such publication, they are supposed to review the initial trial registry data. Thus, COMPare's review excludes most protocols published after the start of a trial and unpublished protocols or their amendments and ignores amendments or updates to the registry after a trial's start date. The initial trial registry data, which often include outdated, vague, or erroneous entries, serve as COMPare's “gold standard.”
Our review indicates problems with COMPare's methods. For 1 trial (5), the group apparently considered the protocol (6) published well after data collection ended. However, they did not consider the protocol (7) published 2 years before MacPherson and associates' primary trial was published (3). That protocol (7) was more specific in describing the timing of the primary outcome (assessment of neck pain at 12 months) than the registry (assessment of neck pain at 3, 6, and 12 months), yet COMPare deemed the authors' presentation of the 12-month assessment as primary in the published trial to be “incorrect.” Similarly, the group's assessment of Gepner and colleagues' trial (2) included an erroneous assumption about one of the prespecified primary outcomes, glycemic control, which the authors had operationalized differently from the abstractors. Furthermore, the protocol for that trial clearly listed the secondary outcomes that the group deemed as being not prespecified.
On the basis of our long experience reviewing research articles, we have learned that prespecified outcomes or analytic methods can be suboptimal or wrong. Regardless of prespecification, we sometimes require the published article to improve on the prespecified methods or not emphasize an end point that misrepresents the health effect of an intervention. Although prespecification is important in science, it is not an altar at which to worship. Prespecification can be misused to sanctify both inappropriate end points, such as biomarkers, when actual health outcomes are available and methods that are demonstrably inferior.
The Centre for Evidence-Based Medicine Outcome Monitoring Project's assessments seem to be based on the premise that trials are or can be perfectly designed at the outset, the initial trial registry fully represents the critical aspects of trial conduct, all primary and secondary end points are reported in a single trial publication, and any changes that investigators make to a trial protocol or analytic procedures after the trial start date indicate bad science. In reality, many trial protocols or reports are changed for justifiable reasons: institutional review board recommendations, advances in statistical methods, low event or accrual rates, problems with data collection, and changes requested during peer review. The Centre for Evidence-Based Medicine Outcome Monitoring Project's rigid evaluations and the labeling of any discrepancies as possible evidence of research misconduct may have the undesired effect of undermining the work of responsible investigators, peer reviewers, and journal editors to improve both the conduct and reporting of science.
We have led or participated in many efforts to improve the transparency and accuracy of scientific reporting. We will continue to encourage authors of clinical trials to make their protocols available to others and to update their trial registry information. We respect COMPare's effort to draw attention to the importance of accurate and complete public description of clinical trial procedures, but we do not believe that their approach—that purports to draw a simple methodological line between “good” and “bad” reporting (or editing)—serves our common cause. Until COMPare's methods are modified to provide a more accurate, complete, and nuanced evaluation of published trial reports, we caution readers and the research community against considering the group's assessments as an accurate reflection of the quality of the conduct or reporting of clinical trials.
The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.
Kyle Lehenbauer, MD FACP
Veterans Affairs Medical Center, Minneapolis
December 29, 2015
Your response makes some valid points but I think one of the main aims of the COMPare group is to point out that having too many outcomes and especially switching outcomes increases the probability of having a good result based more on random chance than the actual intervention. Being able to switch outcomes or not report pre-specified outcomes makes a p value useless. How can you publish a probability that the results were due to random chance after the team has seen the results and gotten to choose which results they publish?
Editor in Chief, Annals of Internal Medicine
February 12, 2016
Conflict of Interest:
Employment: Editor in Chief, Annals of Internal Medicine
Review Process for Clinical Trials
The editors of Annals of Internal Medicine agree with Dr. Lehenbauer that having too many outcomes or switching outcomes can increase the possibility of detecting a favorable outcome in a clinical trial. That is why Annals’ review process for clinical trials requires the authors to submit copies of the pre-enrollment protocols with amendments to assure that the report accurately reflects the scientific and clinical intent outlined in the protocol. When the review process generates requests for authors to report outcomes not specified in the protocol or the authors choose themselves to present such outcomes, we ask authors to indicate these as post hoc or exploratory analyses. Unfortunately, there is no current global mandate that requires clinical trialists to publish or publicly post protocols prior to the start of patient enrollment. Thus, we often need to obtain the protocols from the authors. We do not rely on the trial registries alone.
Ben Goldacre, Henry Drysdale, Kamal R. Mahtani, on behalf of the COMPare Trials Project
Centre for Evidence-Based Medicine, University of Oxford, Oxford, UK
April 7, 2016
Annals needlessly using secret documents behind closed doors
We are extremely concerned by the above response from Annals’ Editor in Chief, Christine Laine. The COMPare project has assessed trials in five major journals and provided extensive documentation of outcome switching in all five, including Annals . Our figures are consistent with extensive previous published work on the prevalence of outcome switching [2,3]. Disappointingly, Annals’ editors have gone to great lengths to avoid open discussion of this issue , argued for the use of publicly inaccesible protocols, and argued against using registers as a source of time-stamped information . This is both concerning and counterproductive, particularly as Annals also presents a public facing stance that apparently supports wholehearted use of clinical trial registries . Trial registers were explicitly designed as a publicly accessible source of information on prespecified outcomes in order to address selective outcome reporting. They were set up with extensive endorsement from ICMJE, journals, professional bodies, and organisations such as WHO. There is no reason for Annals not to use registers, or to dismiss their contents. With their current stance Annals are actively undermining one of the most important initiatives to improve transparency and reporting standards in the past two decades. More concerning is their preference for using protocols that are unpublished, thus reducing transparency, and to use them behind closed doors, in a manner whereby Annals’ assessments cannot be scrutinised. When COMPare initially assessed the Everson trial  published in Annals, we used the pre-specified outcomes posted by the trialists to the trial registry, because this was the only publicly accessible date-stamped source for this information. When Annals’ editors attempted to argue  that protocols are a better source of information, we asked for this protocol, which Annals reported as publicly available, but were told it was only available if we signed a legal document promising not to discuss the information . This is clearly no way to ensure best practice in trial reporting. While creating a public impression of support for improved standards, Annals are perpetuating the worst practices of the past, with a covert system unnecessarily using secret inaccessible documents behind closed doors to check for outcome switching. Worse still, this system is demonstrably failing; data from our team  and others [2, 3] have shown that Annals is repeatedly permitting undisclosed outcome switching and therefore routinely misreporting high impact clinical trials. To compound all this, Annals has sought to avoid public discussion of these shortcomings . We believe Annals’ mechanisms for addressing outcome switching are outdated and overdue for revision. We strongly encourage Annals to reconsider their position and to engage in a serious open discussion of outcome switching and their mechanisms to prevent it, on the pages of the journal. Only then can readers participate and fully understand exactly what Annals is - and is not - doing to ensure that all prespecified trial outcomes are reported.Ben Goldacre, Henry Drysdale, Kamal R. Mahtani, on behalf of the COMPare Trials Project. References: The COMPare Trials Project, www.COMPare-trials.org 2016. Fleming, Padhraig S., Despina Koletsi, Kerry Dwan, and Nikolaos Pandis. “Outcome Discrepancies and Selective Reporting: Impacting the Leading Journals?” PLoS ONE 10, no.5 (May 21, 2015): e0127495. doi:10.1371/journal.pone.0127495. Jones, Christopher W., Lukas G. Keil, Wesley C. Holland, Melissa C. Caughey, and Timothy F. Platts-Mills. “Comparison of Registered and Published Outcomes in Randomized Controlled Trials: A Systematic Review.” BMC Medicine 13 (2015): 282. doi:10.1186/s12916-015-0520-3. COMPare blog, April 5 2016: http://compare-trials.org/blog/breaches-of-publication-ethics-at-annals/  COMPare blog, January 20 2016: http://compare-trials.org/blog/where-does-annals-of-internal-medicine-stand-on-outcome-switching-a-detailed-response/  Sharing Clinical Trial Data: A Proposal From the InternationalCommittee of Medical Journal Editors, Ann Intern Med. 2016. doi:10.7326/M15-2928 Everson G et al, Sofosbuvir With Velpatasvir in Treatment-Naive Noncirrhotic Patients With Genotype 1 to 6 Hepatitis C Virus Infection, Ann Intern Med., published online first at www.annals.org on 10 November 2015. COMPare blog, April 6 2016:http://compare-trials.org/blog/another-ethical-breach-at-annals-misleading-transparency-statements-and-secret-protocols/
Copyright © 2016 American College of Physicians. All Rights Reserved.
Print ISSN: 0003-4819 | Online ISSN: 1539-3704
Conditions of Use
This PDF is available to Subscribers Only