0
Editorials |

Closing in on the Truth About Recombinant Human Bone Morphogenetic Protein-2: Evidence Synthesis, Data Sharing, Peer Review, and Reproducible Research FREE

Christine Laine, MD, MPH, Editor in Chief; Eliseo Guallar, MD, PhD, Associate Editor, Statistics; Cynthia Mulrow, MD, MSc, Senior Deputy Editor; Darren B. Taichman, MD, PhD, Executive Deputy Editor; John E. Cornell, PhD, Associate Editor, Statistics; Deborah Cotton, MD, MPH, Deputy Editor; Michael E. Griswold, PhD, Associate Editor, Statistics; A. Russell Localio, JD, MPH, MS, Associate Editor, Statistics; Anne R. Meibohm, PhD, Associate Editor, Statistics; Catharine B. Stack, PhD, MS, Deputy Editor, Statistics; Sankey V. Williams, MD, Deputy Editor; and Steven N. Goodman, MD, PhD, MHS, Associate Editor, Statistics
[+] Article and Author Information

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M13-1163.

Requests for Single Reprints: Annals of Internal Medicine, American College of Physicians, 190 N. Independence Mall West, Philadelphia, PA 19106.


Ann Intern Med. 2013;158(12):916-918. doi:10.7326/0003-4819-158-12-201306180-00012
Text Size: A A A

This article has been corrected. The original version (PDF) is appended to this article as a supplement.

Readers may think they are seeing double. They are not. This issue includes 2 systematic reviews (12) that use the same data to address the same question: Compared with iliac crest bone grafting, does recombinant human bone morphogenetic protein-2 (rhBMP-2) safely improve outcomes of spinal fusion surgery? The Yale University Open Data Access Project commissioned the reviews, believing that confidence in the findings would be greatest if both reached the same conclusions. Both include patient-level meta-analyses of data from randomized, controlled trials (RCTs) obtained from Medtronic (Minneapolis, Minnesota), the manufacturer of rhBMP-2 (35). Annals’ review involved 2 internal teams of physician and statistical editors and distinct sets of external reviewers. Each team handled 1 manuscript, and neither had access to the other manuscript or associated reviews until both were accepted for publication. The clinical question is of great interest to orthopedic surgeons and relevant to internists who encounter patients who have had or are considering having spinal fusion. Yet, beyond the clinical issues, these dual reviews spotlight the power of evidence synthesis, data sharing, peer review, and reproducible research.

Although the RCT may be the “king” of study designs, 1 RCT is rarely definitive. The first published rhBMP-2 trial (n = 14) reported that disability scores improved sooner with rhBMP-2 than with bone graft, quality of life improved, and no adverse events occurred (6). Although few would consider the results of this small, initial trial as a solid basis for treatment decisions, some may rely on the largest, later trial. That trial (n = 577) reported that rhBMP-2 improved outcomes, including function, pain, and return to work (7). Yet, after systematic evaluation and synthesis of all available evidence, both systematic reviews published here independently conclude that rhBMP-2, compared with iliac crest bone grafting, does not improve pain or function and increases adverse events, possibly including cancer (12). For the outcome of fusion, the focus of initial enthusiasm for rhBMP-2, 1 review concluded that it improved by a marginally statistically significant amount with rhBMP-2 (overall fusion, 1.14 [95% CI, 1.03 to 1.25]), and the other concluded that it did not (anterior fusion, 1.05 [CI, 0.88 to 1.24]; posterior fusion, 1.16 [CI, 0.96 to 1.41]). Of importance, the CIs for both sets of estimates substantially overlap, and neither review found differences in pain or function, the outcomes that reflect patients’ well-being.

This shows us yet again that conclusions not based on a careful synthesis of all available evidence can be misleading. For that reason, Annals considers systematic reviews to be the best evidence for clinical practice.

The reviews in this issue are not the first to synthesize evidence about rhBMP-2 in spine surgery. A 2011 review by Carragee and colleagues (8) suggested that adverse events associated with rhBMP-2 use ranged from 10% to 50% depending on the surgical approach. Because these authors did not have access to all patient-level RCT data, they had limited ability to precisely determine the benefits and harms of rhBMP-2.

Although access to individual-patient data cannot overcome inherent biases in individual studies derived from design or implementation flaws, our experience illustrates that routine sharing of individual-patient RCT data, perhaps planned for prospectively with other researchers as exemplified by the Early Breast Cancer Trialists’ Collaborative Group (9) or efforts to promote sharing of National Institutes of Health–funded research (http://grants.nih.gov/grants/policy/data_sharing/data_sharing_guidance.htm), could lead to more rapid, precise, generalizable, and trustworthy knowledge about medical interventions than the “closed data” world that currently defines medical research. To learn how to access the data used in the current reviews, visit http://medicine.yale.edu/core/projects/yodap/index.aspx.

These reviews provide an opportunity to reflect on peer review. As new methods of rapidly disseminating research proliferate, some question the benefits of traditional peer review and editing (10). The published reviews are accompanied by supplemental material that includes the initial submitted manuscript and correspondence between editors and authors. The changes from submission to acceptance included greater focus on individual-patient data, more sophisticated handling of missing and sparse data, more detailed methodological appendices (flow diagram, search strategy, details on analyses, and SAS codes), and more thorough assessment of study quality.

The perfect peer-review process does not exist, but this peek into the black box of our procedures should show readers that what they see in the journal can be very different (and, we contend, better) than what the authors initially submit. We see helping authors improve their papers to make their methods more transparent and better align the evidence and the conclusions as one of our core missions. Improving reports of medical science facilitates effective postpublication review by the broader medical community, arguably the most important part of the review process, which ultimately improves the quality of the information available for patient care. The substantive changes we noted between submission and publication of these 2 reviews are typical for most articles we and other major journals publish; consistent with the positive changes seen when we formally evaluated the Annals peer review and editing process nearly 2 decades ago (11); and relevant to concerns about alternatives currently being proposed to prepublication, journal-based peer review, and editing. Although peer review continues after publication, the publication of cursorily vetted research can materially impede the attainment of truth.

Was there added value in soliciting 2 separate systematic reviews on this topic? As advocates of “reproducible research” (12), we think so. Despite the frequent use of rhBMP-2, there remained substantial controversy about its benefits and harms. The fact that 2 independent groups armed with the same question and the same patient-level data, along with the guidance of 2 independent sets of reviewers and editors, arrived at essentially the same conclusions should greatly temper enthusiasm for the intervention. Beyond replicating results, this exercise demonstrates additional value in having different scientists tackle the same data. The Table summarizes key characteristics of each review. There were modest differences in approach, and in fact, they came to somewhat different conclusions about the effect of the 2 interventions on bone fusion. This occurred because of differences in evidence selection and different decisions about analyzing posterior and anterior lumbar interbody fusion separately or together. There was another difference worth noting. Fu and colleagues (2) included a second aim of assessing bias in the original trial reports, and the editors handling their manuscript did not discourage this effort. Simmonds and colleagues (1) focused their efforts on analysis of the data rather than on identifying bias in the initial reports, a focus supported by the editorial team handling their manuscript. Different scientists (and different editors) can approach the same question and even the same data in different ways. This variety of perspectives can only enhance our understanding of the evidence and clarify evidence-based disagreement.

Table Jump PlaceholderTable. Key Features of Systematic Reviews and Meta-analyses of rhBMP-2 

In summary, this Yale University Open Data Access–orchestrated project is a novel exercise that illustrates the value of evidence synthesis, data sharing, peer review, editing, and reproducible research in helping us get closer to the truth. Although it is infeasible to apply this elaborate process to every clinical question, we owe it to our patients to reduce the obstacles that prevent similar efforts from being more common.

Simmonds MC, Brown JV, Heirs MK, Higgins JP, Mannion RJ, Rodgers MA, et al. Safety and effectiveness of recombinant human bone morphogenetic protein-2 for spinal fusion. A meta-analysis of individual-participant data. Ann Intern Med. 2013; 158:877-89.
CrossRef
 
Fu R, Selph S, McDonagh M, Peterson K, Tiwari A, Chou R, et al. Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion. A systematic review and meta-analysis. Ann Intern Med. 2013; 158:890-902.
CrossRef
 
Krumholz HM, Ross JS, Gross CP, Emanuel EJ, Hodshon B, Ritchie JD, et al. A historic moment for open science: the Yale University Open Data Access Project and Medtronic [Editorial]. Ann Intern Med. 2013; 158:910-1.
CrossRef
 
Resnick D, Bozic KJ. Meta-analysis of trials of recombinant human bone morphogenetic protein-2: what should spine surgeons and their patients do with this information? [Editorial]. Ann Intern Med. 2013; 158:912-3.
CrossRef
 
Kuntz RE. The changing structure of industry-sponsored clinical research: pioneering data sharing and transparency [Editorial]. Ann Intern Med. 2013; 158:914-5.
CrossRef
 
Boden SD, Zdeblick TA, Sandhu HS, Heim SE. The use of rhBMP-2 in interbody fusion cages. Definitive evidence of osteoinduction in humans: a preliminary report. Spine (Phila Pa 1976). 2000; 25:376-81.
PubMed
CrossRef
 
Gornet MF, Burkus JK, Dryer RF, Peloza JH. Lumbar disc arthroplasty with Maverick disc versus stand-alone interbody fusion: a prospective, randomized, controlled, multicenter investigational device exemption trial. Spine (Phila Pa 1976). 2011; 36:1600-11.
PubMed
CrossRef
 
Carragee EJ, Hurwitz EL, Weiner BK. A critical review of recombinant human bone morphogenetic protein-2 trials in spinal surgery: emerging safety concerns and lessons learned. Spine J. 2011; 11:471-91.
PubMed
 
Early Breast Cancer Trialists' Collaborative Group. Treatment of Early Breast Cancer Volume 1: Worldwide Evidence 1985–1990. Oxford, UK: Oxford University Pr; 1990.
 
Cohen P.  A digital helping hand for scholarly work. The New York Times. 25 June 2011. Accessed at http://query.nytimes.com/gst/fullpage.html?res=9A0CE1D61F3EF936A15755C0A9679D8B63&ref=patriciacohen on 13 May 2013.
 
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994; 121:11-21.
PubMed
 
Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med. 2007; 146:450-3.
PubMed
 

Figures

Tables

Table Jump PlaceholderTable. Key Features of Systematic Reviews and Meta-analyses of rhBMP-2 

References

Simmonds MC, Brown JV, Heirs MK, Higgins JP, Mannion RJ, Rodgers MA, et al. Safety and effectiveness of recombinant human bone morphogenetic protein-2 for spinal fusion. A meta-analysis of individual-participant data. Ann Intern Med. 2013; 158:877-89.
CrossRef
 
Fu R, Selph S, McDonagh M, Peterson K, Tiwari A, Chou R, et al. Effectiveness and harms of recombinant human bone morphogenetic protein-2 in spine fusion. A systematic review and meta-analysis. Ann Intern Med. 2013; 158:890-902.
CrossRef
 
Krumholz HM, Ross JS, Gross CP, Emanuel EJ, Hodshon B, Ritchie JD, et al. A historic moment for open science: the Yale University Open Data Access Project and Medtronic [Editorial]. Ann Intern Med. 2013; 158:910-1.
CrossRef
 
Resnick D, Bozic KJ. Meta-analysis of trials of recombinant human bone morphogenetic protein-2: what should spine surgeons and their patients do with this information? [Editorial]. Ann Intern Med. 2013; 158:912-3.
CrossRef
 
Kuntz RE. The changing structure of industry-sponsored clinical research: pioneering data sharing and transparency [Editorial]. Ann Intern Med. 2013; 158:914-5.
CrossRef
 
Boden SD, Zdeblick TA, Sandhu HS, Heim SE. The use of rhBMP-2 in interbody fusion cages. Definitive evidence of osteoinduction in humans: a preliminary report. Spine (Phila Pa 1976). 2000; 25:376-81.
PubMed
CrossRef
 
Gornet MF, Burkus JK, Dryer RF, Peloza JH. Lumbar disc arthroplasty with Maverick disc versus stand-alone interbody fusion: a prospective, randomized, controlled, multicenter investigational device exemption trial. Spine (Phila Pa 1976). 2011; 36:1600-11.
PubMed
CrossRef
 
Carragee EJ, Hurwitz EL, Weiner BK. A critical review of recombinant human bone morphogenetic protein-2 trials in spinal surgery: emerging safety concerns and lessons learned. Spine J. 2011; 11:471-91.
PubMed
 
Early Breast Cancer Trialists' Collaborative Group. Treatment of Early Breast Cancer Volume 1: Worldwide Evidence 1985–1990. Oxford, UK: Oxford University Pr; 1990.
 
Cohen P.  A digital helping hand for scholarly work. The New York Times. 25 June 2011. Accessed at http://query.nytimes.com/gst/fullpage.html?res=9A0CE1D61F3EF936A15755C0A9679D8B63&ref=patriciacohen on 13 May 2013.
 
Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med. 1994; 121:11-21.
PubMed
 
Laine C, Goodman SN, Griswold ME, Sox HC. Reproducible research: moving toward research the public can really trust. Ann Intern Med. 2007; 146:450-3.
PubMed
 

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Supplements

Supplemental Content
Supplement. Original Version (PDF)

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Topic Collections
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)