0
Editorials |

Comparative Efficacy: What We Know, What We Need to Know, and How We Can Get There FREE

Saul Malozowski, MD, PhD, MBA
[+] Article and Author Information

From National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MD 20892-5460.


Disclaimer: This editorial contains the author's views and does not constitute an official position of the U.S. Department of Health and Human Services.

Potential Financial Conflicts of Interest: None disclosed.

Requests for Single Reprints: Saul Malozowski, MD, PhD, MBA, Division of Diabetes, Endocrinology, and Metabolic Diseases, National Institute of Diabetes and Digestive and Kidney Diseases, Democracy Two, 6707 Democracy Boulevard, Room 607, Bethesda, MD 20892-5460; e-mail, sm87j@nih.gov.


Ann Intern Med. 2008;148(9):702-703. doi:10.7326/0003-4819-148-9-200805060-00010
Text Size: A A A

In the past decade, the number of medications to treat osteoporosis has markedly increased (1). These drugs have distinct mechanisms of action. However, information on the efficacy of these compounds relative to one another remains limited, which frustrates physicians who want to practice according to the evidence.

Some may attribute this gap to the U.S. Food and Drug Administration's regulatory mandate requiring well-controlled studies to assess both efficacy and safety for registration trials (2). Although the use of placebo is not a mandate, studies largely rely on placebo as a comparator. There is no doubt that the application of this regulation since 1962 dramatically and positively changed the way medicines are developed and led to the approval of many new drugs.

We are all aware that a particular study design will be able to answer a set of questions, and that no single study can answer all of the questions. An extensive literature discusses different approaches for designing clinical trials and the use of placebos and active controls (34). I believe that placebo-controlled trials are an invaluable tool for defining a drug's efficacy during the drug development process. Nonetheless, some question the ethics and utility of using efficacy relative to placebo as the only hurdle to clear before bringing a drug to the market (56). They ask why we deny a patient a known effective treatment—in the form of an active comparator—while missing a golden opportunity to assess relative efficacy.

A pharmaceutical company takes a substantial risk when it compares a new product with an established drug. Failure to demonstrate superiority over a less expensive agent could be financially ruinous. Providers and patients may not embrace a new therapy when it proves to be inferior to an established product in efficacy, safety, or affordability. In contrast, manufacturers run no such risk when seeking approval based on a comparison with placebo and can then promote their product to physicians who apparently do not question the limitations of this form of evidence.

Under the current incentives for drug development, we have more medicines and more choices, but we often lack the scientific evidence to make choices among them. The use of “placebo-only practice” reduces risk for the pharmaceutical industry, but clearly it does not translate into good medicine or good public policy. In allowing market entry based on a comparison only with placebo—rather than requiring a comparison with established agents—we lose an opportunity to serve the best interest of our patients.

In this issue, Cadarette and colleagues (7) provide information from an observational study conducted to address the knowledge gap created by the lack of head-to-head randomized trials of drugs to prevent and treat osteoporosis. They analyzed a cohort of 43 135 Medicare beneficiaries who were enrolled in 1 of 2 statewide pharmaceutical benefits programs and had received a new prescription for bisphosphonates (alendronate or risedronate), calcitonin, or raloxifene. The study outcome was a fracture of the hip, humerus, radius, or ulna within 12 months of the first dose of drug, as detected by a claim for Medicare reimbursement for services rendered for these diagnoses. By using multivariate statistics to adjust for differences in the frequency of several potential confounders in the populations receiving the 4 drugs, the authors explored the relative efficacy of these drugs. They found that individuals receiving alendronate, risedronate, and raloxifene had similar fracture risk, which was considerably lower than with calcitonin.

The authors provide a balanced discussion of the strengths and weaknesses of their methodological approach. Their tables contained more than 3600 discrete data points, which underscore the complexity of the analysis. This complexity and the tools the authors used may be foreign to many readers who lack the sophisticated statistical training to fully understand the impact of these analyses. I personally belong in this group and have taken the authors' conclusions at face value.

As with many cohort studies, this report has several limitations (810). First, patients were not randomly assigned. Physicians, and therefore clinical circumstances, determined who received which drug. Second, the authors had no access to the factors that led physicians to choose one drug and not another. Third, 1 group—those who received calcitonin—were more ill, based on the number of comorbid conditions, medications listed, and recent hospitalizations. Fourth, fewer patients receiving calcitonin or raloxifene had an osteoporosis diagnosis. All these factors could influence fracture rates independent of the effectiveness of the drug these patients received, and even the most sophisticated multivariate analysis cannot overcome these limitations. To the authors' credit, they acknowledge several situations in which they cannot exclude confounding by indication—whereby a clinical finding that determines whether a person receives a drug is causally related to the study outcome—as a possible explanation for some results. Randomly assigned treatment would negate most of these concerns, which is why we have come to rely on randomized trials for evidence about drug treatments.

Three points illustrate some of the issues with observational studies based on large data sets. The first is about the clinical importance of a small but statistically significant difference. What is the clinical meaning of a statistically significant difference in mean patient age in the raloxifene group (76.9 years) and the bisphosphonate group (78.7 years)? A 2-year difference could be critical between neonates and 2-year-old children and between 13-year-old prepubertal children and children who are sexually mature at age 15. However, in the older cohort studied by Cadarette and colleagues (7), this “difference” is probably meaningless physiologically. A second issue is multiple comparisons, which raise concern that some of the observed differences are due to chance, even though the authors raised the bar for interpreting a result as statistically significant. Finally, data obtained for administrative purposes, which are the core source for this study, are often suspect because it is difficult to assess the accuracy of the information and what information may be missing. The authors allude to some of these issues without fully explaining how they could affect their conclusions.

In the end, the fracture rates were similar for the 2 bisphosphonates and raloxifene. Will this information be useful to select one drug over another? Will it help physicians decide which of these drugs to use in specific patients? I do not think so. Even if the authors had found a clear, undisputed gradient in efficacy among these drugs, I believe that we would need a prospective randomized study before we could use the findings as the basis for practice guidelines.

These results underscore, once more, the lack of good comparative data for choosing drug therapy. Cadarette and colleagues (7) conclude, in accord with many previous articles and reviews, that more randomized, controlled studies are needed, implicitly acknowledging that their approach has important shortcomings. However, the dearth of head-to-head trials suggests that our system for evaluating health technology has not created the incentives to do these trials. Without incentives, the needed studies may never materialize, and we will continue to cope with uncertainty when choosing a drug. A good example of a creative incentive is the pediatric initiative that allowed manufacturers 6 additional months of exclusivity under patent if they made a commitment to study drug effects in pediatric patients (11). Many manufacturers took advantage of this proposal, and I believe that our knowledge about drug effects in children improved.

I submit that we have potential opportunities to explore the relative efficacy of drugs for the treatment of most conditions. As Cadarette and colleagues (7) show, government agencies now pay for drugs for millions of Americans. Under these programs, the U.S. government has already paid for antiosteoporosis drugs for thousands of patients, despite weak evidence about which are the most effective. I propose that we devise an ethical way to prospectively randomly assign patients to different (and apparently equivalent) drug regimens and measure the outcomes of treatment, potential adverse events, drug interactions, and costs. I assert that if the government pays for tests or treatments, it has an obligation to evaluate them relative to one another.

Although this approach alone will not answer every question, it should narrow the gap between what we know and what we need to know, allow us to make better treatment decisions, and help us to better allocate scarce resources.

Saul Malozowski, MD, PhD, MBA

National Institute of Diabetes and Digestive and Kidney Diseases

Bethesda, MD 20892-5460

References

Epstein S.  Update of current therapeutic options for the treatment of postmenopausal osteoporosis. Clin Ther. 2006; 28:151-73. PubMed
CrossRef
 
U.S. Food and Drug Administration.  Federal Food, Drug, and Cosmetic Act. Accessed athttp://www.fda.gov/opacom/laws/fdcact/fdctoc.htmon 7 April 2008.
 
Temple R, Ellenberg SS.  Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Ann Intern Med. 2000; 133:455-63. PubMed
 
Ellenberg SS, Temple R.  Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 2: practical issues and specific cases. Ann Intern Med. 2000; 133:464-70. PubMed
 
Ragi-Eis S, Zerbini CA, Provenza JR, Griz LH, de Gregório LH, Russo LA. et al.  Is it ethical to use placebos in osteoporosis trials? J Clin Densitom. 2006; 9:274-80. PubMed
 
Levine RJ.  Placebo controls in clinical trials of new therapies for osteoporosis. J Bone Miner Res. 2003; 18:1154-9. PubMed
 
Cadarette SM, Katz JN, Brookhart MA, Stürmer T, Stedman MR, Solomon DH.  Relative effectiveness of osteoporosis drugs for preventing nonvertebral fracture. Ann Intern Med. 2008; 148:637-46.
 
Laupacis A, Mamdani M.  Observational studies of treatment effectiveness: some cautions [Editorial]. Ann Intern Med. 2004; 140:923-4. PubMed
 
Haro JM, Kontodimas S, Negrin MA, Ratcliffe M, Suarez D, Windmeijer F.  Methodological aspects in the assessment of treatment effects in observational health outcomes studies. Appl Health Econ Health Policy. 2006; 5:11-25. PubMed
 
Temple R.  Problems in the use of large data sets to assess effectiveness. Int J Technol Assess Health Care. 1990; 6:211-9. PubMed
 
U.S. Food and Drug Administration.  Pediatric Drug Development. Accessed athttp://www.fda.gov/Cder/pediatric/index.htm#hist-refson 7 April 2008.
 

Figures

Tables

References

Epstein S.  Update of current therapeutic options for the treatment of postmenopausal osteoporosis. Clin Ther. 2006; 28:151-73. PubMed
CrossRef
 
U.S. Food and Drug Administration.  Federal Food, Drug, and Cosmetic Act. Accessed athttp://www.fda.gov/opacom/laws/fdcact/fdctoc.htmon 7 April 2008.
 
Temple R, Ellenberg SS.  Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 1: ethical and scientific issues. Ann Intern Med. 2000; 133:455-63. PubMed
 
Ellenberg SS, Temple R.  Placebo-controlled trials and active-control trials in the evaluation of new treatments. Part 2: practical issues and specific cases. Ann Intern Med. 2000; 133:464-70. PubMed
 
Ragi-Eis S, Zerbini CA, Provenza JR, Griz LH, de Gregório LH, Russo LA. et al.  Is it ethical to use placebos in osteoporosis trials? J Clin Densitom. 2006; 9:274-80. PubMed
 
Levine RJ.  Placebo controls in clinical trials of new therapies for osteoporosis. J Bone Miner Res. 2003; 18:1154-9. PubMed
 
Cadarette SM, Katz JN, Brookhart MA, Stürmer T, Stedman MR, Solomon DH.  Relative effectiveness of osteoporosis drugs for preventing nonvertebral fracture. Ann Intern Med. 2008; 148:637-46.
 
Laupacis A, Mamdani M.  Observational studies of treatment effectiveness: some cautions [Editorial]. Ann Intern Med. 2004; 140:923-4. PubMed
 
Haro JM, Kontodimas S, Negrin MA, Ratcliffe M, Suarez D, Windmeijer F.  Methodological aspects in the assessment of treatment effects in observational health outcomes studies. Appl Health Econ Health Policy. 2006; 5:11-25. PubMed
 
Temple R.  Problems in the use of large data sets to assess effectiveness. Int J Technol Assess Health Care. 1990; 6:211-9. PubMed
 
U.S. Food and Drug Administration.  Pediatric Drug Development. Accessed athttp://www.fda.gov/Cder/pediatric/index.htm#hist-refson 7 April 2008.
 

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)