Systematic Review: Reliability of Compendia Methods for Off-Label Oncology Indications Free

The U.S. Food and Drug Administration (FDA) approves drugs for specific uses, yet physicians routinely prescribe FDA-approved drugs for uses other than those for which they received approval (1); such uses are known as off-label indications. Off-label prescribing is common across medical disciplines; however, it is critical in oncology, in which effective treatment options are often limited, prognoses are grim, and submission of FDA applications for every combination of agent and cancer is impractical. In 1991, a U.S. General Accounting Office study (2) reported that up to 33% of all anticancer drug prescriptions were written for off-label indications. By 2005, the National Comprehensive Cancer Network estimated that 50% to 75% of all uses of cancer therapy were off-label (3).

The Social Security Act, section 1861(t)(2)(B)(ii)(I) and (II), within the Omnibus Budget Reconciliation Act of 1993 (4), stipulates the Medicare insurability of anticancer drugs and biologics for off-label uses. This statute recognizes certain compendia as authoritative sources for determining a “medically accepted indication” of drugs and biological agents, unless the Secretary of Health and Human Services determines otherwise. The statute originally designated 3 compendia: American Medical Association Drug Evaluations, American Hospital Formulary Service Drug Information(5), and United States Pharmacopeia Drug Information for the Health Professional(6). Although the statute pertained specifically to Centers for Medicare & Medicaid Services (CMS), most other payers and state legislatures have followed suit (7).

The list of approved compendia has shifted over the past 15 years. American Medical Association Drug Evaluations was discontinued, and United States Pharmacopeia Drug Information for the Health Professional subsumed its contents. United States Pharmacopeia Drug Information for the Health Professional was then discontinued in 2007, and its contents were rolled into a successor, DrugPoints. In 2008, CMS added Clinical Pharmacology, DRUGDEX, and National Comprehensive Cancer Network Drugs and Biologics Compendium to its list of approved compendia, bringing the total to 5 (8).

In response to concerns about the influence of compendia, CMS proposed various changes, including review of currently approved compendia, additional compendia approval, and an annual review process. To inform policy discussions, they commissioned the Agency for Healthcare Research and Quality to sponsor our project exploring the extent to which compendia provide comprehensive, evidence-based, and timely information for guiding off-label prescribing of cancer drugs.

Our study had 4 components: comparative descriptions of each compendium's stated methods for including new, off-label indications of FDA-approved drugs; systematic literature review for 14 selected off-label indications in 2006; an updated comparison for 1 indication in 2008; and analysis of each compendium's content and citations against their stated methods and the evidence identified in our systematic reviews.

To better understand how compendia develop their content, we compared 6 compendia that we chose through discussions with 10 oncology pharmacists and oncologists at Duke University and Tufts Medical Centers. We confirmed the list through discussions with the Agency for Healthcare Research and Quality, CMS, and financial officers at Duke University Medical Center. Our final compendia list included American Hospital Formulary Service Drug Information(5), United States Pharmacopeia Drug Information for the Health Professional(6), DRUGDEX Information System (United States Pharmacopeia Drug Information for the Health Professional and DRUGDEX, both available through Thomson Reuters) (9), Drug Facts and Comparisons(10), National Comprehensive Cancer Network Drugs and Biologics Compendium(11), and Clinical Pharmacology(12). This list includes the compendia approved by CMS as well as 2 nonapproved compendia. These compendia list more than 40 000 drugs, and their editors are generally pharmacologists with training in research and evidence synthesis.

In 2006, we gathered information on each compendium's methods by abstracting descriptive information from publicly available sources, conducting a 1-hour telephone interview with a senior editor from each compendium, and allowing the editors to respond to a methods table that summarized the information collected in the interview. We developed a list of evaluation criteria which we used to compare published methods and guide interviews. We then compared each compendium's methods with evaluation criteria and summarized the results. In 2008, we repeated the data abstraction to determine whether the compendia had updated their publicized methods. We did not include DrugPoints in the 2008 review because it differed structurally from the other compendia and could not be directly compared.

We selected off-label indications for 14 agent and cancer combinations that included both newer and older agents, common and rare types of cancer, and biologics and drugs. In 2006, we systematically searched MEDLINE, the Cochrane Central Register of Controlled Trials, and American Society of Clinical Oncology annual meeting abstracts (2004 and 2005) to identify English-language studies in humans. We included prospective clinical trials (phases I to III), case reports, and retrospective case series that reported tumor response, survival, quality of life, symptoms, and adverse effects. We excluded narrative reviews and studies that described only predictors of response, pharmacokinetics, or nonhuman results.

For eligible studies, 1 reviewer abstracted data into evidence tables and a second reviewer verified the completed tables. For each agent and cancer combination, we created a summary table that listed the number of articles identified by study design and key outcomes. We scored identified studies by design (for example, phase I to III), size, and outcomes reported (for example, tumor response, survival, quality of life, or adverse effects). Our intention was not to expose the compendia—which perform a vital function in oncology—but rather to examine their methods and the resulting comprehensiveness of their contents. We therefore did not score the evidence included in the compendia according to classic quality criteria. Instead, we adopted a conservative approach to content evaluation that entailed a simple count of independent studies, presentations of data, and other metrics but did not include evaluation of the quality and validity of the studies themselves. We used study design as a proxy for quality and validity, with phase III studies considered generally more valid than phase I or II studies.

In 2008, we repeated the systematic review for a single agent and cancer combination (gemcitabine for bladder cancer) by using the same methods, except we sought 2006 and 2007 conference abstracts. We chose the gemcitabine for bladder cancer indication because gemcitabine has been FDA-approved for longer than the other reviewed agents; gemcitabine has been widely studied (with the largest number of citations in our 2006 review); and, of the 3 gemcitabine combinations included, gemcitabine for bladder cancer had the fastest-evolving evidence base. This combination thus afforded compendia the most opportunity to demonstrate improvement.

In 2006, we abstracted data for each of the 14 agent and cancer combinations from each of the 6 compendia. We evaluated all available versions (print and electronic) of the 6 compendia in 2006 but recorded data from the most current and complete version; in all cases, this was an electronic version. Data included whether the indication was explicitly stated; how the indication was graded; and comments on further refinement regarding the stage of cancer, treatment timing, route of administration, and use of monotherapy or combination therapy. We recorded outcomes mentioned specifically for the off-label use; toxicity data; presence of citations to evidence specifically regarding the off-label indication; number, identity, and years of citations; and time since any updates to the monograph or entry. We stratified publications cited in the compendia by study design, noting abstracts separately, and tabulated whether each compendium cited each publication; we compared the results with those of our systematic review. We updated this process in 2008 for the gemcitabine for bladder cancer indication.

The CMS funded the initial 2006 report through the Agency for Healthcare Research and Quality's Evidence-Based Practice Center program. These agencies participated in formulating questions, reviewing the list of agent and cancer combinations, and commenting on report findings. The 2008 update of the report was unfunded. The agencies had no role in the writing or approval of the manuscript.

All compendia described their scope and editorial policies, which did not change between the 2006 and 2008 reviews (Appendix Table 1). Because United States Pharmacopeia Drug Information for the Health Professional was discontinued in 2007, our results do not include that compendium even though it was part of the 2006 review. Here, we focus our evaluation on frequency of updates, use of available evidence, and transparency of decision-making processes.

Compendia had different update cycles for electronic versions, which varied from daily to quarterly (Table 1). Several compendia published multiple electronic editions; these differed in content and update schedules. Compendia did not always provide revision dates for drug monographs, and when provided, they did not clearly indicate which content was revised. All compendia included off-label indications (Table 2).

Use of evidence to evaluate off-label indications also differed across compendia (Appendix Table 1). In evaluating the quality of evidence, each compendium stated that they considered validity assessment to be a component of its editorial process. However, these practices were generally not provided in published material. Not all compendia explicitly linked recommendations to supporting evidence through specific citations. Methods for citing and updating evidence varied across compendia. In interviews, each of the editors noted that they either had made, or planned to make, changes in editorial policies to become more transparent with regard to the use of evidence in their evaluations; these changes were not evident in our 2008 review of the electronic compendia.

Compendia policies with respect to inclusion of off-label indications varied in consistency and transparency. According to the compendia editors, the decision to recommend a non-FDA approved indication required a judgment by editorial staff regarding the quality and quantity of evidence and the magnitude of benefit versus harm. Several editors mentioned that a high degree of interest or evidence of clinical use would also be considered when deciding whether to include an off-label listing. Editorial discretion was used particularly in the case of equivocal indications; in general, editors reported favoring remaining silent rather than listing an equivocal indication.

We retrieved citations in 2006 from a literature search for the off-label use of the following agent and cancer combinations: bevacizumab for breast and lung cancer (subsequently FDA-approved for breast cancer in February 2008); oxaliplatin for breast and lung cancer; irinotecan for breast cancer; docetaxel for esophageal, gastric, and ovarian cancer; gemcitabine for biliary tract, bladder, and ovarian cancer; rituximab for chronic lymphocytic leukemia; and erlotinib for head, neck, and pancreatic cancer.

Our search identified 1314 citations for all 14 combinations. We identified an additional 18 unique citations from the Cochrane Central Register of Controlled Trials and 179 abstracts from the American Society of Clinical Oncology Web site. We sorted the studies that met our systematic review criteria by study design and report type: prospective published report (phase I to II, II, or III), retrospective series, case report, and abstract. Our original report (13) lists the complete search results. The systematic review that we conducted in 2006 on gemcitabine for bladder cancer identified 1 phase III studies (2 publications [14, 15]), 28 phase II studies, 14 phase I to II studies, and 3 case report or retrospective series published from 1994 to 2005 and 15 conference abstracts presented at the 2004 or 2005 American Society of Clinical Oncology annual meetings.

In 2008, we repeated the systematic review on gemcitabine for bladder cancer. Published phase I to III studies increased by 25, from 43 in the 2006 review to 68 in 2008 (22 additional phase II studies and 3 phase I to II studies), and 4 case reports or retrospective studies were found. Three conference abstracts were presented at the 2006 and 2007 American Society of Clinical Oncology annual meetings. The phase III study we identified in our original systematic review had a new published report to update findings (16).

The compendia varied substantially in whether they listed any given agent and cancer combination (Table 3). For example, of the 14 off-label indications, only 1 compendium included all 14 combinations (DRUGDEX), whereas the others included 2 to 9. The only 2 indications discussed by all compendia were gemcitabine for bladder cancer and gemcitabine for ovarian cancer. When we repeated this tabulation in 2008, we found 1 change in the listing of off-label indications since the 2006 evaluation: Clinical Pharmacology added bevacizumab for breast cancer (which is now an FDA-approved indication).

The compendia also varied in how they addressed aspects of prescribing and assessment of outcomes, including stage of cancer during which the agent should be prescribed, treatment order (first line or other), uses of the agent (monotherapy or combination), comparator discussed (placebo, standard therapy, or other), toxicity, outcomes, and method of administration. Moreover, the wording used by the compendia to provide recommendations varied, and several compendia used different terminology or approaches internally for different agent and cancer combinations. For example, the National Comprehensive Cancer Network Drugs and Biologics Compendium recommended gemcitabine plus cisplatin as a standard neoadjuvant, adjuvant, and metastatic treatment for bladder cancer but did not recommend gemcitabine monotherapy, whereas other compendia indicated gemcitabine as monotherapy or combined with other agents for metastatic bladder cancer. Appendix Table 2 presents a comparative description of compendia monographs of gemcitabine for bladder cancer; we generated compendia comparisons for each of the 14 agent and cancer combinations in 2006, with similar results.

The compendia referenced different literature for the same indications, and within each compendium, the cited literature diverged from the literature retrieved through our systematic review. The example of gemcitabine for bladder cancer illustrates this discrepancy. In 2006, our review identified 43 published phase I to III studies and 15 conference abstracts; compendia varied in their use of these citations, referencing between 0 and 7. In 2008, our review identified an additional 25 reports published between 2006 and 2007. In this period, 1 compendium (DRUGDEX) increased from 3 to 11 citations, whereas the others had little or no change. All references cited by compendia that met our systematic review eligibility criteria were also identified by our systematic review.

Compendia did not necessarily cite the most up-to-date evidence to support recommendations. Despite documentation indicating that the compendia were last updated in 2008, they all cited literature from 2001 or earlier, except DRUGDEX, which included citations through 2007. Two compendia included conference abstracts from 1995 and 1996; only 1 (DRUGDEX) updated its monograph with the citation of the published report. The best evidence for efficacy identified in the 2006 review resided in a single phase III trial (14), which was cited by all compendia except Drug Facts and Comparisons. No compendium had cited the long-term follow-up report of the phase III trial published in 2005 (15) at the time of our 2006 review. Our 2008 systematic review update yielded an additional follow-up report for this trial (16); only DRUGDEX had incorporated either of these follow-up reports into its monograph. Personal comments and narrative reviews were cited as supporting evidence by 1 compendium (American Hospital Formulary Service Drug Information). Drug Facts and Comparisons did not have any references.

Compendia play an integral role in oncology practice. They serve as a mechanism for ensuring that patients have access to the newest, most effective registered drugs when evidence becomes available to support specific off-label indications. For off-label prescribing in oncology, legislation dating to 1993 has conferred disproportionate weight to the recommendations provided in a few specified compendia. These compendia have essentially functioned as gatekeepers; it is critical to examine their processes for evidence review, the reliability of the evidence they include, their decision-making pathways for adding new drugs, and their practices with regard to timely updating. Our review establishes a baseline against which to compare the performance of included and additional compendia in the future.

According to their stated methods, compendia use evidence as a critical gauge for including and recommending off-label anticancer indications. We found little agreement between the results of systematic reviews of 14 off-label indications of cancer drugs and the evidence cited for those indications in these commonly used compendia. Cited evidence was scanty and inconsistent across compendia, which raises questions about the processes by which evidence is identified and selected to generate recommendations, the potential biases or conflicts of interest that affect decisions of whether to include an indication or how to present the evidence, and the comprehensiveness and quality of the evidence that the compendia include.

The evidence included in the compendia we evaluated did not seem to be updated in a timely, regular, and explicit manner. We confirmed this observation with the 2008 update of 1 agent and cancer combination, which provided a case-in-point demonstration that the compendia did not follow their stated policies with regard to update cycles. This finding casts doubt on the compendia's adherence to other stated policies. Whether updated information would have changed recommendations is outside the scope of this review but is an important area for future consideration.

The compendia varied in numerous ways, including which off-label indications they included, what they considered evidence, the level of detail each provided regarding use of the agent, and how the evidence was linked to the recommendation and referenced. The primary area of general uniformity across compendia was in discussions about adverse effects.

In addition to the limited number of research studies cited, the citations were often neither the most recent nor derived from the highest available level of evidence. All compendia lacked explicit, systematic procedures for determining inclusion of off-label indications, and stated conditions for including non-FDA indications did not match actual practices of inclusion. A lack of transparency regarding inclusion policies obscured whether missing indications had been identified but excluded or were never identified. If indications were excluded, the rationale for exclusion was rarely provided, which again raises the possibility of bias or conflict of interest. A current study is exploring the potential for conflict of interest in editorial policies, expert panels, and evidence review at 4 compendia.

The handling of equivocal information was particularly divergent across compendia. Editors mentioned in interviews that they exercised discretion in decisions regarding equivocal evidence. In many cases, the editorial decision would be to remain silent rather than to list an indication that would be qualified as equivocal. Here, expert evaluation of the existing evidence takes on critical importance. As with any decision to include or omit an off-label indication, silence constitutes a powerful decision in itself but is difficult to interpret; it may mean that the evidence was not deemed sufficiently strong to justify recommendation or that other considerations were involved. Because drugs for unlisted indications are unlikely to be prescribed, the absence of an indication from a compendium's listing may strongly influence clinical practice.

Our study has limitations. We assessed a limited number of agent and cancer combinations—14 in 2006 and 1 in both 2006 and 2008. We included only off-label indications for cancer drugs and cannot claim generalizability to noncancer drugs or indications. We did not thoroughly evaluate identified studies in terms of magnitude of effects or methodological quality. Similarly, we did not consider the accuracy of compendia conclusions. We could not ascertain whether the relatively small number of studies referenced by the compendia accurately reflected the evidence on which they based their recommendations. We evaluated only current listings at 2 time points (2006 and 2008). Our methods did not permit us to ascertain whether silence about certain indications reflected a withdrawn off-label indication listing, indicating updated review, or one that had never been listed—and we could not determine the reason for silence about an indication.

Generalizability of findings is a valid concern. We selected 14 combinations that reflected newer and older drugs, on and off patent, that treat a range of tumor types, but all are being used in areas with frequent requests to CMS for reimbursement of off-label indications. Although we cannot determine the generalizability of our findings to other agent and disease combinations or to other disease areas, one might expect the compendia's performance to be highest in oncology, given the importance of compendia in cancer drug reimbursement. We believe it would be useful for future studies to evaluate changes in inclusion of these same agent and disease combinations over time, to assess for improvement in compendia content.

Although our study was not intended to develop recommendations, it does raise certain pivotal questions: In their current state, can we rely on the compendia as authoritative, comprehensive, and timely sources of information on off-label indications in oncology? If we improve the transparency and rigor of compendia processes, such as through more explicit practices and timely updates, can we expect an improvement in the quality and completeness of compendia content? In other words, is this a good and serviceable system which, although imperfectly implemented, could be improved? Alternatively, is the challenge of near-continuous systematic review of large numbers of indications a colossal task that would be unmanageable for any compendium publisher? With at least 6 compendia attending to off-label oncology indications, is there unnecessary duplication and waste of scarce systematic review capabilities when all compendia are expected to fully review all topics? If we accept this task as unrealistic for all compendia to achieve simultaneously, might another party—perhaps a government agency—have the capacity to manage it? No such government agency currently exists. The FDA is neither equipped nor authorized to provide the up-to-date, rigorous, and comprehensive review that is being expected of the compendia. Of note, the compendia's role in listing off-label indications largely came about because of lags in the FDA review and approval process. All discussions of process improvement or alternate scenarios must retain a focus on the goal: to ensure that patients have access to the most recent, evidence-based, effective, and safe treatments. Although important in any medical context, the timeliness of access to the latest evidence is most critical in oncology, in which dire prognoses, short windows in which to treat, the high expense of many drugs, and corollary acute emotional distress make medical decision making highly charged.

Compendia are, in many ways, like clinical practice guidelines in that they examine the benefits and harms of pharmaceutical agents and present clinical conclusions based on an assessment of net benefit. Our review establishes a baseline for the methods and content of the compendia used in oncology. With this baseline, we can evaluate compendia in the future by comparing the proportion of qualifying evidence cited in included compendia in 2006 with that proportion at future time points, thereby benchmarking improvement. In the context of an annual review of the approved compendia, newly announced by CMS, this sort of tracking of compendia improvement could inform discussions regarding which compendia to retain in, or add to, the list of approved sources for coverage decisions. Evaluation of compendia should consider not only the volume of available evidence that they encompass but also their consistency, quality, transparency, and timeliness.