0
Challenges of Summarizing Better Information for Better Health: The Evidence-based Practice Center Experience |

Challenges in Systematic Reviews of Complementary and Alternative Medicine Topics FREE

Paul G. Shekelle, MD, PhD; Sally C. Morton, PhD; Marika J. Suttorp, MS; Nina Buscemi, PhD; and Carol Friesen, MA, MLIS
[+] Article and Author Information

From the Southern California Evidence-based Practice Center (RAND Corporation, Santa Monica, California; the Greater Los Angeles Veterans Affairs Healthcare System, Los Angeles, California; and the University of Alberta Evidence-based Practice Center, Edmonton, Alberta, Canada).


Disclaimer: The authors of this article are responsible for its contents. No statement in this article should be construed as an official position of the Agency for Healthcare Research and Quality, the National Center for Complementary and Alternative Medicine, the Office of Dietary Supplements, the National Institutes of Health, or the U.S. Department of Health and Human Services.

Acknowledgments: The authors thank Dr. Terry Klassen for feedback on the manuscript, Di Valentine and Cony Rolon for their assistance, and Marilyn Josefsson for administrative support.

Grant Support: This research was performed by the Southern California Evidence-based Practice Center based at RAND, Santa Monica, California, with assistance from the University of Alberta Evidence-based Practice Center at Edmonton, Alberta, Canada, under contract with the Agency for Healthcare Research and Quality (contract 290-02-0003) and is based on work originally supported by the National Center for Complementary and Alternative Medicine and the Office of Dietary Supplements, both at the National Institutes of Health. Dr. Shekelle was a Senior Research Associate of the Veterans Affairs Health Services Research and Development Service.

Potential Financial Conflicts of Interest: Authors of this paper have received funding for Evidence-based Practice Center reports.

Requests for Single Reprints: Paul G. Shekelle, MD, PhD, RAND Corporation, 1776 Main Street, Santa Monica, CA 90401; e-mail, Paul_Shekelle@rand.org.

Current Author Addresses: Dr. Shekelle and Ms. Suttorp: RAND Corporation, 1776 Main Street, Santa Monica, CA 90401.

Dr. Morton: RTI International, 3040 Cornwallis Road, PO Box 12194, Research Triangle Park, NC 27709-2194.

Dr. Buscemi and Ms. Friesen: University of Alberta Evidence-based Practice Center, 11402 University Avenue, Room 9420, Edmonton, Alberta T6G 2J3, Canada.


Ann Intern Med. 2005;142(12_Part_2):1042-1047. doi:10.7326/0003-4819-142-12_Part_2-200506211-00003
Text Size: A A A

The use of complementary and alternative medicine (CAM) continues to grow in the United States. The Agency for Healthcare Research and Quality has devoted a substantial proportion of the Evidence-based Practice Center (EPC) program to systematic reviews of CAM. Such syntheses present different challenges from those conducted on western medicine topics, and in many ways are more difficult. We discuss 3 challenges: identifying evidence about CAM, assessing the quality of individual studies, and addressing rare serious adverse events. We use illustrations from EPC evidence reports to show readers approaches to the 3 areas and then present specific recommendations for each issue.

The popularity of complementary and alternative medicine (CAM) in the United States continues to grow (1). The Agency for Healthcare Research and Quality (AHRQ) has devoted a substantial proportion of the Evidence-based Practice Center (EPC) program to reports on CAM. As of 2004, 19 of the more than 100 evidence reports concerned CAM interventions. The reports spanned a wide scope of topics, including botanicals and herbal supplements (such as garlic and ephedra), traditional medicine (such as acupuncture and ayurvedic medicine), and vitamins (such as supplementation).

In preparing CAM reports, EPC reviewers faced several challenges. In this paper, we discuss approaches used to address the following 3 issues: 1) identifying evidence about CAM interventions, 2) assessing the quality of individual studies, and 3) addressing rare serious adverse events. We then suggest recommendations in each area for future reviews of CAM interventions.

The many biases in the publication and indexing of CAM research pose a challenge for locating literature. Publication bias refers to the tendency of investigators, reviewers, and editors to submit or accept manuscripts on the basis of the strength or direction of the findings. While publication bias is a concern in conventional medical research (24), in CAM research the issue is particularly complicated. Most studies published in leading CAM journals have positive results (5). Some countries, such as China, Japan, Russia, and Taiwan, publish more studies with positive results than studies with negative results; this imbalance may reflect publication bias (6).

Language-related publication bias also exists in CAM research, as it does for conventional medicine research (7). Negative CAM findings are more likely to be published in mainstream medical journals (that is, English-language journals), while positive CAM findings are more likely to be published in CAM journals (which tend to be non–English-language journals) (810). Thus, all else being equal, the exclusion of non–English-language reports may result in a lower estimate of intervention effect for CAM interventions than would be seen if these reports were included (10). The direction of bias, however, may depend on the CAM topic. Studies about some CAM therapies may initially be published only in non–English-language journals (8). On the other hand, in pediatric CAM, some evidence suggests that randomized, controlled trials (RCTs) tend to be published in English, and often in mainstream medical journals (9).

Another bias that hinders access is the incomplete or improper indexing of CAM journals and articles by mainstream databases such as MEDLINE (1112). For example, MEDLINE indexed only 10% of CAM journals identified worldwide by the National Center for Complementary and Alternative Medicine and the National Library of Medicine (12), while approximately 35% of all biomedical journals published worldwide are indexed in MEDLINE (13). Inconsistent use of keywords, descriptors, and subject headings, along with differing indexing procedures across databases, may also pose a challenge in locating CAM literature (14). The inconsistency in the terminology may partially explain why a MEDLINE search of the term alternative medicine does not capture all studies relevant to the CAM field (12). We examined the 19 evidence reports and 2 technology assessments related to CAM for sources and methods used to locate CAM literature (Appendix Table). For almost all reports, investigators searched specialized CAM databases in addition to mainstream databases. At least 10 databases specialize in CAM: Allied and Complementary Medicine Database (AMED); Cochrane Complementary Medicine Trials Registry; Alt HealthWatch; HerbMed; Manual, Alternative and Natural Therapy Index System (MANTIS); Natural Products Alert (NAPRALERT); International Bibliographic Information on Dietary Supplements (IBIDS); Hom-Inform; The Arthritis and Complementary Medicine Database (ARCAM); and Complementary and Alternative Medicine and Pain Database (CAMPAIN). Many can be accessed without a subscription. The search strategy used for most reports did not involve English-language restrictions. For almost all reports, investigators included an extensive list of keywords to identify articles that may not be indexed by standard subject headings, or for which terminology was inconsistent (14).

Only a few investigators used hand searching (15). The searching of grey literature is another method that slightly more than half of the investigators used to address publication bias. Grey literature was searched in about half the reports. For most reports, investigators specified that they had reviewed reference lists to identify articles improperly indexed by mainstream databases. However, investigators should be aware of the possibility of citation bias, a phenomenon whereby studies supportive of a beneficial effect are cited more frequently (16). There is evidence of this bias in mainstream medical literature (16), and it could be relevant to CAM. Similarly, for most reports, investigators specified that they had communicated with authors or experts for additional data and relevant studies. However, investigators should be aware of the possibility of bias in the provision of information, such that studies with positive results may be more likely to be communicated to investigators than are studies with negative results (7).

Complementary and alternative medicine presents special challenges in the design and execution of studies, with respect to both internal validity and generalizability. These problems relate to the tension between specifying the intervention sufficiently that others can apply it and the desire to study CAM therapy as it is applied in CAM practices; they also concern the difficulties in controlling expectation bias (the systematic effect on the results of the participants' belief that a certain therapy will help them). Most CAM interventions are investigated only after they are so widespread that they can no longer be ignored, and by that time, the CAM practices are highly diversified in practices, personal experiences, biases, and expectations.

For some CAM therapies, specifying the intervention sufficiently is not a great problem, at least in theory. The use of herbs or dietary supplements such as St. John's wort, antioxidant supplements, glucosamine, and saw palmetto can be specified in a manner analogous to that used in drug trials, with a standard concentration or formula given regularly. (However, there is concern about the variation in formulation and bioactivity of some supplements from lot to lot, and this variation may be of even greater concern for botanicals in which the active ingredient is not known.)

For many other CAM therapies, however, the conceptual basis for the therapy requires an interaction between the practitioner and patient that modifies the therapy to the individual. Traditional Chinese acupuncture, spinal manipulation by chiropractors, and ayurvedic medicine require individualization of treatment based on an examination and understanding of the patient's condition using concepts that do not have an analogue in western allopathic medicine. Consequently, CAM advocates have criticized randomized clinical trials that reported no effect for not having allowed the necessary tailoring of the intervention. For this reason, “pragmatic” trials are frequently advocated for studying CAM.

In a pragmatic trial, patients are assigned to a CAM practitioner rather than a tightly specified CAM therapy. The CAM practitioners can provide their treatments in their usual fashion, individualizing the therapy for each particular patient. While this strategy allows the CAM practice to occur in its traditional fashion, it makes blinding or otherwise controlling expectation bias very difficult. Furthermore, while in one way individualizing the therapy increases generalizability, it also increases the sensitivity of the results to the skill of the practitioners. Since the intervention relies on practitioner expertise in understanding the patient and delivering the therapy, the study results are more difficult to apply to other practitioners. Thus, pragmatic trials should discuss the training and experience of the CAM practitioner. Large pragmatic trials that include many practitioners and that compare a CAM therapy with a credible control or alternative therapy would be particularly useful in assessing CAM.

The traditional way to manage expectation bias in drug trials is to use a matched placebo and a double-blind design. The magnitude of a potential “placebo response,” or even its existence at all, is controversial (1720). However, there is agreement that the placebo response is particularly important in studies that use subjective patient measures of outcome, especially pain. Because chronic painful conditions account for a large proportion of the reasons patients seek CAM therapies, the use of placebo controls is especially important. Even medical procedures such as knee surgery have been shown to be amenable to studies with a sham control (21), so the issue of placebo and shams spans the gamut of CAM therapies. If CAM treatments, such as herbs and dietary supplements, are amenable to a matched-placebo, double-blind design, anything less is a limitation.

For many other CAM therapies, however, placebos and blinding are a much greater challenge. The use of sham procedures, rather than placebos, is possible when the CAM therapy being evaluated is well-specified. One mechanism for assessing the success of the “blinding” ability of a sham therapy is to survey patients after receipt of the therapy and ask them to guess whether they received active or sham therapy. Similar responses in patients receiving active and sham therapy are good evidence that the sham was successful in controlling expectation bias. Successful sham therapies have been reported for acupuncture and spinal manipulation (2224).

Pragmatic trials, of course, do not compare CAM therapies with sham treatment. However, a patient's expectation of how likely it is that a therapy will be beneficial for their condition can substantially influence the results. In a study comparing massage, acupuncture, and usual care for patients with chronic low back pain, patients were asked their expectation before treatment. When patients were randomly assigned to treatments they thought more likely to be effective, they in fact reported greater efficacy (25). In a large pragmatic trial comparing chiropractic care with physical therapy, benefit of chiropractic care was much more pronounced in patients who originally presented to a chiropractor's office and were then randomly assigned to chiropractic care than it was in patients who initially presented to physical therapy and were then randomly assigned to chiropractic care (26). While blinding of patients or providers is clearly not possible in a pragmatic trial, it is possible to use outcome assessors who are blind to treatment. If feasible, such use should be considered a methodologic strength.

In sum, this variability in sham therapies across studies presents a challenge for a systematic review. If sham groups do vary in important ways, they cannot all be equated and treated as generic control groups in either narrative reviews or meta-analyses. The investigator must examine each sham treatment thoroughly, just as active-treatment groups are assessed. Categories of sham therapies may need to be constructed and each study classified. This categorization may result in a loss of sample size, since fewer studies will be comparable. Investigators should consider choosing the active treatment of interest, such as spinal manipulation, as the reference group, and then constructing effect sizes for alternative therapies versus this reference (27).

Although most CAM therapies are presumed to be safe, there is still the problem of how to assess and quantify the possibility of very rare, but potentially devastating, adverse events. Two examples illustrate this. Spinal manipulation given by U.S. chiropractors is performed tens of millions of times per year. A handful of reports describe devastating events, including death, stroke, and the cauda equina syndrome, after treatment with chiropractic spinal manipulation. Similarly, before it was removed from the U.S. market by the U.S. Food and Drug Administration (FDA), millions of portions of the herb ephedra were consumed. The FDA MedWatch program has received reports of several hundred very serious events (including death, heart attack, and stroke) that followed the consumption of ephedra (28). How are we to assess whether there are causal relationships between the use of spinal manipulation or the consumption of ephedra and these rare but serious events? If the benefit for any CAM therapy is modest or unproven, then the presence of even a very small increased risk for a serious event is enough to tip the scales against use of the CAM therapy.

Because these are rare events, RCT data will almost never be sufficient to prove or disprove a causal relationship between a CAM therapy and a rare adverse event. One of the best-studied CAM therapies is spinal manipulation. At this time, 38 RCTs have been published, with a total of 2302 patients who received spinal manipulation by itself or in addition to another treatment (27). No RCT reported a serious adverse event, and the upper bound of a one-sided 95% CI for the event rate with all patients combined is 0.13%. Similarly, at this time 52 RCTs of ephedra or purified ephedrine could be included in an adverse-event analysis (28). Again, none of these studies reported a serious adverse event, and the upper bound of the CI is 0.18%. Indeed, for a future randomized trial to have sufficient power to assess adverse events occurring at a rate of 1 per million, we calculate that such a study would need to enroll about 3 million treated patients. Clearly, this threshold will not be crossed for most CAM interventions, or any other interventions for that matter.

With RCT data insufficient to draw conclusions, the next place to look for evidence is hypothesis-testing observational studies. The case–control study is the traditional epidemiologic tool for evaluating possible relationships between exposure and rare events, such as the relationship between smoking and lung cancer or between phenylpropanolamine and stroke (29). Indeed, 1 case–control study concluded that case-patients were 5 times more likely than controls to have visited a chiropractor in the week before a vertebrobasilar accident (30). Unfortunately, most CAM therapies have not been the subject of case–control studies.

A third source of potential evidence is large registries of people who have received the intervention. Such postmarketing surveillance has produced sufficient evidence to warrant the removal of numerous drugs (such as troglitazone [31]) because of many unusual side effects, such as liver failure. Examination of large administrative databases is another way to assess a potential relationship between an intervention and an outcome. However, this type of study requires that the intervention be coded in administrative databases, and for many CAM therapies this is unlikely (or will underrepresent the actual delivery of the procedure) since only reimbursed interventions are coded for in administrative databases. This technique has been used for spinal manipulation but would not be useful for herbs or dietary supplements that are available over the counter.

The last form of evidence is the case report. In our ephedra analysis, we examined 66 case reports or series found in the published literature, 1474 reports submitted to MedWatch, and 18 502 reports submitted to Metabolife, a manufacturer of an ephedra dietary supplement (32). This analysis had several potential limitations. First, we did not have access to all reported adverse event files, and in any event, many authorities consider MedWatch case reports to underestimate the number of events. Many of the reports did not contain all the data that we needed to make assessments. How these cases might have influenced our findings had they contained appropriate documentation is unknown. Another important limitation is that we do not have an estimate of the number of people using ephedra. In addition, the use of ephedra and ephedrine is increasing over time, as is the probability that someone will report an adverse event because of publicity. Finally, the most important limitation is that an assessment of case reports is generally insufficient to reach conclusions regarding causality. Even with these limitations, the FDA regularly uses biological plausibility and the baseline rate of events to infer causality.

In sum, the possibility of rare but serious adverse events needs to be considered for CAM therapies, since benefits are usually small or unproven; thus, even small risks may be decisive. Controlled trials are the best way to assess causality, but will almost never have studied sufficient patients to assess risks at a rate greater than 1 in 1000. Case–control or cohort studies can be searched for, but these are rare. The decision to include case reports must be made on a case-by-case basis. Additionally, reviewers should carefully note whether the inclusion and exclusion criteria for a trial included a medical evaluation. Such evaluations could identify, and exclude, persons at presumed increased risk for an adverse event. The generalizability of safety findings to a general population that may have access to the CAM therapy without a medical evaluation—such as an over-the-counter herbal product—needs to be considered.

The Table summarizes our recommendations. The EPC evidence reports of CAM topics incorporated many sources and methods for locating CAM literature. Investigators of these reviews commonly searched specialized databases, used an extensive list of keywords, and communicated with authors and experts; they rarely used hand searching. All of the approaches addressed one of the key biases inherent in the publication and indexing of CAM research, although in some cases, the approaches themselves may be prone to bias. Nonetheless, a review of CAM literature necessitates that all of these sources and methods be used to identify relevant literature, including the use of non–English-language reports for particular CAM topics. Research should be conducted to determine how the various sources and methods of locating CAM literature affect the estimates of treatment effectiveness in CAM systematic reviews, among various topics of the field. For example, how is the estimate of treatment effect of systematic reviews of various CAM topics affected, if at all, by the use of hand searching for identification of potentially relevant articles?

Table Jump PlaceholderTable.  Recommendations for Improving Syntheses of Complementary and Alternative Medicine Topics
Table Jump PlaceholderAppendix Table.  Types of Sources and Methods Used To Locate Complementary and Alternative Medicine Literature for Evidence Reports

Investigators should also clearly report how they avoided bias in selecting primary studies for inclusion in evidence reports, since evidence suggests that CAM systematic reviews tend to be weak in this respect (33). More research is required to determine the biases that may affect the identification of CAM literature, such as citation bias and bias in provision of information by authors and experts. In addition, more research should be conducted to determine whether the prevalence of biases differs across CAM topics and, therefore, whether the relative importance of various sources and methods for locating CAM literature differs across these topics.

In assessing the internal validity of CAM therapies, a reviewer should first decide whether the therapy in question is amenable to a placebo-controlled, double-blind design. If so, internal validity criteria similar to those used for drug trials should be assessed. If the CAM therapy is not amenable to placebo control, then the reviewer should determine whether the study assessed a defined CAM therapy; if so, a sham procedure could be used. In that case, the investigators should assess the sham, usually by asking patients via a survey to guess which therapy they received. Using this technique and reporting similar proportions of patients guessing that they had received active therapy provides good evidence that the sham was successful. Reporting unequal proportions, or failing to perform the survey at all, leaves the success of the sham in question.

In pragmatic trials, the reviewer should evaluate whether the investigators provided some information about the training of the CAM practitioners, such that the results of the study could be applied in other situations. In addition, pragmatic trials should determine the patient's expectations for therapy and consider whether this information might have influenced the treatment. Studies that lack such an assessment must necessarily be considered potentially more prone to expectation bias.

Finally, it is a challenge to identify evidence sufficient to prove or refute the existence of serious adverse events associated with CAM. Randomized trials will never be sufficient, and other hypothesis-testing studies are likely to be unavailable. Frequently, the review must rely on observational studies that lack a comparison group or case reports. Except in unusual circumstances, these types of evidence will not provide conclusive proof of the presence or absence of an elevated risk for an adverse event and at best can only provide a signal that a potential for increased risk exists.

Eisenberg DM, Davis RB, Ettner SL, Appel S, Wilkey S, Van Rompay M. et al.  Trends in alternative medicine use in the United States, 1990-1997: results of a follow-up national survey. JAMA. 1998; 280:1569-75. PubMed
CrossRef
 
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr.  Publication bias and clinical trials. Control Clin Trials. 1987; 8:343-53. PubMed
 
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A. et al.  Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993; 269:2749-53. PubMed
 
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR.  Publication bias in clinical research. Lancet. 1991; 337:867-72. PubMed
 
Ernst E, Pittler MH.  Alternative therapy bias [Letter]. Nature. 1997; 385:480. PubMed
 
Vickers A, Goyal N, Harland R, Rees R.  Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998; 19:159-66. PubMed
 
Egger M, Smith GD.  Bias in location and selection of studies. BMJ. 1998; 316:61-6. PubMed
 
Linde K, Ramirez G, Mulrow CD, Pauls A, Weidenhammer W, Melchart D.  St John's wort for depression—an overview and meta-analysis of randomised clinical trials. BMJ. 1996; 313:253-8. PubMed
 
Sampson M, Campbell K, Ajiferuke I, Moher D.  Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found? BMC Pediatr. 2003; 3:1. PubMed
 
Moher D, Pham B, Lawson ML, Klassen TP.  The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003; 7:1-90. PubMed
 
Kleijnen J, Knipschild P.  The comprehensiveness of Medline and Embase computer searches. Searches for controlled trials of homoeopathy, ascorbic acid for common cold and ginkgo biloba for cerebral insufficiency and intermittent claudication. Pharm Weekbl Sci. 1992; 14:316-20. PubMed
 
Ezzo J, Berman BM, Vickers AJ, Linde K.  Complementary medicine and the Cochrane Collaboration. JAMA. 1998; 280:1628-30. PubMed
 
U.S. National Library of Medicine. Fact Sheet: Response to Inquiries about Journal Selection for Indexing at NLM. U.S. Accessed athttp://www.nlm.nih.gov/pubs/factsheets/j_sel_faq.htmlon 28 April 2005.
 
Murphy LS, Reinsch S, Najm WI, Dickerson VM, Seffinger MA, Adams A, et al.  Searching biomedical databases on complementary medicine: the use of controlled vocabulary among authors, indexers and investigators. BMC Complement Altern Med. 2003; 3:3. PubMed
 
Alderson P, Green S, Higgins JP.  Handsearching. Cochrane Reviewer's Handbook 4.2.2. The Cochrane Library. Chichester, United Kingdom: J Wiley; 2004; Issue 1.
 
Ravnskov U.  Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992; 305:15-9. PubMed
 
Turner JA, Deyo RA, Loeser JD, Von Korff M, Fordyce WE.  The importance of placebo effects in pain treatment and research. JAMA. 1994; 271:1609-14. PubMed
 
Walsh BT, Seidman SN, Sysko R, Gould M.  Placebo response in studies of major depression: variable, substantial, and growing. JAMA. 2002; 287:1840-7. PubMed
 
Rochon PA, Binns MA, Litner JA, Litner GM, Fischbach MS, Eisenberg D. et al.  Are randomized control trial outcomes influenced by the inclusion of a placebo group?: A systematic review of nonsteroidal antiinflammatory drug trials for arthritis treatment. J Clin Epidemiol. 1999; 52:113-22. PubMed
 
Hrobjartsson A, Gotzsche PC.  Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. N Engl J Med. 2001; 344:1594-602. PubMed
 
Moseley JB, O'Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH. et al.  A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med. 2002; 347:81-8. PubMed
 
Sherman KJ, Hogeboom CJ, Cherkin DC, Deyo RA.  Description and validation of a noninvasive placebo acupuncture procedure. J Altern Complement Med. 2002; 8:11-9. PubMed
 
Shen J, Wenger N, Glaspy J, Hays RD, Albert PS, Choi C. et al.  Electroacupuncture for control of myeloablative chemotherapy-induced emesis: a randomized controlled trial. JAMA. 2000; 284:2755-61. PubMed
 
Triano JJ, McGregor M, Hondras MA, Brennan PC.  Manipulative therapy versus education programs in chronic low back pain. Spine. 1995; 20:948-55. PubMed
 
Kalauokalani D, Cherkin DC, Sherman KJ, Koepsell TD, Deyo RA.  Lessons from a trial of acupuncture and massage for low back pain: patient expectations and treatment effects. Spine. 2001; 26:1418-24. PubMed
 
Meade TW, Dyer S, Browne W, Townsend J, Frank AO.  Low back pain of mechanical origin: randomised comparison of chiropractic and hospital outpatient treatment. BMJ. 1990; 300:1431-7. PubMed
 
Assendelft WJ, Morton SC, Yu EI, Suttorp MJ, Shekelle PG.  Spinal manipulative therapy for low back pain. A meta-analysis of effectiveness relative to other therapies. Ann Intern Med. 2003; 138:871-81. PubMed
 
Shekelle PG, Hardy ML, Morton SC, Maglione M, Mojica WA, Suttorp MJ. et al.  Efficacy and safety of ephedra and ephedrine for weight loss and athletic performance: a meta-analysis. JAMA. 2003; 289:1537-45. PubMed
 
Kernan WN, Viscoli CM, Brass LM, Broderick JP, Brott T, Feldmann E. et al.  Phenylpropanolamine and the risk of hemorrhagic stroke. N Engl J Med. 2000; 343:1826-32. PubMed
 
Rothwell DM, Bondy SJ, Williams JI.  Chiropractic manipulation and stroke: a population-based case-control study. Stroke. 2001; 32:1054-60. PubMed
 
From the Food and Drug Administration.  Withdrawal of troglitazone and cisapride. JAMA. 2000; 283:2228. PubMed
 
Shekelle P, Hardy M, Morton SC, Maglione M, Suttorp M, Roth E, et al.  Ephedra and ephedrine for weight loss and athletic performance enhancement: clinical efficacy and side effects. Evidence Report/Technology Assessment No. 76 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001, task order no. 9). Rockville, MD: Agency for Healthcare Research and Quality; February 2003. AHRQ publication no. 03-E022.
 
Moher D, Soeken K, Sampson M, Ben-Porat L, Berman B.  Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002; 2:3. PubMed
 
Heidenreich PA, McDonald KM, Hastie T, Fadel B, Hagan V, Lee BK, et al.  An evaluation of beta-blockers, calcium antagonists, nitrates, and alternative therapies for stable angina. Evidence Report/Technology Assessment No. 10 (Prepared by University of California San Francisco-Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; November 1999. AHRQ publication no. 00-E003.
 
Mulrow C, Lawrence V, Ackermann R, Ramirez G, Morbidoni L, Aguilar C, et al.  Garlic: effects on cardiovascular risks and disease, protective effects against cancer, and clinical adverse effects. Evidence Report/Technology Assessment No. 20 (Prepared by San Antonio Evidence-based Practice Center and the Veterans Evidence-based Research, Dissemination, and Implementation Center, a Veterans Affairs Health Services Research and Development Center of Excellence, under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2000. AHRQ publication no. 01-E023.
 
Mulrow C, Lawrence V, Jacobs B, Dennehy C, Sapp J, Ramirez G, et al.  Milk thistle: effects on liver disease and cirrhosis and clinical adverse effects. Evidence Report/Technology Assessment No. 21 (Prepared by San Antonio Evidence-based Practice Center and the Veterans Evidence-based Research, Dissemination, and Implementation Center, a Veterans Affairs Services Research and Development Center of Excellence, under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2000. AHRQ publication no. 01-E025.
 
Hardy ML, Favreau JT, Elfenbaum PD, Morton SC, Roth EA, Genovese BJ, et al.  Mind-body interventions for gastrointestinal conditions. Evidence Report/Technology Assessment No. 40 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; July 2001. AHRQ publication no. 01-E030.
 
Hardy M, Coulter I, Venuturupalli S, Roth EA, Favreau J, Morton SC, et al.  Ayurvedic interventions for diabetes mellitus: a systematic review. Evidence Report/Technology Assessment No. 41 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; September 2001. AHRQ publication no. 01-E040.
 
Mulrow CD, Ramirez G, Cornell JE, Allsup K.  Defining and managing chronic fatigue syndrome. Evidence Report/Technology Assessment No. 42 (Prepared by San Antonio Evidence-based Practice Center under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2001. AHRQ publication no. 02-E001.
 
Coates VH, Turkelson CM, Chapell R, Bruening W, Mitchell MD, Reston JT, et al.  Diagnosis and treatment of worker-related musculoskeletal disorders of the upper extremity. Evidence Report/Technology Assessment No. 62 (Prepared by ECRI under contract 290-97-0020). Rockville, MD: Agency for Healthcare Research and Quality; December 2002. AHRQ publication no. 02-E038.
 
Hardy ML, Coulter I, Morton SC, Favreau J, Venuturupalli S, Chiappelli F, et al.  S-Adenosyl-L-methionine for treatment of depression, osteoarthritis, and liver disease. Evidence Report/Technology Assessment No. 64 (Prepared by Southern California Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; October 2002. AHRQ publication no. 02-E034.
 
Shekelle P, Coulter I, Hardy ML, Morton SC, Udani J, Spar M, et al.  Effect of the supplemental use of antioxidants vitamin C, vitamin E, and coenzyme Q10 for the prevention and treatment of cancer. Evidence Report/Technology Assessment No. 75 (Prepared by Southern California Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; August 2003. AHRQ publication no. 03-E047.
 
Shekelle P, Coulter I, Hardy ML, Morton SC, Favreau JT, Gagne J, et al.  Best-case series for the use of immuno-augmentation therapy and naltrexone for the treatment of cancer. Evidence Report/Technology Assessment No. 78 (Prepared by Southern California-RAND Evidence-based Practice Center under contract No 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; April 2003. AHRQ publication no. 03-E030.
 
Shekelle P, Morton S, Hardy M, Coulter I, Udani J, Spar M.  Effect of supplemental antioxidants vitamin C, vitamin E, and coenzyme Q10 for the prevention and treatment of cardiovascular disease. Evidence Report/Technology Assessment No. 83 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; July 2003. AHRQ publication no. 03-E043.
 
MacLean CH, Mojica WA, Morton SC, Pencharz J, Hasenfeld Garland R, Tu W, et al.  Effects of omega-3 fatty acids on lipids and glycemic control in type II diabetes and the metabolic syndrome and on inflammatory bowel disease, rheumatoid arthritis, renal disease, systemic lupus erythematosus, and osteoporosis. Evidence Report/Technology Assessment No. 89 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-02-0003). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E012-2.
 
Schachter H, Reisman J, Tran K, Dales B, Kourad K, Barnes D, et al.  Health effects of omega-3 fatty acids on asthma. Evidence Report/Technology Assessment No. 91 (Prepared by University of Ottawa Evidence-based Practice Center under contract 290-02-0021). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E013-2.
 
Jordan H, Matthan N, Chung M, Balk E, Chew P, Kupelnick B, et al.  Effects of omega-3 fatty acids on arrhythmogenic mechanisms in animal and isolated organ/cell culture studies. Evidence Report/Technology Assessment No. 92 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E011-2.
 
Balk E, Chung M, Lichtenstein A, Chew P, Kupelnick B, Lawrence A, et al.  Effects of omega-3 fatty acids on cardiovascular risk factors and intermediate markers of cardiovascular disease. Evidence Report/Technology Assessment No. 93 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E010-2.
 
Wang C, Chung M, Lichtenstein A, Balk E, Kupelnick B, DeVine D, et al.  Effects of omega-3 fatty acids on cardiovascular disease. Evidence Report/Technology Assessment No. 94 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E009-2.
 
Agency for Healthcare Research and Quality, Center for Practice and Technology Assessment. Acupuncture for fibromyalgia (technology assessment). Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.cms.hhs.gov/coverage/download/id83.pdfon 29 April 2005.
 
Agency for Healthcare Research and Quality. Acupuncture for osteoarthritis (technology assessment). Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.cms.hhs.gov/coverage/download/id84.pdfon 29 April 2005.
 
Ritenbaugh C, Streit K, Helfand M.  Routine vitamin supplementation to prevent cancer: summary of evidence from randomized controlled trials. Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.ahrq.gov/clinic/3rduspstf/vitamins/vitasum.htmon 29 April 2005.
 
Morris CD, Carson S.  Routine vitamin supplementation to prevent cardiovascular disease. A summary of the evidence for the U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.ahrq.gov/clinic/3rduspstf/vitamins/vitcvdsum.htmon 29 April 2005.
 
Last JM.  Gray literature. In: A Dictionary of Epidemiology. 4th ed. Oxford: Oxford Univ Pr; 2001.
 
Helmer D.  Grey literature. In: Etext on Health Technology Assessment (HTA) Information Resources. National Information Center on Health Services Research and Health Care Technology; 2004. Accessed athttp://www.nlm.nih.gov/nichsr/ehta/chapter10.htmlon 28 April 2005.
 
Hopewell S, McDonald S, Clarke M, Egger M.  Grey literature in meta-analyses of randomized trials of health care interventions (Cochrane Methodology Review). The Cochrane Database of Methodology Reviews. 2002; 4:Article MR000010. DOI: 10.1002/14651858.MR000010.
 
Hartling L, McAlister FA, Rowe BH, Ezekowitz J, Friesen C, Klassen TP.  Challenges in systematic reviews of therapeutic devices and procedures. Ann Intern Med. 2005; 142:1100-11.
 
Alderson P, Green S, Higgins JP.  Checking reference lists. In: Cochrane Reviewers' Handbook. Chichester, United Kingdom: J Wiley; 2004.
 

Figures

Tables

Table Jump PlaceholderTable.  Recommendations for Improving Syntheses of Complementary and Alternative Medicine Topics
Table Jump PlaceholderAppendix Table.  Types of Sources and Methods Used To Locate Complementary and Alternative Medicine Literature for Evidence Reports

References

Eisenberg DM, Davis RB, Ettner SL, Appel S, Wilkey S, Van Rompay M. et al.  Trends in alternative medicine use in the United States, 1990-1997: results of a follow-up national survey. JAMA. 1998; 280:1569-75. PubMed
CrossRef
 
Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr.  Publication bias and clinical trials. Control Clin Trials. 1987; 8:343-53. PubMed
 
Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A. et al.  Should unpublished data be included in meta-analyses? Current convictions and controversies. JAMA. 1993; 269:2749-53. PubMed
 
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR.  Publication bias in clinical research. Lancet. 1991; 337:867-72. PubMed
 
Ernst E, Pittler MH.  Alternative therapy bias [Letter]. Nature. 1997; 385:480. PubMed
 
Vickers A, Goyal N, Harland R, Rees R.  Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998; 19:159-66. PubMed
 
Egger M, Smith GD.  Bias in location and selection of studies. BMJ. 1998; 316:61-6. PubMed
 
Linde K, Ramirez G, Mulrow CD, Pauls A, Weidenhammer W, Melchart D.  St John's wort for depression—an overview and meta-analysis of randomised clinical trials. BMJ. 1996; 313:253-8. PubMed
 
Sampson M, Campbell K, Ajiferuke I, Moher D.  Randomized controlled trials in pediatric complementary and alternative medicine: where can they be found? BMC Pediatr. 2003; 3:1. PubMed
 
Moher D, Pham B, Lawson ML, Klassen TP.  The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health Technol Assess. 2003; 7:1-90. PubMed
 
Kleijnen J, Knipschild P.  The comprehensiveness of Medline and Embase computer searches. Searches for controlled trials of homoeopathy, ascorbic acid for common cold and ginkgo biloba for cerebral insufficiency and intermittent claudication. Pharm Weekbl Sci. 1992; 14:316-20. PubMed
 
Ezzo J, Berman BM, Vickers AJ, Linde K.  Complementary medicine and the Cochrane Collaboration. JAMA. 1998; 280:1628-30. PubMed
 
U.S. National Library of Medicine. Fact Sheet: Response to Inquiries about Journal Selection for Indexing at NLM. U.S. Accessed athttp://www.nlm.nih.gov/pubs/factsheets/j_sel_faq.htmlon 28 April 2005.
 
Murphy LS, Reinsch S, Najm WI, Dickerson VM, Seffinger MA, Adams A, et al.  Searching biomedical databases on complementary medicine: the use of controlled vocabulary among authors, indexers and investigators. BMC Complement Altern Med. 2003; 3:3. PubMed
 
Alderson P, Green S, Higgins JP.  Handsearching. Cochrane Reviewer's Handbook 4.2.2. The Cochrane Library. Chichester, United Kingdom: J Wiley; 2004; Issue 1.
 
Ravnskov U.  Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992; 305:15-9. PubMed
 
Turner JA, Deyo RA, Loeser JD, Von Korff M, Fordyce WE.  The importance of placebo effects in pain treatment and research. JAMA. 1994; 271:1609-14. PubMed
 
Walsh BT, Seidman SN, Sysko R, Gould M.  Placebo response in studies of major depression: variable, substantial, and growing. JAMA. 2002; 287:1840-7. PubMed
 
Rochon PA, Binns MA, Litner JA, Litner GM, Fischbach MS, Eisenberg D. et al.  Are randomized control trial outcomes influenced by the inclusion of a placebo group?: A systematic review of nonsteroidal antiinflammatory drug trials for arthritis treatment. J Clin Epidemiol. 1999; 52:113-22. PubMed
 
Hrobjartsson A, Gotzsche PC.  Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment. N Engl J Med. 2001; 344:1594-602. PubMed
 
Moseley JB, O'Malley K, Petersen NJ, Menke TJ, Brody BA, Kuykendall DH. et al.  A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med. 2002; 347:81-8. PubMed
 
Sherman KJ, Hogeboom CJ, Cherkin DC, Deyo RA.  Description and validation of a noninvasive placebo acupuncture procedure. J Altern Complement Med. 2002; 8:11-9. PubMed
 
Shen J, Wenger N, Glaspy J, Hays RD, Albert PS, Choi C. et al.  Electroacupuncture for control of myeloablative chemotherapy-induced emesis: a randomized controlled trial. JAMA. 2000; 284:2755-61. PubMed
 
Triano JJ, McGregor M, Hondras MA, Brennan PC.  Manipulative therapy versus education programs in chronic low back pain. Spine. 1995; 20:948-55. PubMed
 
Kalauokalani D, Cherkin DC, Sherman KJ, Koepsell TD, Deyo RA.  Lessons from a trial of acupuncture and massage for low back pain: patient expectations and treatment effects. Spine. 2001; 26:1418-24. PubMed
 
Meade TW, Dyer S, Browne W, Townsend J, Frank AO.  Low back pain of mechanical origin: randomised comparison of chiropractic and hospital outpatient treatment. BMJ. 1990; 300:1431-7. PubMed
 
Assendelft WJ, Morton SC, Yu EI, Suttorp MJ, Shekelle PG.  Spinal manipulative therapy for low back pain. A meta-analysis of effectiveness relative to other therapies. Ann Intern Med. 2003; 138:871-81. PubMed
 
Shekelle PG, Hardy ML, Morton SC, Maglione M, Mojica WA, Suttorp MJ. et al.  Efficacy and safety of ephedra and ephedrine for weight loss and athletic performance: a meta-analysis. JAMA. 2003; 289:1537-45. PubMed
 
Kernan WN, Viscoli CM, Brass LM, Broderick JP, Brott T, Feldmann E. et al.  Phenylpropanolamine and the risk of hemorrhagic stroke. N Engl J Med. 2000; 343:1826-32. PubMed
 
Rothwell DM, Bondy SJ, Williams JI.  Chiropractic manipulation and stroke: a population-based case-control study. Stroke. 2001; 32:1054-60. PubMed
 
From the Food and Drug Administration.  Withdrawal of troglitazone and cisapride. JAMA. 2000; 283:2228. PubMed
 
Shekelle P, Hardy M, Morton SC, Maglione M, Suttorp M, Roth E, et al.  Ephedra and ephedrine for weight loss and athletic performance enhancement: clinical efficacy and side effects. Evidence Report/Technology Assessment No. 76 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001, task order no. 9). Rockville, MD: Agency for Healthcare Research and Quality; February 2003. AHRQ publication no. 03-E022.
 
Moher D, Soeken K, Sampson M, Ben-Porat L, Berman B.  Assessing the quality of reports of systematic reviews in pediatric complementary and alternative medicine. BMC Pediatr. 2002; 2:3. PubMed
 
Heidenreich PA, McDonald KM, Hastie T, Fadel B, Hagan V, Lee BK, et al.  An evaluation of beta-blockers, calcium antagonists, nitrates, and alternative therapies for stable angina. Evidence Report/Technology Assessment No. 10 (Prepared by University of California San Francisco-Stanford Evidence-based Practice Center under contract 290-97-0013). Rockville, MD: Agency for Healthcare Research and Quality; November 1999. AHRQ publication no. 00-E003.
 
Mulrow C, Lawrence V, Ackermann R, Ramirez G, Morbidoni L, Aguilar C, et al.  Garlic: effects on cardiovascular risks and disease, protective effects against cancer, and clinical adverse effects. Evidence Report/Technology Assessment No. 20 (Prepared by San Antonio Evidence-based Practice Center and the Veterans Evidence-based Research, Dissemination, and Implementation Center, a Veterans Affairs Health Services Research and Development Center of Excellence, under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2000. AHRQ publication no. 01-E023.
 
Mulrow C, Lawrence V, Jacobs B, Dennehy C, Sapp J, Ramirez G, et al.  Milk thistle: effects on liver disease and cirrhosis and clinical adverse effects. Evidence Report/Technology Assessment No. 21 (Prepared by San Antonio Evidence-based Practice Center and the Veterans Evidence-based Research, Dissemination, and Implementation Center, a Veterans Affairs Services Research and Development Center of Excellence, under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2000. AHRQ publication no. 01-E025.
 
Hardy ML, Favreau JT, Elfenbaum PD, Morton SC, Roth EA, Genovese BJ, et al.  Mind-body interventions for gastrointestinal conditions. Evidence Report/Technology Assessment No. 40 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; July 2001. AHRQ publication no. 01-E030.
 
Hardy M, Coulter I, Venuturupalli S, Roth EA, Favreau J, Morton SC, et al.  Ayurvedic interventions for diabetes mellitus: a systematic review. Evidence Report/Technology Assessment No. 41 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; September 2001. AHRQ publication no. 01-E040.
 
Mulrow CD, Ramirez G, Cornell JE, Allsup K.  Defining and managing chronic fatigue syndrome. Evidence Report/Technology Assessment No. 42 (Prepared by San Antonio Evidence-based Practice Center under contract 290-97-0012). Rockville, MD: Agency for Healthcare Research and Quality; October 2001. AHRQ publication no. 02-E001.
 
Coates VH, Turkelson CM, Chapell R, Bruening W, Mitchell MD, Reston JT, et al.  Diagnosis and treatment of worker-related musculoskeletal disorders of the upper extremity. Evidence Report/Technology Assessment No. 62 (Prepared by ECRI under contract 290-97-0020). Rockville, MD: Agency for Healthcare Research and Quality; December 2002. AHRQ publication no. 02-E038.
 
Hardy ML, Coulter I, Morton SC, Favreau J, Venuturupalli S, Chiappelli F, et al.  S-Adenosyl-L-methionine for treatment of depression, osteoarthritis, and liver disease. Evidence Report/Technology Assessment No. 64 (Prepared by Southern California Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; October 2002. AHRQ publication no. 02-E034.
 
Shekelle P, Coulter I, Hardy ML, Morton SC, Udani J, Spar M, et al.  Effect of the supplemental use of antioxidants vitamin C, vitamin E, and coenzyme Q10 for the prevention and treatment of cancer. Evidence Report/Technology Assessment No. 75 (Prepared by Southern California Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; August 2003. AHRQ publication no. 03-E047.
 
Shekelle P, Coulter I, Hardy ML, Morton SC, Favreau JT, Gagne J, et al.  Best-case series for the use of immuno-augmentation therapy and naltrexone for the treatment of cancer. Evidence Report/Technology Assessment No. 78 (Prepared by Southern California-RAND Evidence-based Practice Center under contract No 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; April 2003. AHRQ publication no. 03-E030.
 
Shekelle P, Morton S, Hardy M, Coulter I, Udani J, Spar M.  Effect of supplemental antioxidants vitamin C, vitamin E, and coenzyme Q10 for the prevention and treatment of cardiovascular disease. Evidence Report/Technology Assessment No. 83 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-97-0001). Rockville, MD: Agency for Healthcare Research and Quality; July 2003. AHRQ publication no. 03-E043.
 
MacLean CH, Mojica WA, Morton SC, Pencharz J, Hasenfeld Garland R, Tu W, et al.  Effects of omega-3 fatty acids on lipids and glycemic control in type II diabetes and the metabolic syndrome and on inflammatory bowel disease, rheumatoid arthritis, renal disease, systemic lupus erythematosus, and osteoporosis. Evidence Report/Technology Assessment No. 89 (Prepared by Southern California-RAND Evidence-based Practice Center under contract 290-02-0003). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E012-2.
 
Schachter H, Reisman J, Tran K, Dales B, Kourad K, Barnes D, et al.  Health effects of omega-3 fatty acids on asthma. Evidence Report/Technology Assessment No. 91 (Prepared by University of Ottawa Evidence-based Practice Center under contract 290-02-0021). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E013-2.
 
Jordan H, Matthan N, Chung M, Balk E, Chew P, Kupelnick B, et al.  Effects of omega-3 fatty acids on arrhythmogenic mechanisms in animal and isolated organ/cell culture studies. Evidence Report/Technology Assessment No. 92 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E011-2.
 
Balk E, Chung M, Lichtenstein A, Chew P, Kupelnick B, Lawrence A, et al.  Effects of omega-3 fatty acids on cardiovascular risk factors and intermediate markers of cardiovascular disease. Evidence Report/Technology Assessment No. 93 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E010-2.
 
Wang C, Chung M, Lichtenstein A, Balk E, Kupelnick B, DeVine D, et al.  Effects of omega-3 fatty acids on cardiovascular disease. Evidence Report/Technology Assessment No. 94 (Prepared by Tufts-New England Medical Center Evidence-based Practice Center under contract 290-02-0022). Rockville, MD: Agency for Healthcare Research and Quality; March 2004. AHRQ publication no. 04-E009-2.
 
Agency for Healthcare Research and Quality, Center for Practice and Technology Assessment. Acupuncture for fibromyalgia (technology assessment). Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.cms.hhs.gov/coverage/download/id83.pdfon 29 April 2005.
 
Agency for Healthcare Research and Quality. Acupuncture for osteoarthritis (technology assessment). Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.cms.hhs.gov/coverage/download/id84.pdfon 29 April 2005.
 
Ritenbaugh C, Streit K, Helfand M.  Routine vitamin supplementation to prevent cancer: summary of evidence from randomized controlled trials. Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.ahrq.gov/clinic/3rduspstf/vitamins/vitasum.htmon 29 April 2005.
 
Morris CD, Carson S.  Routine vitamin supplementation to prevent cardiovascular disease. A summary of the evidence for the U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; June 2003. Accessed athttp://www.ahrq.gov/clinic/3rduspstf/vitamins/vitcvdsum.htmon 29 April 2005.
 
Last JM.  Gray literature. In: A Dictionary of Epidemiology. 4th ed. Oxford: Oxford Univ Pr; 2001.
 
Helmer D.  Grey literature. In: Etext on Health Technology Assessment (HTA) Information Resources. National Information Center on Health Services Research and Health Care Technology; 2004. Accessed athttp://www.nlm.nih.gov/nichsr/ehta/chapter10.htmlon 28 April 2005.
 
Hopewell S, McDonald S, Clarke M, Egger M.  Grey literature in meta-analyses of randomized trials of health care interventions (Cochrane Methodology Review). The Cochrane Database of Methodology Reviews. 2002; 4:Article MR000010. DOI: 10.1002/14651858.MR000010.
 
Hartling L, McAlister FA, Rowe BH, Ezekowitz J, Friesen C, Klassen TP.  Challenges in systematic reviews of therapeutic devices and procedures. Ann Intern Med. 2005; 142:1100-11.
 
Alderson P, Green S, Higgins JP.  Checking reference lists. In: Cochrane Reviewers' Handbook. Chichester, United Kingdom: J Wiley; 2004.
 

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)