What Comparative Effectiveness Research Is Needed? A Framework for Using Guidelines and Systematic Reviews to Identify Evidence Gaps and Research Priorities FREE

Comparative effectiveness research (CER) is a key element of current efforts in health care reform in the United States (1). Before initiating new research, investigators must ascertain gaps in the evidence by identifying the important clinical questions in a topic area and then determining whether existing research has answered these questions. Prioritizing new CER to address identified gaps presents an ongoing challenge (2). The Patient-Centered Outcomes Research Institute (PCORI) has committed funds to develop, test, refine, and evaluate methods that can inform the process of establishing and updating national priorities for CER and patient-centered outcomes research (3 - 4).

We propose and assess a general framework for prioritizing CER, using primary open-angle glaucoma (POAG) as a topic area. Worldwide, glaucoma of all types is a major source of morbidity, decreased quality of life, and increased health care costs (5 - 6). It has been estimated to be the second-leading cause of blindness and visual impairment in 2010 (5 - 6). In the United States, open-angle glaucoma accounts for more than 90% of all glaucoma cases; it affected more than 2.25 million Americans aged 40 years or older in 2000, and this number is expected to increase to 3.36 million by 2020 because of the aging population (5 - 6).

We aimed to develop, implement, and evaluate a framework for identifying evidence gaps and prioritizing CER. Because clinical practice guidelines (CPGs) reflect questions of interest to clinicians, we considered them a starting point for building our framework to prioritize CER. Once the clinical questions addressed in a CPG are identified, one can search for relevant systematic reviews to determine evidence gaps. For questions of intervention effectiveness, a systematic review of high-quality randomized, controlled trials (RCTs) can show whether research has answered the question; uncertainty reflected by systematic reviews indicates a potential candidate area for primary CER (2,7 - 8). Opinions on topics and outcomes can be sought from stakeholders, such as practicing clinicians, patients, and evidence users, and incorporated during the prioritization process (for example, when framing or ranking the research questions). Our objective for this report was to identify systematic reviews that addressed clinical questions derived from the CPGs, classify their methodological quality, propose a CER agenda, and assess the utility of our framework.

For practical purposes, we implemented and evaluated the utility of our framework in 2 phases (Figure 1). In phase 1, reported elsewhere (9), we used the 2005 American Academy of Ophthalmology (AAO) Preferred Practice Patterns for the management of POAG (10) to derive 45 answerable clinical questions that could be addressed by RCTs and systematic reviews of RCTs. We conducted a 2-round Delphi survey of 620 members of the American Glaucoma Society to prioritize the questions for research that would inform good patient care. Using responses from 169 participating clinicians, we classified 9 clinical questions as high priority (9).

In phase 2, we determined whether up-to-date systematic reviews were available for each of the 45 clinical questions derived from the AAO CPGs. To identify evidence gaps, we appraised the methodological quality of the reviews and mapped evidence from the reviews to the 45 questions. We derived the CER agenda for the management of POAG by combining the prioritized questions from phase 1 with the evidence gaps.

We prespecified the following criteria for including systematic reviews: full-text articles that described a systematic review, defined as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize similar but separate studies” (11 - 12); inclusion of patients with POAG, as defined by the individual review authors; comparison of any intervention described in the 45 clinical questions with another intervention, placebo, or no intervention; and reporting of any outcome. We excluded reports that concerned only health economic evaluation and meta-analyses with no systematic review.

We worked with information specialists at the William H. Welch Medical Library, Johns Hopkins University School of Medicine, and developed a search strategy that combined eye and vision terms with a validated search filter designed to identify systematic reviews (13). We originally searched PubMed in November 2006; EMBASE in March 2007; and The Cochrane Library, Issue 4, 2006, with no language restrictions, to build a register of systematic reviews addressing eye and vision research questions (Appendix 1 lists our search strategies). We updated our searches in September 2009.

We imported all citations from our electronic searches into a ProCite, version 5.0, database (Thomson Scientific, Philadelphia, Pennsylvania) and removed duplicate citations. Two investigators independently reviewed titles and abstracts, and we retrieved full-text articles of all records classified by at least 1 investigator as possibly eligible. Two investigators independently reviewed the full-text reports for final eligibility and resolved discrepancies through discussion.

We extracted data from eligible full-text systematic reviews for study objective, population, interventions compared, outcomes examined, eligibility criteria for studies in the systematic review, search methods used to identify studies, and other study characteristics.

We assessed methodological quality by using a pilot-tested form that adapted items from related instruments (14 - 18). Our assessment included the 13 quality criteria listed in Figure 2. When a meta-analysis was conducted as part of a systematic review, we abstracted data on the statistical methods used to calculate the meta-analytic estimate and its variance and on the assessment of statistical heterogeneity (the variability in the intervention effects across studies).

We selected 4 key deficiencies from the 13 quality items to classify the findings from a review as at low, high, or unclear risk of bias. Findings from a systematic review were classified as at high risk of bias if it contained a noncomprehensive literature search, did not assess the methodological quality of included studies, used inappropriate statistical methods for meta-analysis, or presented conclusions inconsistent with the review findings (for example, if the review finding was based on 1 statistically significant meta-analysis out of 26 meta-analyses conducted (19)). Findings were classified as at low risk of bias if none of these deficiencies was observed and as at unclear risk of bias if insufficient details were provided to permit judgment on 1 or more of the 4 deficiencies (Table 1) (20).

One reviewer abstracted data, assessed methodological quality, and classified risk of bias for the included systematic reviews. A second reviewer verified the abstracted data against the original publication; discrepancies were resolved through discussion.

We mapped systematic reviews that met our eligibility criteria to the 45 clinical questions; a systematic review could be associated with more than 1 question. We classified evidence cited in the AAO Preferred Practice Patterns to support the recommendations by study design (systematic review or RCT).

We compared the systematic reviews that were at low risk of bias with the results of the Delphi survey to identify evidence gaps and CER priorities. We considered evidence from systematic reviews to be most useful for decision making if it was at low risk of bias, up to date (the literature search was done within 2 years of publication of the review), and conclusive. We considered evidence to be conclusive if further research was unlikely to change our confidence in the estimate of effect and not conclusive if the reviews indicated uncertainty or unresolved questions and further research was likely to change our confidence in the estimate of effect (18).

We summarized the characteristics of included systematic reviews and tabulated the number and proportion of reviews that fulfilled each methodological quality criterion. All data analyses were performed with Stata, version 10 (StataCorp, College Station, Texas). We estimated the resources required and staff needed to complete the project.

This phase of our study was funded by the National Eye Institute, National Institutes of Health. Phase 1 was supported by the Cochrane Prioritization Fund, The Cochrane Collaboration. The sponsors had no role in the design or conduct of the study; the collection, management, analysis, or interpretation of the data; or the preparation, review, or approval of the manuscript.

We identified 5990 unique records from our electronic searches, of which 547 systematic reviews addressed eye and vision research questions. Of these, 39 systematic reviews (42 articles) addressed questions related to management of POAG and met our eligibility criteria (Appendix Figure 1) (19,21 - 61). Eight of the 39 were Cochrane reviews, 18 were found in the Database of Abstracts of Reviews of Effects in The Cochrane Library, and the remaining 13 were found only in PubMed or EMBASE. The 39 systematic reviews were published between 2000 and 2009, with more than half published after 2007 (Appendix Table 1). The literature search was done within 2 years of publication of the review in 35 of 39 cases (89.7%).

Figure 2 illustrates the findings from our appraisal of methodological quality of the 39 systematic reviews. Nearly 40% of the reviews did not report having prespecified eligibility criteria or performing a comprehensive literature search. For nearly half of the reviews, independent evaluation by more than 1 review author was not reported for eligibility assessment, data abstraction, or quality assessment of the included studies.

Thirteen of the 39 reviews (33.3%) included in this study were classified as at low risk of bias, 22 (56.4%) at high risk of bias, and 4 (10.3%) at unclear risk of bias. Reviews at high risk of bias were classified as such for 1 or more of the following reasons: noncomprehensive literature search (12 reviews), no assessment of the methodological quality of the included studies (11 reviews), inappropriate statistical analyses (13 reviews), or conclusions that were inconsistent with the review findings (1 review). Fifteen of 22 reviews were classified as at high risk of bias because of at least 2 reasons.

We identified 2 major types of inappropriate statistical analyses in 13 of 32 reviews (40.6%) with a meta-analysis (3 reviews had both types of inappropriate analyses) (Appendix 2). Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison (for example, data from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials were pooled separately and compared in a review (39)). In addition, 7 reviews that reported percentage change in intraocular pressure (IOP) from baseline as the effect estimate used an incorrect formula to calculate the variance of the effect estimate. Appendix 2 summarizes the incorrect formulas used in the reviews to calculate the variance for the percentage of change in IOP from baseline.

Table 2 shows the results of applying our method to identify evidence gaps in the context of the overall framework that we evaluated. We identified at least 1 systematic review for 23 of 45 clinical questions (51.1%) and at least 1 review at low risk of bias for 13 of 45 questions (28.8%) (Table 2 and Appendix Table 2). Five of the 9 questions identified by clinicians as high priority for research were associated with at least 1 systematic review that was at low risk of bias. Of these, 2 had reviews with findings that we classified as not conclusive; therefore, these 2 questions need new primary research (such as an RCT). The remaining 4 high-priority questions require a new systematic review because no systematic review at low risk of bias was found. Of the 36 lower-priority questions, 28 require a new systematic review (Appendix Table 2). The 2005 AAO Preferred Practice Patterns cited only 5 of 10 systematic reviews published before the guidelines were prepared (32 - 35,37,49,59 - 60), and we classified 3 of the 5 as at low risk of bias (37,59 - 60).

We report the approximate time associated with each step and staff needed to complete the prioritization project (Figure 1 also lists stakeholder involvement).

For phase 1, deriving and refining clinical questions from CPGs took 80 hours. Systematic review methodologists, epidemiologists, and clinical trialists derived the questions from the CPG. Glaucoma specialists with expertise in clinical trials and systematic reviews and methodologists from the CPG development committee verified and refined the questions. Prioritizing the clinical questions took 200 hours, including the time for creating the survey questionnaire, conducting pilot testing, and administering the survey. In addition, we estimate that it took 30 minutes for each participating member of the American Glaucoma Society to complete our prioritization survey. The online survey interface and underlying database were created with donated service from a data management specialist. The survey was administered and data were managed by a graduate student.

For phase 2, we cannot estimate the time for identifying existing systematic reviews because this step was part of another project, for which we searched for and identified systematic reviews covering all eye and vision topics for a subject-specific database. Systematic review methodologists worked together with information specialists to complete this step. Extracting data and appraising the methodological quality of the systematic reviews took 3 hours per included review for data extraction, verification, and adjudication of discrepancies. A senior epidemiologist oversaw the process. Identifying the evidence gaps and prioritizing CER took 120 hours. Systematic review methodologists, epidemiologists, clinical trialists, and glaucoma specialists worked together in identifying evidence gaps. Finally, the time for drafting and revising the final report or manuscript will vary depending on the ultimate goal of the prioritization project, the sponsor's requirements, and the publication process.

Our overall goal was to test a framework for prioritizing clinical questions and identifying evidence gaps by using existing systematic reviews and CPGs. The framework we tested includes the following steps: deriving clinical research questions from CPGs to reflect issues that clinicians encounter frequently, asking clinicians to prioritize the 45 questions for research to incorporate opinions from evidence users, and determining whether high-quality systematic reviews of all previous research exist for each clinical question and identifying evidence gaps. By mapping evidence gaps to clinicians' priorities, we proposed a CER agenda.

Our experience demonstrated the feasibility of engaging key stakeholders in setting priorities at several junctures. We collaborated with practicing clinicians, researchers, CPG developers (AAO), and a relevant professional association (American Glaucoma Society) (9). In phase 1, we asked clinicians which questions they needed answered to practice effectively. Patient views could easily be incorporated into this phase. In phase 2, we identified the questions for which research is needed. Clinicians and patients may be in the best position to decide what is needed in clinical practice but may not be aware of or may discount existing research for various reasons (for example, because the patient population differs from the clinician's own). Thus, a 2-phase framework that engages stakeholders with different perspectives and expertise is essential for setting CER priorities.

Our framework for setting CER priorities provides a clear guide for the type of research needed to address each clinical question (such as studies to synthesize existing evidence or to generate new evidence). We found that high-quality systematic reviews were needed to address 32 of the 45 clinical questions on the management of POAG, including 4 of the 9 high-priority clinical questions. We also identified 2 clinical questions for which evidence from new RCTs is needed because the existing reviews were not conclusive. However, a good practice would be to update the existing high-quality systematic reviews to include recent and ongoing research as a confirmatory step before initiating any new RCTs.

The framework we tested is a robust addition to existing approaches for setting CER priorities (2,62 - 70). It differs from nomination-based methods, such as those used by the Agency for Healthcare Research and Quality (62 - 63) and the Institute of Medicine (64), in which research topics are suggested by clinicians, payers, members of the public, or others, and a set of rules are applied to determine the final priorities. These other methods are quite broad and have produced overall priorities for health care research (71). In contrast, our framework focuses on a specialty area and thus addresses the entire range of questions specialists face.

Our framework has limitations. When translating CPG recommendations into answerable clinical questions, we relied on interventions and outcome measures that were stated explicitly in the CPG. This may be of concern when guideline recommendations are out of date or do not address emerging research, techniques, practices, or outcomes considered important and relevant to clinicians and patients. We addressed this concern by asking stakeholders to supplement and refine the list of derived clinical questions as part of survey development. For our ongoing project in which we are applying our framework to diabetic retinopathy, a condition in which new technologies and drugs are developing rapidly, we are seeking the involvement of both clinical experts and patients to develop clinical questions to supplement those derived from CPGs. In contrast, when we applied our framework to interventions for POAG, we found no new interventions in the 2010 update of the 2005 guidelines by the AAO. In addition, we have involved stakeholders from regions where the condition is prevalent to ensure broad experience with the topic (Yu T, Li T, Friedman DS, Puhan MA, Dickersin K. Setting priorities for comparative effectiveness research on the management of primary angle closure: a survey of Asia-Pacific clinicians. In preparation). In a third example, we are planning to seek input from patients for research questions on dry eye, for which patient-reported outcomes are of primary interest. We encourage others to test our framework by extending our methods according to the topic and their needs.

Our framework relies on published systematic reviews; thus, the underlying trial data may not be completely up to date. We believe timeliness is a problem regardless of the prioritization method used; this is addressed to some extent when prioritized questions are first reported in a clinical journal because the community has an opportunity to assess the applicability of the question to current practice and knowledge. Our prioritization framework is less sensitive to timeliness than the questions themselves. For example, in our study of diabetic retinopathy, involving clinicians and patients will almost certainly result in adding clinical questions about the effectiveness of antivascular endothelial growth factor interventions for treating diabetic macular edema to those derived from CPGs. The trials on this topic are fairly recent, and existing systematic reviews may not yet include them. Thus, we suggest that CER researchers update existing systematic reviews before undertaking a new trial.

We focused on RCTs and systematic reviews of RCTs because we were interested in clinical questions of intervention effectiveness. Other study types may be equally or more appropriate in some instances, such as for questions about intervention harms. In addition, for questions that involve both effectiveness and harm, existing data may be conclusive for benefit but not for harm or conclusive for intervention effectiveness on some but not all outcomes. For example, we classified the main review findings about effectiveness as conclusive for our priority question on the effectiveness and safety of intraoperative mitomycin C; however, we also concluded that additional data may be needed to clarify certain serious but rare side effects, such as endophthalmitis (Table 2). The clinical and research community will need to decide when unresolved questions of harm are sufficiently important to warrant undertaking a new clinical trial or observational study.

Although we searched 3 major databases with a comprehensive and validated search strategy to identify systematic reviews, we did not search for unpublished systematic reviews, which may have a different methodological quality profile and findings from those published (72). In addition, our assessment of methodological quality relied on what the authors reported in the article, which may not reflect the actual methods used. Finally, if systematic review authors made errors when abstracting trial data, we could have underestimated the number of unreliable systematic reviews (73 - 78).

Our findings show that although systematic reviews of RCTs are considered the highest level of evidence for questions about intervention effectiveness (7), their findings should not be accepted without critical evaluation of their methodological rigor. Consistent with others (79 - 83), we identified a large proportion of systematic reviews that did not use rigorous methods, such as those suggested by the Institute of Medicine and others (84). We classified the findings of two thirds of the reviews as at high risk of bias for either having a noncomprehensive search for relevant studies or using inappropriate statistical methods. The negative impact of these methodological problems has been well-described in the literature (20,85 - 94).

The clinical questions we prioritized are restricted to pairwise comparisons and did not directly answer the question, “Which intervention works best?” because of the way clinical questions are addressed in most RCTs. Because this question must be answered to facilitate optimal health care decision making, a network meta-analysis should be considered once data from all high-quality systematic reviews and trials are identified for a given question (95). Our framework, which identifies all high-quality systematic reviews related to a health topic generally (96 - 97), has an important scoping role in this regard.

We believe that our framework, which uses CPGs and systematic reviews to prioritize CER involving both new primary research and systematic reviews, is broadly applicable and could be applied in conjunction with other methods (for example, by PCORI) to formulate future research agendas.

(ABNORMAL ACCOMMODATION[tiab] OR Abnormal color vision[tiab] OR ABNORMAL LACRIMATION[tiab] OR Abnormal vision[tiab] OR accommodative disorders[tiab] OR Amblyopia[tiab] OR Ametropia[tiab] OR ANISOCORIA[tiab] OR ANOPHTHALMIA[tiab] OR Anterior CHAMBER hemorrhage[tiab] OR Aphakia[tiab] OR aqueous outflow obstruction[tiab] OR Asthenopia[tiab] OR Balint's syndrome[tiab] OR Bilateral visual field constriction[tiab] OR Binocular Vision Disorder[tiab] OR BLEPHARITIS[tiab] OR BLEPHAROSPASM[tiab] OR BLINDNESS[tiab] OR blurred vision[tiab] OR CATARACT[tiab] OR Cataracts[tiab] OR Chorioretinal disorder[tiab] OR Chorioretinitis[tiab] OR Choroid Diseases[tiab] OR Choroidal[tiab] OR Choroiditis[tiab] OR CHROMATOPSIA[tiab] OR Color Blindness[tiab] OR Color Vision Defects[tiab] OR Color vision deficiency[tiab] OR Colour blindness[tiab] OR Conjunctival Diseases[tiab] OR CONJUNCTIVAL HAEMORRHAGE[tiab] OR Conjunctival Injury[tiab] OR CONJUNCTIVAL ULCERATION[tiab] OR CONJUNCTIVITIS[tiab] OR CORNEAL DEPOSITS[tiab] OR Corneal Diseases[tiab] OR Corneal Disorder[tiab] OR Corneal injuries[tiab] OR Corneal Injury[tiab] OR CORNEAL OEDEMA[tiab] OR CORNEAL OPACITY[tiab] OR CORNEAL ULCERATION[tiab] OR decreased Lacrimation[tiab] OR Decreased vision[tiab] OR defective vision[tiab] OR Delayed visual maturation[tiab] OR Difficulty seeing[tiab] OR difficulty with vision[tiab] OR Dim vision[tiab] OR Diminished Vision[tiab] OR DIPLOPIA[tiab] OR disturbed vision[tiab] OR ENDOPHTHALMITIS[tiab] OR EPIPHORA[tiab] OR Episcleritis[tiab] OR Equatorial staphyloma[tiab] OR EXOPHTHALMOS[tiab] OR Eye Abnormalities[tiab] OR EYE ABNORMALITY[tiab] OR Eye Burns[tiab] OR Eye disease[tiab] OR Eye Diseases[tiab] OR Eye disorder[tiab] OR Eye disorders[tiab] OR Eye edema[tiab] OR Eye Foreign Bodies[tiab] OR Eye Hemorrhage[tiab] OR EYE INFECTION[tiab] OR Eye Infections[tiab] OR Eye Injuries[tiab] OR EYE MALFORMATION[tiab] OR Eye Manifestations[tiab] OR EYE MUSCLE PARALYSIS[tiab] OR Eye Neoplasm[tiab] OR Eye Neoplasms[tiab] OR EYE PAIN[tiab] OR Eye swelling[tiab] OR Eyelid Disease[tiab] OR Eyelid Diseases[tiab] OR Eyelid Disorder[tiab] OR Eyelid Disorders[tiab] OR Eyelid pain[tiab] OR EYELID RETRACTION[tiab] OR FIXED PUPILS[tiab] OR GLAUCOMA[tiab] OR Glaucomas[tiab] OR Hazy vision[tiab] OR HEMIANOPIA[tiab] OR Hemianopsia[tiab] OR Hepatolenticular Degeneration[tiab] OR Horner's syndrome[tiab] OR HYPOPYON[tiab] OR Impaired vision[tiab] OR impaired visual acuity[tiab] OR Interference with vision[tiab] OR IRITIS[tiab] OR KERATITIS[tiab] OR KERATOCONJUNCTIVITIS[tiab] OR Keratoconus[tiab] OR Lacrimal Apparatus Diseases[tiab] OR lacrimal disorder[tiab] OR LACRIMAL DUCT OBSTRUCTION[tiab] OR legally blind[tiab] OR Lens Diseases[tiab] OR Lens Disorder[tiab] OR Lens Opacities[tiab] OR Lens Subluxation[tiab] OR Localized anterior staphyloma[tiab] OR Low vision[tiab] OR MACULAR DEGENERATION[tiab] OR MEIBOMIANITIS[tiab] OR Metastases to eye[tiab] OR MIOSIS[tiab] OR MYDRIASIS[tiab] OR MYOPIA[tiab] OR Night Blindness[tiab] OR Nystagmus[tiab] OR Ocular degeneration[tiab] OR Ocular discomfort[tiab] OR Ocular disease[tiab] OR OCULAR HAEMORRHAGE[tiab] OR OCULAR HEMORRHAGE[tiab] OR OCULAR HERPES[tiab] OR Ocular Hypertension[tiab] OR Ocular Hypotension[tiab] OR Ocular infections[tiab] OR Ocular inflammations[tiab] OR Ocular injuries[tiab] OR Ocular Injury[tiab] OR Ocular Motility Disorders[tiab] OR Ocular neoplasms[tiab] OR oculopathy[tiab] OR Open wound of ocular adnexa[tiab] OR OPHTHALMIC DISORDERS[tiab] OR Ophthalmological disorder[tiab] OR Ophthalmopathy[tiab] OR OPTIC ATROPHY[tiab] OR Optic Nerve Diseases[tiab] OR optic nerve disorder[tiab] OR Optic nerve injury[tiab] OR OPTIC NEURITIS[tiab] OR Orbital Diseases[tiab] OR PAPILLOEDEMA[tiab] OR Partial sight[tiab] OR Partial vision loss[tiab] OR Partially Sighted[tiab] OR Penetrating Eye Injuries[tiab] OR Periorbital fat herniation[tiab] OR Photalgia[tiab] OR PHOTOPHOBIA[tiab] OR PHOTOPSIA[tiab] OR PIGMENT PRECIPITATION[tiab] OR Poor vision[tiab] OR Posterior dislocation of lens[tiab] OR Posterior synechiae[tiab] OR Problem seeing[tiab] OR Proliferative Vitreoretinopathy[tiab] OR Pupil Disorders[tiab] OR Redness[All Fields] OR discharge of eye[tiab] OR Reduced ability to see[tiab] OR Reduced Vision[tiab] OR Refraction Errors[tiab] OR Refractive disorders[tiab] OR Refractive Errors[tiab] OR Retinal defects[tiab] OR RETINAL DEPOSITS[tiab] OR RETINAL DETACHMENT[tiab] OR Retinal detachments[tiab] OR Retinal Disease[tiab] OR Retinal Diseases[tiab] OR RETINAL DISORDER[tiab] OR RETINAL EDEMA[tiab] OR RETINAL HAEMORRHAGE[tiab] OR RETINAL HEMORRHAGE[tiab] OR RETINAL OEDEMA[tiab] OR RETINITIS[tiab] OR RETINOBLASTOMA[tiab] OR retinopathy[tiab] OR RETROBULBAR NEURITIS[tiab] OR Scleral Diseases[tiab] OR Scleral staphyloma[tiab] OR SCLERITIS[tiab] OR SCOTOMA[tiab] OR Sight impaired[tiab] OR Staphyloma posticum[tiab] OR STRABISMUS[tiab] OR Subnormal Vision[tiab] OR SUNKEN EYES[tiab] OR SYMBLEPHARON[tiab] OR THROMBOSIS RETINAL VEIN[tiab] OR Tunnel Vision[tiab] OR Uveal Diseases[tiab] OR Uveal Disorder[tiab] OR UVEITIS[tiab] OR vision defects[tiab] OR vision disorder[tiab] OR Vision Disorders[tiab] OR vision disturbance[tiab] OR vision impaired[tiab] OR VISION IMPAIRMENT[tiab] OR Vision problem[tiab] OR Visual Agnosia[tiab] OR visual defect[tiab] OR Visual difficulty[tiab] OR Visual disorders[tiab] OR Visual disturbance[tiab] OR VISUAL FIELD DEFECT[tiab] OR Visual Field Disorder[tiab] OR Visual field disorders[tiab] OR Visual impairment[tiab] OR Visual loss[tiab] OR Visual Pathway Disorder[tiab] OR Visual pathway disorders[tiab] OR Visual system disorder[tiab] OR VITREOUS DETACHMENT[tiab] OR VITREOUS HAEMORRHAGE[tiab] OR Vitreous Hemorrhage[tiab] OR Vitreous membranes and strands[tiab] OR Vitreous prolapse[tiab] OR Vitreous syneresis[tiab] OR Wavefront aberration[tiab] OR Weak vision[tiab] OR Wilson's Disease[tiab] OR XEROPHTHALMIA[tiab]

OR

((accommodation[tiab] OR aqueous humor[tiab] OR binocular movement[tiab] OR Capsule[tiab] OR choroid[tiab] OR ciliary body[tiab] OR conjunctiva[tiab] OR Cornea[tiab] OR (eye[tiab] AND adnexa[tiab]) OR eye[tiab] OR eyelid[tiab] OR globe[tiab] OR intraocular pressure[tiab] OR Iris[tiab] OR lacrimal passage[tiab] OR lacrimal system[tiab] OR lens[tiab] OR ocular adnexa[tiab] OR optic nerve[tiab] OR orbit[tiab] OR refraction[tiab] OR retina[tiab] OR sclera[tiab] OR trochlear nerve[tiab] OR uvea[tiab] OR Uveal Tract[tiab] OR vision[tiab] OR visual pathway[tiab] OR vitreous body[tiab] OR vitreous humor[tiab])

AND

(“disease”[MeSH Terms] OR Disease[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Diseases[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Disorder[Text Word]) OR ((“disease”[TIAB] NOT Medline[SB]) OR “disease”[MeSH Terms] OR Disorders[Text Word]) OR ((“contusions”[TIAB] NOT Medline[SB]) OR “contusions”[MeSH Terms] OR Contusion[Text Word]) OR (“contusions”[MeSH Terms] OR Contusions[Text Word]) OR (“syndrome”[MeSH Terms] OR Syndrome[Text Word]) OR ((“syndrome”[TIAB] NOT Medline[SB]) OR “syndrome”[MeSH Terms] OR Syndromes[Text Word]) OR (“dislocations”[MeSH Terms] OR Dislocations[Text Word]) OR ((“dislocations”[TIAB] NOT Medline[SB]) OR “dislocations”[MeSH Terms] OR Dislocation[Text Word]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular[Text Word]) AND Occlusion[All Fields]) OR (((“blood vessels”[TIAB] NOT Medline[SB]) OR “blood vessels”[MeSH Terms] OR Vascular[Text Word]) AND Occlusions[All Fields]) OR ((“wounds and injuries”[TIAB] NOT Medline[SB]) OR “wounds and injuries”[MeSH Terms] OR Injury[Text Word]) OR (“injuries”[Subheading] OR (“wounds and injuries”[TIAB] NOT Medline[SB]) OR “wounds and injuries”[MeSH Terms] OR Injuries[Text Word]) OR (“coloboma”[MeSH Terms] OR Coloboma[Text Word]))

OR

“amaurosis fugax”[MeSH Terms] OR “amblyopia”[MeSH Terms] OR “asthenopia”[MeSH Terms] OR “blindness”[MeSH Terms] OR “blindness, cortical”[MeSH Terms] OR “color vision defects”[MeSH Terms] OR “conjunctival diseases”[MeSH Terms] OR “corneal diseases”[MeSH Terms] OR “diplopia”[MeSH Terms] OR “eye abnormalities”[MeSH Terms] OR “eye burns”[MeSH Terms] OR “eye diseases”[MeSH Terms] OR “eye diseases, hereditary”[MeSH Terms] OR “eye foreign bodies” [MeSH Terms] OR “eye hemorrhage”[MeSH Terms] OR “eye infections”[MeSH Terms] OR “eye injuries”[MeSH Terms] OR “eye injuries, penetrating”[MeSH Terms] OR “eye manifestations”[MeSH Terms] OR “eye neoplasms”[MeSH Terms] OR “eyelid diseases”[MeSH Terms] OR “hemianopsia”[MeSH Terms] OR “lacrimal apparatus diseases”[MeSH Terms] OR “lens diseases”[MeSH Terms] OR “night blindness”[MeSH Terms] OR “ocular hypertension”[MeSH Terms] OR “ocular hypotension”[MeSH Terms] OR “ocular motility disorders” [MeSH Terms] OR “optic nerve diseases”[MeSH Terms] OR “orbital diseases”[MeSH Terms] OR “photophobia”[MeSH Terms] OR “pupil disorders”[MeSH Terms] OR “refractive errors”[MeSH Terms] OR “retinal diseases”[MeSH Terms] OR “scleral diseases”[MeSH Terms] OR “scotoma”[MeSH Terms] OR “uveal diseases”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vision disorders”[MeSH Terms] OR “vitreoretinopathy, proliferative”[MeSH Terms] OR “vitreous detachment” [MeSH Terms])

AND

(“therapy”[Subheading] OR (“therapeutics”[TIAB] NOT Medline[SB]) OR “therapeutics”[MeSH Terms] OR treatment[Text Word]) OR ((“therapeutics”[TIAB] NOT Medline[SB]) OR “therapeutics”[MeSH Terms] OR treatments[Text Word]) OR (“diagnosis”[Subheading] OR “diagnosis”[MeSH Terms] OR diagnosis[Text Word]) OR intervention[All Fields] OR interventions[All Fields] OR (“prevention and control”[Subheading] OR prevention[Text Word])

AND

(Humans[MeSH] OR (Humans[Mesh] NOT Animals[MeSH])

AND

cochrane database syst rev[ta] OR search[tiab] OR meta-analysis[pt] OR MEDLINE[tiab] OR (systematic[tiab] AND review[tiab])

((*‘asthenopia’*/de OR *‘asthenopia’*) OR (*‘cerebral blindness’*/de OR *‘cerebral blindness’*) OR (*‘diplopia’*/de OR *‘diplopia’*) OR (*‘eye malformation’*/de OR *‘eye malformation’*) OR (*‘eye burn’*/de OR *‘eye burn’*) OR (*‘eye injury’*/de OR *‘eye injury’*) OR (*‘perforating eye injury’*/de OR *‘perforating eye injury’*) OR (*‘hemianopia’*/de OR *‘hemianopia’*) OR (*‘lacrimal gland disease’*/de OR *‘lacrimal gland disease’*) OR (*‘intraocular hypotension’*/de OR *‘intraocular hypotension’*) OR (*‘photophobia’*/de OR *‘photophobia’*) OR (*‘pupil disease’*/de OR *‘pupil disease’*) OR (*‘sclera disease’*/de OR *‘sclera disease’*) OR (*‘scotoma’*/de OR *‘scotoma’*) OR (*‘vitreoretinopathy’*/de OR *‘vitreoretinopathy’*) OR (*‘vitreous body detachment’*/de OR *‘vitreous body detachment’*) OR (*‘transitional blindness’*/de OR *‘transitional blindness’*) OR (*‘amblyopia’*/de OR *‘amblyopia’*) OR (*‘blindness’*/de OR *‘blindness’*) OR (*‘color vision defect’*/de OR *‘color vision defect’*) OR (*‘conjunctiva disease’*/de OR *‘conjunctiva disease’*) OR (*‘cornea disease’*/de OR *‘cornea disease’*) OR (*‘eye disease’*/de OR *‘eye disease’*) OR (*‘intraocular foreign body’*/de OR *‘intraocular foreign body’*) OR (*‘intraocular hemorrhage’*/de OR *‘intraocular hemorrhage’*) OR (*‘eye infection’*/de OR *‘eye infection’*) OR (*‘eye tumor’*/de OR *‘eye tumor’*) OR (*‘eyelid disease’*/de OR *‘eyelid disease’*) OR (*‘lens disease’*/de OR *‘lens disease’*) OR (*‘night blindness’*/de OR *‘night blindness’*) OR (*‘intraocular hypertension’*/de OR *‘intraocular hypertension’*) OR (*‘eye movement disorder’*/de OR *‘eye movement disorder’*) OR (*‘optic nerve disease’*/de OR *‘optic nerve disease’*) OR (*‘orbit disease’*/de OR *‘orbit disease’*) OR (*‘refraction error’*/de OR *‘refraction error’*) OR (*‘retina disease’*/de OR *‘retina disease’*) OR (*‘uvea disease’*/de OR *‘uvea disease’*) OR (*‘visual disorder’*/de OR *‘visual disorder’*)) AND ((*search*) OR (*‘meta analysis’* OR *‘systematic review’*))

We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Health Technology Assessment database, and the United Kingdom National Health Service Economic Evaluation Database in the Cochrane Library by using the following Medical Subject Heading terms as terms and as keywords in “Title, Abstract, and Keyword”: amaurosis fugax; amblyopia; blindness; color vision defects; conjunctival diseases; corneal diseases; eye diseases; eye diseases, hereditary; eye foreign bodies; eye hemorrhage; eye infections; eye injuries; eye manifestations; eye neoplasms; eyelid diseases; lens diseases; night blindness; ocular hypertension; ocular motility disorders; optic nerve diseases; orbital diseases; refractive errors; retinal diseases; uveal diseases; and vision disorders.

We also searched the following non–Medical Subject Heading terms in “Title, Abstract, and Keyword”: vision, visual, eye diseases, eye disorders, macular degeneration, cataract, glaucoma, strabismus, hyphema, conjunctivitis, eye, optic nerve, choroiditis, chorioretinitis, blepharitis, blindness, ocular, lacrimal, iris, uveitis, pupil, and retina.

Thirteen of the 32 systematic reviews with at least 1 meta-analysis that we included in our sample applied at least 1 inappropriate statistical analysis. We identified 2 major types of inappropriate analysis among the 13 reviews. Nine reviews pooled data from similar groups across studies, resulting in a nonrandom comparison, and 7 used an incorrect formula to calculate the variance of the effect estimate (3 reviews had both types of inappropriate analyses).

Nine of the 32 systematic reviews with a meta-analysis that we included in our sample pooled data from treatment groups (for example, 1 review (39) pooled data separately from the timolol treatment groups from 21 trials and latanoprost treatment groups from 33 trials and compared them [Appendix Figure 2]). In all 9 cases, the reviews examined at least 3 interventions for a given condition. The analysis that pooled individual treatment groups may have been an attempt to indirectly compare interventions; that is, comparison of 2 interventions through a common comparator (20,95). For example, indirect evidence on A versus C can be obtained from RCTs of either A or C versus a common comparator B.

The consensus among methodologists is that pooling treatment groups is an incorrect way of conducting a meta-analysis (20,98). Pooling treatment groups has been described by Glenny and colleagues (20) as the “naive method” for indirect comparison. This approach has been criticized for negating the randomized nature of the comparison between interventions and thereby increasing the likelihood of selection and confounding biases and falsely precise estimates. The correct way of comparing multiple treatments for the same condition is through a network meta-analysis (20,95,98 - 99).

Many systematic reviews included in our sample described differences between the intervention and comparison groups in mean percentage of change in IOP from baseline to the end point as a measure of treatment effect and to gauge the success of IOP-lowering treatments. Mean percentage of change in IOP from baseline can be expressed as follows:

Where i indicates individual participant and n indicates the total number of participants.

We found that incorrect formulas, in different forms, were used for estimating the SD for the percentage of change in IOP from baseline to the end point (Appendix Figures 3 and 4). Incorrect formulas for estimating the SD can invalidate results and conclusions from a review, because the SD directly relates to the SE through sample size. The SE is used to determine the weight of each study in the meta-analysis and to calculate the CIs for the pooled treatment effect.

The formulas to calculate the SD of the absolute change in IOP from baseline to the end point were incorrect in Appendix Figures 3(61) and 4(19) because the analyses ignored the correlation between the repeated IOP measurements from the same individual at baseline and the end point. In Appendix Figure 3, the authors used the SE instead of the SD, which could be a notation error. The incorrect formulas used to calculate the SD of change in IOP from baseline to the end point were expressed as

and

in Appendix Figures 3 and 4, respectively. The correct formula could be written as

in which ρ is the correlation between the repeated IOP measurements from the same individual at baseline and the end point.

When individual-patient data are available, that is, a data set with intervention, outcome, and covariate information for each patient, another way to compute SD_IOPR is to take the differences (end point minus baseline) for each individual and compute the SD of these values.

The formulas to calculate the SD of the percentage of change in IOP from baseline to the end point were also incorrect in Appendix Figures 3 and 4 because the percentage of change in IOP from baseline to the end point is a ratio of 2 quantities, and estimating the SD for a ratio is mathematically complex (100). The incorrect formulas used to calculate the SD of the percentage of change in IOP from baseline to the end point were expressed as

and

in Appendix Figures 3 and 4, respectively.

If all trials included in the meta-analysis had reported the mean percentage of change in IOP from baseline to the end point and SD for each treatment group, or for between-group comparisons, then using these data to conduct a meta-analysis would have been straightforward. However, because 1 or more of the included trials did not report mean percentage of change in IOP from baseline and its SD, accurate estimation of these values requires individual-patient data.

Using percentage of change in IOP from baseline as an outcome in a meta-analysis is recommended only when individual-patient data are available and only with advice from an experienced statistician because of the myriad potential problems associated with the assumptions and the realities of available data. The clinical community should reach a consensus on whether percentage of change in IOP from baseline is an appropriate measure of treatment effect for IOP-lowering interventions. Such a consensus would inform design, analyses, and reporting for clinical trials of interventions that affect IOP. This discussion has been initiated through the National Eye Institute/U.S. Food and Drug Administration symposium on glaucoma clinical trial design and end points (101).

In the Clinic is a monthly feature in Annals that focuses on practical management of patients with common clinical conditions. It offers evidence-based answers to frequently asked questions about screening, prevention, diagnosis, therapy, and patient education and provides physicians with tools to improve the quality of care. In the Clinic includes links to PIER and other evidence sources and continiuing medical education quizzes offering category 1 CME credit.

For more information on In the Clinic, visit www.annals.org/site/intheclinic.