Test Performance of Positron Emission Tomography and Computed Tomography for Mediastinal Staging in Patients with Non–Small-Cell Lung Cancer: A Meta-Analysis FREE

–The Editors

Accurate mediastinal staging is crucial in managing patients with non–small-cell lung cancer. Regional lymph node status is an important determinant of prognosis, and decisions about treatment depend critically on tumor stage. Conventional methods for mediastinal staging include computed tomography (CT) and various biopsy procedures. However, CT has poor sensitivity and specificity for identifying mediastinal metastases (1 - 3), and biopsy procedures are inconvenient and potentially risky.

Positron emission tomography (PET) with 18-fluorodeoxyglucose (FDG) is a promising but expensive functional imaging test that is rapidly gaining acceptance as a tool for lung cancer staging (4 - 5). Positron emission tomography with 18-fluorodeoxyglucose identifies malignant cells in tumors and lymph nodes on the basis of their increased metabolic rate (6). In the past decade, several studies of PET imaging for mediastinal staging were published. These studies suggested that FDG-PET is more accurate than CT for identifying mediastinal metastases. However, most were small and potentially limited by other methodologic shortcomings. In addition, previous studies have not systematically addressed the conditional test performance of FDG-PET and CT. Conditional test performance refers to the possibility that the sensitivity and specificity of 1 test might differ depending on the results of the other test (7). The results of FDG-PET and CT might be mutually dependent, despite the fact that they identify malignant lymph nodes by different mechanisms. In a preliminary analysis, we found that FDG-PET was more sensitive but less specific in patients with lymph node enlargement on CT (8). If confirmed, this finding has important implications for selecting and interpreting tests in mediastinal staging. For example, if FDG-PET is more sensitive when lymph node enlargement is present on CT, then a negative PET result would “rule out” disease more reliably (because its negative predictive value would be higher). Consequently, confirmatory mediastinal biopsy might not be necessary in some of these patients, especially when pretest probability is low.

We performed this meta-analysis to compare the accuracy of FDG-PET and CT for identifying mediastinal metastasis in patients with non–small-cell lung cancer. We also aimed to determine whether the results of FDG-PET and CT are conditionally dependent, that is, whether the sensitivity and specificity of FDG-PET depend on the presence or absence of lymph node enlargement on CT. Finally, we explored whether various aspects of study methods affected diagnostic accuracy.

We used systematic review methods to identify potentially relevant studies, assess studies for eligibility, evaluate study quality, and derive summary estimates of diagnostic test performance (9 - 12). We previously used similar methods to evaluate the accuracy of FDG-PET imaging for diagnosis of pulmonary nodules and mass lesions (13). Additional details about our methods can be found in the Appendix.

We attempted to identify all published studies that examined FDG-PET imaging for mediastinal staging in patients with known or suspected non–small-cell lung cancer. We sought studies that evaluated both FDG-PET and CT, but we did not attempt to identify studies that examined only CT for mediastinal staging. An investigator and a professional librarian searched MEDLINE, CancerLit, and EMBASE databases in August 2001 and repeated searches in June 2002 (Appendix Table 1). We updated the literature search in MEDLINE, EMBASE, Current Contents, and BIOSIS through 27 March 2003 as part of a technology assessment performed for the U.S. Department of Veterans Affairs (Appendix Table 2). We augmented our computerized literature searches by manually reviewing the reference lists of identified studies and review articles. We included studies published in any language but did not include abstracts. For English-language studies, 2 investigators independently evaluated studies for inclusion, rated the methodologic quality of included studies, and abstracted relevant data. Disagreements were resolved by discussion. One reviewer performed these tasks for non-English-language studies. Reviewers were blinded to journal, author, institutional affiliation, and date of publication.

We included studies that examined FDG-PET imaging for mediastinal lymph node staging in patients with known or suspected non–small-cell lung cancer; enrolled at least 10 participants, including at least 5 participants with lymph node metastases; and provided enough data to permit calculation of sensitivity and specificity for identifying malignant lymph node involvement.

We adapted an existing instrument (11,13) to examine 7 aspects of study quality: technical quality of the index tests, technical quality and application of the reference test, independence of test interpretation, description of the study population, cohort assembly, sample size, and unit of analysis (Appendix Table 3).

We abstracted data about the demographic characteristics of participants, the prevalence of malignant lymph node involvement, and the sensitivity and specificity of CT and FDG-PET for identifying malignant lymph nodes. For studies that reported results by using the patient as the unit of analysis, we determined the ability of CT and FDG-PET to distinguish ipsilateral or contralateral mediastinal lymph node involvement (N2 or N3) from hilar, intrapulmonary, or no lymph node involvement (N0 or N1). This distinction is critical because involvement of N2 or N3 nodes usually indicates non-surgically treatable disease. When it was not possible to make this distinction, we determined test sensitivity and specificity for distinguishing N0 lymph node status from N1, N2, or N3 lymph node status. For studies in which the individual patient was not the unit of analysis, we determined the test sensitivity and specificity for identifying malignant lymph nodes or lymph node stations. Because observations are not independent when several lymph nodes from the same patient are analyzed separately, these studies may yield biased estimates of diagnostic test performance. Therefore, we analyzed data from these studies separately.

To determine whether the sensitivity and specificity of FDG-PET depended on the presence or absence of enlarged nodes on CT, we recorded the results of FDG-PET, CT, and the reference test or tests for each patient. This enabled us to derive separate estimates for the sensitivity and specificity of FDG-PET in patients with and without lymph node enlargement on CT.

For each study, we constructed 2 × 2 contingency tables in which all participants were classified as having positive (N2 or N3) or negative (N0 or N1) results and as having or not having mediastinal lymph node involvement as determined by the reference test or tests. We calculated the true-positive rate (true-positive rate = sensitivity), the false-positive rate (false-positive rate = 1 − specificity), and the log odds ratio (log odds true-positive rate − log odds false-positive rate) for CT and FDG-PET. The log odds ratio is a measure of diagnostic test performance that accounts for the correlation between the true-positive rate and the false-positive rate. We calculated exact 95% CIs for the true-positive rate and the false-positive rate on the basis of the binomial distribution (14).

To derive summary estimates of diagnostic test performance, we constructed summary receiver-operating characteristic (ROC) curves by using the method of Moses (12 - 13,15 - 16), which confirmed that the curves were symmetrical and could be described by a single parameter, the summary log odds ratio. Because this method requires the use of a correction factor when the reported sensitivity or specificity is 100%, we calculated the summary diagnostic odds ratios by using a fixed-effects model (17), or a random-effects model when there was evidence of heterogeneity (18), and reported results derived from these models. Because the summary log odds ratio is difficult to interpret clinically, we express our results in terms of the maximum joint sensitivity and specificity (12), a transformation of the summary log odds ratio that is a global measure of diagnostic accuracy, similar to the area under the ROC curve. The maximum joint sensitivity and specificity is the point on the summary ROC curve at which sensitivity and specificity are equal. It varies from 0.5 for a test that provides no diagnostic information to 1.0 for a test that is perfect.

We used meta-regression to make all statistical comparisons (19), with 1 exception. To compare the sensitivity and specificity of FDG-PET in patients with and without lymph node enlargement, we used discriminant function analysis (20) and a nonparametric permutation test (21). We considered a 2-sided P value less than 0.05 to be significant for all statistical tests.

In prespecified analyses, we examined the effect of year of publication, language, and individual aspects of study quality on the diagnostic accuracy of FDG-PET. We used meta-regression to compare groups of studies. To check for publication bias, we created inverted funnel plots of individual study log odds ratios plotted against sample size (22).

The funding source had no role in the collection, analysis, or interpretation of the data or in the decision to publish the manuscript.

Our initial search identified 570 studies, including 73 studies that were potentially relevant to mediastinal staging with FDG-PET (Figure 1). Of these, we excluded 29 studies because the number of participants was not sufficient (23 - 31), data necessary to permit calculation of sensitivity and specificity were not provided (25,28,32 - 46), or duplicate data were reported (32,47 - 51). Four additional studies were excluded because they evaluated FDG imaging with a modified γ camera (52 - 55). We also excluded 3 non-English-language papers that were review articles or case reports (56 - 58). Another study of mediastinal staging was excluded because almost one third of the participants had small-cell lung cancer or mesothelioma (59).

The supplemental search conducted for the Department of Veterans Affairs identified 508 studies, including 70 studies that were potentially relevant to mediastinal staging with FDG-PET (Figure 1). Of these, 6 studies that had not been identified previously were judged to be potentially eligible for inclusion and underwent detailed review. Three of these studies were excluded because they did not present enough data to permit calculation of sensitivity and specificity (60 - 62).

Thirty-nine studies met the inclusion criteria (63 - 101). Of these, 28 studies reported results by using the patient as the unit of analysis, 6 studies reported results by using lymph nodes or lymph node stations as the unit of analysis, and 5 studies reported results in both ways (Appendix Table 4). The median number of participants per study was 51 (range, 18 to 237). The mean age of participants was 56 to 69 years, and the median proportion of male participants was 64% (range, 48% to 99%). In studies that reported results by using the patient as the unit of analysis, the median prevalence of malignant lymph nodes was 32% (range, 5% to 64%). In studies that reported results by using lymph nodes or lymph node stations as the unit of analysis, the median prevalence of malignancy was lower (median, 16% [range, 7% to 37%]; P = 0.001). Ten studies provided usable data for FDG-PET but not for CT (73,85,89,91 - 93,96,98 - 99,101).

Because our criteria for assessing quality were stringent, no study met all of them. Seventeen studies (44%) met at least 70% of the 22 criteria on our study quality checklist (63 - 67,70 - 71,73 - 75,77,79,90,92,94,97 - 98). Five studies met fewer than 50% of the criteria (82,85,93,95,100). Appendix Table 4 shows selected aspects of methodologic quality for each study. In general, studies followed guidelines published by the Society of Nuclear Medicine for performing FDG-PET imaging. However, only 11 studies (28%) indicated that participants with hyperglycemia were excluded. Most studies adequately described the technical aspects of CT, although only 32% specified the use of spiral CT or an acquisition time of 2 seconds or less. While more than 90% of the studies required positive results from biopsy specimens to confirm mediastinal metastasis, only 47% required thoracotomy with systematic sampling of both normal- and abnormal-appearing lymph nodes at all accessible mediastinal stations to verify the absence of mediastinal involvement. In 56% of the studies, readers of FDG-PET and CT were blinded to the final diagnosis, and imaging tests were interpreted independently in less than half of the studies. Ninety percent of the studies enrolled a clinically relevant cohort of participants with known or suspected non–small-cell lung cancer, but participants were enrolled prospectively in only 51% of the studies. Characteristics of participants were completely described in just over half of the studies. Almost 45% of the studies enrolled at least 35 participants with lymph node metastases or 35 participants without lymph node metastases.

In studies that reported results by using the individual patient as the unit of analysis, FDG-PET was more accurate than CT for identifying mediastinal metastasis (P < 0.001). The median sensitivity and specificity of CT were 61% (interquartile range, 50% to 71%) and 79% (interquartile range, 66% to 89%), respectively (Figure 2, Table). The median sensitivity and specificity of FDG-PET were 85% (interquartile range, 67% to 91%) and 90% (interquartile range, 82% to 96%), respectively (Figure 3, Table).

The maximum joint sensitivity and specificity of CT was 70% (95% CI, 67% to 73%), indicating that diagnostic accuracy was only fair. Sensitivity was 59% (CI, 52% to 66%) at the point on the summary ROC curve that corresponded to the median specificity of 79% (Figure 4). Corresponding likelihood ratios for positive and negative CT results were 2.8 and 0.5, respectively.

The maximum joint sensitivity and specificity of FDG-PET was 86% (CI, 83% to 88%), indicating that diagnostic accuracy was very good. Sensitivity was 81% (CI, 74% to 86%) at the point on the summary ROC curve that corresponded to the median specificity of 90% (Figure 4). Corresponding likelihood ratios for positive and negative FDG-PET results were 8.1 and 0.2, respectively.

In studies that reported results by using lymph nodes or lymph node stations as the unit of analysis, the median sensitivity and specificity of CT were 62% (interquartile range, 52% to 65%) and 91% (interquartile range, 85% to 93%), respectively. The median sensitivity and specificity of FDG-PET were 83% (interquartile range, 73% to 89%) and 97% (interquartile range, 90% to 98%), respectively. Compared with studies that reported results by using the patient as the unit of analysis, these studies overestimated the diagnostic accuracy of both CT (P = 0.02) and FDG-PET (P = 0.04).

We estimated the sensitivity and specificity of FDG-PET for identifying mediastinal metastasis in patients with and without enlarged lymph nodes on CT on the basis of data from 14 studies (63 - 64,66 - 67,69 - 71,80 - 83,86 - 87,90). In these studies, the median prevalence of mediastinal metastasis was higher in patients with enlarged lymph nodes (63% [interquartile range, 51% to 69%]) than in patients who did not have lymph node enlargement (20% [interquartile range, 17% to 27%]; P < 0.001).

The maximum joint sensitivity and specificity of FDG-PET was similar in patients with and without lymph node enlargement on CT (P > 0.2). Accordingly, summary ROC curves for FDG-PET in patients with and without lymph node enlargement were almost identical (Figure 5). However, the presence or absence of enlarged lymph nodes appeared to influence the operating point on the ROC curves. When CT showed enlarged lymph nodes, median sensitivity was 91% (CI, 79% to 96%) at the point on the summary ROC curve that corresponded to the median specificity of 78% (Table). In contrast, when CT did not show enlarged nodes, sensitivity was 75% (CI, 59% to 87%) at the point on the summary ROC curve that corresponded to the median specificity of 93% (Table). Discriminant function analysis confirmed that the sensitivity and specificity of FDG-PET differed in the 2 groups of patients (P = 0.002)

The maximum joint sensitivity and specificity of CT was poor when FDG-PET results were positive (54% [CI, 45% to 63%]) or negative (59% [CI, 49% to 67%]). Thus, CT provided little diagnostic information once the results of FDG-PET were known.

Diagnostic accuracy was better in studies published before 1999 than for studies published after 1999 (P = 0.03). Likewise, diagnostic accuracy was better when studies reported that PET imaging was performed on patients in the fasting state (P = 0.02), but no other aspect of study quality affected test performance. Accuracy was similar in prospective and retrospective studies (P > 0.2), in English-language and non-English-language studies (P > 0.2), and in studies that did and did not meet at least 70% of our criteria for methodologic quality (P > 0.2). An inverted funnel plot did not raise suspicion of publication bias.

When the pretest probability of mediastinal metastasis is 35% and CT shows enlarged lymph nodes, estimated post-test probabilities when FDG-PET results are positive and negative are 86% (CI, 85% to 87%) and 13% (CI, 3% to 24%), respectively (Figure 6). If unconditional estimates of the sensitivity and specificity of FDG-PET had been used to make the calculations, post-test probabilities would have been 93% and 24% when FDG-PET results were positive and negative, respectively (Appendix Figure). When pretest probability is 35% and there is no lymph node enlargement, estimated post-test probabilities are 79% (CI, 72% to 83%) and 9% (CI, 4% to 14%) when FDG-PET results are positive and negative, respectively (Figure 6). Post-test probabilities based on unconditional estimates of sensitivity and specificity would have been 69% and 6%, respectively (Appendix Figure).

In this meta-analysis, we found that FDG-PET is more accurate than CT for mediastinal staging in patients with non–small-cell lung cancer. We estimate that, in current practice, the sensitivity and specificity of CT are approximately 59% and 79%, respectively. In comparison, the sensitivity and specificity of FDG-PET are approximately 81% and 90%, respectively. While FDG-PET is an important advance in noninvasive staging, it is not perfect. False-positive FDG-PET results have been reported in patients with postobstructive pneumonia and granulomatous disease. In addition, the spatial resolution of the current generation of PET scanners is approximately 7 mm. While it is possible for PET imaging to detect smaller lesions that are intensely hypermetabolic, the high false-negative rate (approximately 25%) in patients without lymph node enlargement on CT confirms that nodal size has an important effect on diagnostic accuracy. Another consequence of the limited spatial resolution of PET imaging is that the distinction between hilar (N1) and mediastinal (N2) lymph nodes, which has important implications for both treatment selection and prognosis, is sometimes difficult to make. Advances in PET technology, including the refinement of hybrid PET-CT scanners, may help to overcome some of these limitations in the future (102).

Although we identified considerable variability in study methods, only 3 factors affected PET performance. First, we found that diagnostic accuracy was better when studies reported that FDG-PET imaging was performed on patients in the fasting state. Second, we found that accuracy was better in studies published before 1999. Nevertheless, our estimates of sensitivity and specificity are similar to estimates previously reported in a meta-analysis by Dwamena and colleagues (103) who examined studies published before January 1998. As we stated previously in a meta-analysis of studies of FDG-PET for pulmonary nodule diagnosis, the lower accuracy observed in more recent studies could be due to enrollment of less highly selected patient samples or dissemination of PET technology to centers with less experience (13). Finally, we found that studies in which lymph nodes or lymph node stations were used as the unit of analysis tended to overestimate diagnostic accuracy, especially specificity. In these studies, observations are not statistically independent, that is, if a given patient has 1 positive lymph node, that patient is more likely to have other positive lymph nodes. In addition, it is important to note that the clinically relevant unit of analysis is the patient, not the lymph node. In general, treatment decisions depend on the presence or absence of lymph node involvement rather than the number of involved nodes. Because of these considerations, we recommend that future studies of tests for mediastinal staging report results by using the individual patient as the unit of the analysis.

To our knowledge, this is the first study to systematically evaluate the conditional test performance of FDG-PET and CT, although others have briefly addressed this issue using less rigorous methods (104 - 105). Summary ROC curves for FDG-PET in patients with and without enlarged mediastinal nodes were almost identical, suggesting that PET and CT results might be independent (Figure 5). However, FDG-PET appeared to operate at different points on the curves, depending on the presence or absence of lymph node enlargement. In patients with enlarged lymph nodes, FDG-PET operated near a point where sensitivity and specificity were 91% and 78%, respectively. In contrast, in patients without lymph node enlargement, FDG-PET operated near a point where sensitivity and specificity were 75% and 93%, respectively. In patients with enlarged lymph nodes, FDG-PET is more likely to reveal both true-positive findings that are due to metastasis and false-positive findings that are due to infection or inflammation, respectively. The increase in true-positive findings leads to higher estimated sensitivity, and the increase in false-positive findings results in lower estimated specificity. Conversely, FDG-PET is more likely to yield both true-negative and false-negative findings in patients without lymph node enlargement because of the test's inherent limitations in its ability to detect small hypermetabolic lesions of any origin. The increase in true-negative findings leads to higher estimated specificity, and the increase in false-negative findings results in decreased sensitivity.

Because the negative consequences of false-positive staging evaluations are so serious (missed opportunities for surgical cure), we believe that a positive FDG-PET result does not “rule in” disease with enough certainty unless pretest probability is very high (>85% to 90%) and that confirmatory mediastinal biopsy should be performed before excluding surgery as a treatment option. In fact, when CT shows lymph nodes that are accessible by transbronchial needle aspiration biopsy or endoscopic ultrasound-guided biopsy, these tests should be considered before PET imaging, especially if bronchoscopy is already planned for primary tumor diagnosis.

When FDG-PET results are negative, the decision to perform biopsy or surgery should be guided by the pretest probability of mediastinal metastasis, the presence or absence of lymph node enlargement, the risk for surgical complications, and patient preferences. While our results can help to inform clinical decision making, additional studies are needed to determine the threshold post-test probability below which surgery without previous mediastinal biopsy can be performed.

This study has several limitations. First, we did not attempt to identify all published studies of CT for mediastinal staging, only studies of FDG-PET that also reported results for CT. Thus, our estimates of diagnostic accuracy for CT may be biased. However, our estimates are similar to those reported in recent studies of CT for lymph node staging (1 - 3), as well as those reported in the meta-analysis by Dwamena and colleagues (103). We present the additional finding that CT provides little diagnostic information once the results of FDG-PET are known, confirming an observation made by Pieterman and colleagues (90) in a study conducted at a single center in the Netherlands. Second, it is possible that we did not identify all studies of FDG-PET for mediastinal staging, particularly unpublished studies. However, we conducted an exhaustive search for studies published in any language, and an inverted funnel plot did not support the hypothesis that several small “negative” studies were not identified. Finally, our estimates of diagnostic accuracy do not capture all of the potential benefits of staging with whole-body PET, which identifies unsuspected distant metastasis in approximately 10% of patients with otherwise resectable non–small-cell lung cancer (81,90).

We conclude that FDG-PET is more accurate than CT for mediastinal staging in patients with potentially resectable non–small-cell lung cancer and that the sensitivity and specificity of FDG-PET depend on the presence or absence of enlarged mediastinal lymph nodes on CT. Positive findings on PET imaging should be confirmed by biopsy before curative surgery is excluded as a treatment option. Negative findings on FDG-PET should be interpreted in light of the patient's pretest probability of mediastinal metastasis and whether CT reveals enlarged mediastinal nodes.

We performed the initial literature search in August 2001 and repeated searches of computerized databases in June 2002. These searches identified 570 potentially relevant studies, including 7 studies that did not appear in MEDLINE, CancerLit, or EMBASE (Figure 1). We excluded 497 studies after scanning their titles and abstracts: 123 review articles, meta-analyses, and cost-effectiveness analyses; 117 studies that examined FDG-PET for applications in thoracic oncology rather than mediastinal lymph node staging; 68 studies that examined FDG-PET for oncologic applications outside of lung cancer; 48 studies that focused on technical aspects of PET imaging; 25 case reports; and 116 other studies for miscellaneous reasons.

We subsequently evaluated 73 full reports for inclusion. Of these, we excluded 29 studies that did not enroll the required number of participants (23 - 31), that did not provide enough data to permit calculation of sensitivity and specificity (25,28,32 - 46), or that reported duplicate data (32,47 - 51). Four additional studies were excluded because they evaluated FDG imaging with a modified γ camera (52 - 55). We also excluded 3 non-English-language papers that were review articles or case reports (56 - 58). Another study of mediastinal staging was excluded because almost one third of the participants had small-cell lung cancer or mesothelioma (59).

The κ coefficient for inter-rater agreement for study eligibility was 0.64, indicating very good agreement for studies identified during the initial search.

More recently, 1 author participated in a technology assessment of FDG-PET imaging for the Department of Veterans Affairs. The technology assessment focused on the role of FDG-PET in managing patients with solitary pulmonary nodules, colon cancer, and non–small-cell lung cancer. A professional librarian searched MEDLINE, EMBASE, Current Contents, CancerLit (which is now defunct and rolled into MEDLINE), and BIOSIS by using Dialog, covering 1998 through January 2003. A full range of descriptors, text, keywords, and synonyms was used: tomography, emission computed, positron emission tomography, γ camera, PET, staging, solitary nodules, coin lesions, and colorectal cancer. The search retrieved 340 unique reports. A second search using various approaches was done on 27 March 2003 to minimize the possibility of poor retrieval due to indexing flaws. In another series of searches, we expanded retrieval by using the “related articles” link available only from PubMed and retrieved articles related to citations that were deemed highly relevant but had not been retrieved with the first search. The original search strategy was then edited and expanded by adding additional terms for PET and eliminating selected terms for study type and quality but keeping terms for predictive value of tests, sensitivity and specificity, and neoplasm staging. This second search retrieved an additional 181 citations along with many duplicates. In total, after eliminating duplications, 508 full records were retrieved for the Veterans Affairs' review.

Of these, 70 studies were judged to be potentially relevant to mediastinal staging with FDG-PET. Many were previously identified in the original literature review. One author evaluated 6 unique reports that were not identified previously. Three reports were excluded because they did not present enough data to permit calculation of sensitivity and specificity (60 - 62). In 1 of these studies, we could not calculate sensitivity and specificity because the final diagnosis was not reported for 6 participants with “indeterminate” results on FDG-PET imaging. Three studies from the supplemental search were included in the meta-analysis (91,99,101).

To assess study quality, we adapted criteria developed by Kent and colleagues (11), who evaluated imaging tests for the diagnosis of lumbar spinal stenosis. The revised criteria include 22 items that cover 8 dimensions of study quality: technical quality of FDG-PET, technical quality of CT, technical quality and application of the reference test or tests, independence of test interpretation, description of the study sample, cohort assembly, sample size, and unit of data analysis (Appendix Table 3). To develop criteria for the technical quality of FDG-PET, we consulted 2 nuclear medicine physicians experienced in FDG-PET imaging and referred to guidelines published by the Society of Nuclear Medicine (106). We used a 2-part definition to determine whether studies met criteria for adequacy of the reference test. To verify mediastinal lymph node involvement, we required confirmation by any type of biopsy. To verify the absence of mediastinal lymph node involvement, we required thoracotomy with systematic sampling of both normal- and abnormal-appearing lymph nodes at all accessible mediastinal stations. The median κ coefficient for inter-rater reliability was 0.67, indicating very good agreement.

For each study, we constructed 2 × 2 contingency tables in which all participants were classified as having positive (N2 or N3) or negative (N0 or N1) results and as having or not having mediastinal lymph node involvement, as determined by the reference test or tests. We calculated the true-positive rate (true-positive rate = sensitivity), the false-positive rate (false-positive rate = 1 − specificity), and the log odds ratio (log odds true-positive rate − log odds false-positive rate) for CT and FDG-PET. The log odds ratio is a measure of diagnostic test performance that accounts for the positive correlation between the true-positive rate and the false-positive rate. We added 0.5 to each cell in any 2 × 2 table that contained 1 or more zero values; otherwise, it would not be possible to compute the log odds ratio for studies that reported perfect sensitivity or specificity. We calculated exact 95% CIs for the true-positive rate and the false-positive rate on the basis of the binomial distribution (14).

We used several methods for constructing summary ROC curves. These methods have been described previously (13,15 - 16). The ROC curves illustrate the tradeoff between sensitivity and specificity, as the threshold that defines a positive test result varies from most stringent to least stringent. Our methods for constructing summary ROC curves depend on the assumption that individual study estimates of sensitivity and specificity represent unique points on a common ROC curve.

We used the method of Moses and colleagues (12) to test the hypothesis that the ROC curve is symmetrical and therefore can be described by a single parameter, the summary log odds ratio. In this method, the true-positive rate and false-positive rate are logistically transformed and simple linear regression is performed by using the log odds ratio as the dependent variable and an implied function of the test threshold (log odds true-positive rate + log odds false-positive rate) as the independent variable. As this function increases, the test threshold becomes less stringent. The slope of the regression equation indicates the degree to which the summary ROC curve is not symmetrical, while the intercept is a measure of diagnostic accuracy. A limitation of this method is that the logistic transformation requires the use of a correction factor when the reported sensitivity or specificity is 100%.

When the slope of the regression equation is not statistically significantly different from 0, the resulting ROC curve is symmetrical and can be described by the intercept, which is the summary log odds ratio. When this condition was met, we used a fixed-effects model for combining odds ratios when there was no evidence of heterogeneity (17) and a random-effects model when there was statistical evidence of heterogeneity (18), because these methods do not require the use of a correction factor (107). Nevertheless, all methods produced similar results.

For a global measure of test performance, we expressed our results in terms of the maximum joint sensitivity and specificity (12). The maximum joint sensitivity and specificity is the point on the summary ROC curve at which sensitivity and specificity are equal. It varies from 0.5 for a diagnostic test that has no diagnostic value to 1.0 for a diagnostic test that is perfect. Of note, the maximum joint sensitivity and specificity does not necessarily define the optimal operating point on the summary ROC curve but rather is a global measure of test performance, similar to the area under the curve. We calculated the maximum joint sensitivity and specificity by using the formula Q* = (1 +e^{− A/2}) ^{− 1}, where Q* is the maximum joint sensitivity and specificity and A is the summary log odds ratio (12).

To estimate the approximate sensitivity and specificity of FDG-PET and CT in current clinical practice, we selected a point on the summary ROC curve that corresponded to the median specificity in the individual studies (108). Other approaches are possible but not necessarily better. We selected the point that corresponded to the median specificity because the data were not normally distributed; specificity was less variable than sensitivity for FDG-PET; and if we had selected the point on the curve that corresponded to the median sensitivity, the estimated sensitivity and specificity of FDG-PET in patients with enlarged lymph nodes on CT would have been 100% and 0%, respectively. It is beyond the scope of this analysis to identify the optimal operating point on the summary ROC curve for FDG-PET. This question can be addressed by using decision analysis.

To compare the sensitivity and specificity of FDG-PET in patients with and without lymph node enlargement, we performed discriminant function analysis (20). This technique is useful for comparing groups on the basis of 1 or more attributes. Because sensitivity and specificity were not normally distributed and their covariances were not equal in patients with and without lymph node enlargement, we used a permutation test to obtain a P value rather than relying on normal theory (21). We computed the discriminant analysis statistic and its permutation distribution under the null hypothesis of no group difference, that is, we randomly assigned individual study estimates of sensitivity and specificity to the 2 groups and recomputed the statistic 500 times. One of 500 permutations resulted in an F statistic that was more extreme than the one observed (F = 15.03), resulting in a nonparametric P value of 0.002.

To estimate posterior (post-test) probabilities and their 95% CIs, we used the bootstrap (109) and resampled data from the individual studies 999 times. For each of the 999 bootstrap samples, we calculated the summary log odds ratio by using the Mantel-Haenszel method, the median specificity, the sensitivity at the point on the ROC curve that corresponded to the median specificity, likelihood ratios for positive and negative test results, and post-test probabilities for different values of pretest probability. To estimate 95% CIs for post-test probabilities, we assumed that pretest probabilities were not uncertain but incorporated the uncertainty in our estimates of sensitivity and specificity.

We used Microsoft Excel 2000 (Microsoft, Inc., Redmond, Washington) to estimate summary diagnostic odds ratios by using fixed- and random-effects models and to perform statistical tests for heterogeneity. We used JMP, version 3.2.6 (SAS Institute, Cary, North Carolina), to perform all regression analyses and SPSS for Windows, version 11.5 (SPSS, Inc., Chicago, Illinois), to perform discriminant function analysis. We used Microsoft Excel and Crystal Ball 2000 standard edition simulation software (Decioneering, Inc., Denver, Colorado) to generate bootstrap confidence intervals for post-test probabilities.

To determine whether particular study characteristics affected diagnostic accuracy, we used meta-regression. We performed multiple linear regression analysis by introducing additional covariates into the simple linear regression model described above, one at a time. Specifically, we tested the effect of each item in our study quality instrument, as well as language and year of publication. We also examined the effect of overall study quality by entering a variable that indicated whether the study satisfied at least 70% of our criteria for methodologic quality. In other analyses, we examined the effect of excluding studies that reported the sensitivity and specificity of FDG-PET for distinguishing no lymph node involvement (N0) from any lymph node involvement (N1, N2, or N3) and studies that enrolled fewer than 10 participants with and without mediastinal metastasis.

We used a chi-square test to identify heterogeneity in log odds ratios for studies that analyzed results by using the patient as the unit of analysis (18). There was no evidence of statistical heterogeneity in studies of CT (P > 0.2) or in studies of FDG-PET that reported results for patients with (P > 0.2) and without (P > 0.2) lymph node enlargement separately. Therefore, we used a fixed-effects model to derive estimates of diagnostic test performance for these data. Because we identified evidence of statistical heterogeneity in the log odds ratios of studies of FDG-PET (P = 0.05), we used a random-effects model to derive summary estimates of test performance for these studies.

We also inspected log odds ratios of individual studies to identify “outliers” that contributed to heterogeneity in the FDG-PET studies. Three studies were responsible for nearly 30% of the total heterogeneity (74,87,94). One high-quality study in 68 patients with a relatively high prevalence of mediastinal metastasis (41%) reported very high estimates of sensitivity (93%) and specificity (95%) (74). Another high-quality study of 81 participants with a lower prevalence of mediastinal metastasis (26%) reported poor sensitivity (52%) and intermediate specificity (88%) (94). A smaller study (22 participants) of average quality with a low prevalence of metastasis (27%) reported relatively low estimates for both sensitivity (67%) and specificity (69%) (87). Excluding these studies individually did not greatly affect statistical heterogeneity (P = 0.08 to 0.12), but excluding all 3 studies reduced the heterogeneity to a nonsignificant level (P > 0.2). Summary estimates of diagnostic accuracy were the same (maximum joint sensitivity and specificity, 86%) before and after excluding each of these studies individually and all 3 of them simultaneously.

We could not identify other sources of heterogeneity besides the year of publication and specifying whether participants underwent FDG-PET imaging in the fasting state. Of note, all 3 of the studies we mentioned earlier reported that participants underwent FDG-PET in the fasting state.

To examine diagnostic accuracy in studies that were least likely to be biased, we restricted the analysis to 9 studies of FDG-PET in which participants were enrolled prospectively, FDG-PET and CT readers were blinded to the results of the reference test or tests, and systematic sampling of both normal- and abnormal-appearing lymph nodes at all accessible mediastinal stations was used to exclude mediastinal metastasis (64,66 - 67,70,73 - 76,90). In this analysis, there was no evidence of statistical heterogeneity (P = 0.18), and diagnostic accuracy was slightly higher than our base-case estimate (maximum joint sensitivity and specificity, 89% [CI, 85% to 91%]), although the difference between these studies and those that did not meet all 3 criteria was not statistically significant (P = 0.10).

To illustrate how not using the patient as the unit of analysis can result in biased estimates of sensitivity and specificity, we consider 10 hypothetical patients with potentially resectable non–small-cell lung cancer (Appendix Tables 5 and 6). Each patient undergoes FDG-PET imaging and has normal- and abnormal-appearing lymph nodes sampled from 5 accessible mediastinal stations during thoracotomy. At thoracotomy, 4 patients have evidence of mediastinal metastasis at 1 of 5 lymph node stations; FDG-PET results are true-positive at the involved lymph node station in all 4 of these patients. One patient has metastasis at all 5 lymph node stations; FDG-PET results are false-negative at all 5 stations. Five other patients have no evidence of lymph node metastasis; FDG-PET results are true-negative at all 5 lymph node stations in 4 of the patients and are false-positive at 1 lymph node station in 1 patient. When the lymph node station is the unit of analysis, the calculated prevalence of mediastinal metastasis is 18%, sensitivity is 44% (4 true-positive lymph node stations among 9 stations with mediastinal metastasis), and specificity is 98% (40 true-negative lymph node stations among 41 stations without mediastinal metastasis). In contrast, when the individual patient is the unit of analysis, the calculated prevalence of mediastinal metastasis is 50%, sensitivity is 80% (4 patients with true-positive findings among 5 patients with mediastinal metastasis), and specificity is 80% (4 patients with true-negative findings among 5 patients without mediastinal metastasis). Thus, in this example, not using the patient as the unit of analysis overestimates specificity and underestimates disease prevalence and sensitivity.

We obtained estimates of diagnostic accuracy that were very similar to our base-case estimate when we restricted the analysis to studies that enrolled at least 10 patients with mediastinal metastasis and 10 patients without mediastinal metastasis (maximum joint sensitivity and specificity, 87% [CI, 85% to 88%]). We also obtained similar estimates when we restricted the analysis to studies that reported the sensitivity and specificity of FDG-PET for distinguishing N0 or N1 disease from N2 or N3 lymph node involvement (maximum joint sensitivity and specificity, 86% [CI, 84% to 88%]).