0

The full content of Annals is available to subscribers

Subscribe/Learn More  >
Original Research |

Predictive Accuracy of the Liverpool Lung Project Risk Model for Stratifying Patients for Computed Tomography Screening for Lung Cancer: A Case–Control and Cohort Validation Study

Olaide Y. Raji, PhD; Stephen W. Duffy, MSc; Olorunshola F. Agbaje, PhD; Stuart G. Baker, ScD; David C. Christiani, MD, MPH; Adrian Cassidy, PhD; and John K. Field, PhD, FRCPath
[+] Article and Author Information

From Roy Castle Lung Cancer Research Programme, The University of Liverpool, Liverpool, United Kingdom; Wolfson Institute of Preventive Medicine, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, King's College London, and Guy's Hospital, London, United Kingdom; National Cancer Institute, Bethesda, Maryland; and Harvard School of Public Health, Boston, Massachusetts.

Acknowledgment: The authors thank all study participants, the EUELC Consortium, Dr. Andrew J. Vickers for his useful discussion and helpful comments during the statistical data analysis and preparation of the manuscript, and Professor Anne Field for reading the manuscript as a nonexpert clinician. For members of the EUELC Consortium, see the Appendix.

Grant Support: By the Roy Castle Lung Cancer Foundation, the National Institute for Health Research Health Technology Assessment program, and the American Cancer Society, as well as grants CA74386, CA092824, and CA090578 from the National Cancer Institute, National Institutes of Health (Dr. Christiani).

Potential Conflicts of Interest: Disclosures can be viewed at www.acponline.org/authors/icmje/ConflictOfInterestForms.do?msNum=M11-1994.

Reproducible Research Statement:Study protocol: Available from Professor Field (e-mail, J.K.Field@liv.ac.uk). Statistical code: Available from Dr. Raji (e-mail, O.Y.Raji@liv.ac.uk). Relative utility curves are available from Dr. Baker (e-mail, sb16i@nih.gov). Data set: LLCC and LLPC data are available from Professor Field (e-mail, J.K.Field@liv.ac.uk) on completion of a data transfer agreement. Harvard and EUELC data need separate approval for their release.

Requests for Single Reprints: John K. Field, PhD, FRCPath, Roy Castle Lung Cancer Research Programme, The University of Liverpool Cancer Research Centre, Institute of Translational Medicine, The University of Liverpool, Liverpool L3 9TA, United Kingdom; e-mail, J.K.Field@liv.ac.uk.

Current Author Addresses: Dr. Raji and Professor Field: Roy Castle Lung Cancer Research Programme, The University of Liverpool Cancer Research Centre, Institute of Translational Medicine, The University of Liverpool, Liverpool L3 9TA, United Kingdom.

Professor Duffy: Wolfson Institute of Preventive Medicine, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, Charterhouse Square, London EC1M 6BQ, United Kingdom.

Dr. Agbaje: Division of Cancer Studies, Cancer Epidemiology Unit, King's College London, School of Medicine, Academic Oncology, 3rd Floor, Bermondsey Wing, Guy's Hospital, London SE1 9RT, United Kingdom.

Dr. Baker: National Cancer Institute, EPN 3131, 6130 Executive Boulevard MSC, Bethesda, MD 20892.

Dr. Christiani: Harvard School of Public Health, 665 Huntington Avenue, Boston, MA 02115.

Dr. Cassidy: GlaxoSmithKline Vaccines, Parc de la Noire Epine, Rue Fleming 20, 1300 Wavre, Belgium.

Author Contributions: Conception and design: S.W. Duffy, A. Cassidy, J.K. Field.

Analysis and interpretation of the data: O.Y. Raji, S.W. Duffy, O.F. Agbaje, S.G. Baker, D.C. Christiani, A. Cassidy, J.K. Field.

Drafting of the article: O.Y. Raji, O.F. Agbaje, A. Cassidy, J.K. Field.

Critical revision of the article for important intellectual content: O.Y. Raji, S.G. Baker, D.C. Christiani, A. Cassidy, J.K. Field.

Final approval of the article: O.Y. Raji, S.W. Duffy, O.F. Agbaje, D.C. Christiani, A. Cassidy, J.K. Field.

Provision of study materials or patients: D.C. Christiani, J.K. Field.

Statistical expertise: S.W. Duffy, S.G. Baker.

Obtaining of funding: D.C. Christiani, J.K. Field.

Administrative, technical, or logistic support: D.C. Christiani, J.K. Field.

Collection and assembly of data: O.Y. Raji, A. Cassidy, J.K. Field.


Ann Intern Med. 2012;157(4):242-250. doi:10.7326/0003-4819-157-4-201208210-00004
Text Size: A A A

Background: External validation of existing lung cancer risk prediction models is limited. Using such models in clinical practice to guide the referral of patients for computed tomography (CT) screening for lung cancer depends on external validation and evidence of predicted clinical benefit.

Objective: To evaluate the discrimination of the Liverpool Lung Project (LLP) risk model and demonstrate its predicted benefit for stratifying patients for CT screening by using data from 3 independent studies from Europe and North America.

Design: Case–control and prospective cohort study.

Setting: Europe and North America.

Patients: Participants in the European Early Lung Cancer (EUELC) and Harvard case–control studies and the LLP population-based prospective cohort (LLPC) study.

Measurements: 5-year absolute risks for lung cancer predicted by the LLP model.

Results: The LLP risk model had good discrimination in both the Harvard (area under the receiver-operating characteristic curve [AUC], 0.76 [95% CI, 0.75 to 0.78]) and the LLPC (AUC, 0.82 [CI, 0.80 to 0.85]) studies and modest discrimination in the EUELC (AUC, 0.67 [CI, 0.64 to 0.69]) study. The decision utility analysis, which incorporates the harms and benefit of using a risk model to make clinical decisions, indicates that the LLP risk model performed better than smoking duration or family history alone in stratifying high-risk patients for lung cancer CT screening.

Limitations: The model cannot assess whether including other risk factors, such as lung function or genetic markers, would improve accuracy. Lack of information on asbestos exposure in the LLPC limited the ability to validate the complete LLP risk model.

Conclusion: Validation of the LLP risk model in 3 independent external data sets demonstrated good discrimination and evidence of predicted benefits for stratifying patients for lung cancer CT screening. Further studies are needed to prospectively evaluate model performance and evaluate the optimal population risk thresholds for initiating lung cancer screening.

Primary Funding Source: Roy Castle Lung Cancer Foundation.

Figures

Grahic Jump Location
Figure 1.

Distribution of predicted absolute risk for participants in the EUELC, Harvard, and LLPC studies.

EUELC = European Early Lung Cancer; LLPC = Liverpool Lung Project prospective cohort.

Grahic Jump Location
Grahic Jump Location
Appendix Figure 1.

Comparison of observed and predicted cases of cancer, by quartile of predicted risk in the LLPC data.

Improved model calibrations were observed at higher quartiles of predicted absolute risks, which demonstrates a better calibration of the model at risk probabilities at which screening might be initiated. LLPC = Liverpool Lung Project prospective cohort.

Grahic Jump Location
Grahic Jump Location
Figure 2.

Decision curves and relative utility curve of the LLP risk model compared with alternative strategies for lung cancer screening decisions.

The smoking duration for relative utility curves involves 4 categories, whereas it had to be restricted to 2 categories for decision curves for technical reasons. DCA = decision curve analysis; EUELC = European Early Lung Cancer; LLP = Liverpool Lung Project; LLPC = Liverpool Lung Project prospective cohort; RU = relative utility.

Grahic Jump Location
Grahic Jump Location
Appendix Figure 2.

Relative utility of the LLP risk model and smoking duration in the EUELC and Harvard data sets.

At high risk thresholds, the LLP model showed moderate relative utility compared with smoking duration in the EUELC and Harvard data sets. The good performance at higher predicted absolute risks (greater than the probability of disease) are relevant for a default strategy of screening no one. The smoking duration for relative utility curves involves 4 categories. EUELC = European Early Lung Cancer; LLP = Liverpool Lung Project.

Grahic Jump Location

Tables

References

Letters

NOTE:
Citing articles are presented as examples only. In non-demo SCM6 implementation, integration with CrossRef’s "Cited By" API will populate this tab (http://www.crossref.org/citedby.html).

Comments

Submit a Comment
Predictive Accuracy of the Liverpool Lung Project (LLP) Risk Model for Stratifying Patients for Computed Tomography Screening for Lung Cancer
Posted on September 12, 2012
Robert P Young, MD, Raewyn J Hopkins, MD, David E Midthun, MD
Drs. Young and Hopkins -Schools of Biological Science and Faculty of Medical and Health Sciences, Univ of Auckland, New Zealand. Dr. Midthun - Mayo Clinic, Rochester, Minnesota, USA.
Conflict of Interest: Financial/nonfinancial disclosures: RPY, and the funding of his research, has been supported by grants from the University of Auckland, Health Research Council of New Zealand and Synergenz BioSciences Ltd. DEM, has received payment for preparation of chapters on lung cancer screening for the American College of Physicians and royalties for a chapter on lung cancer in Up-to-Date.

TO THE EDITOR:

We agree with Raji et al.(1), that the selection of current and former smokers for computed tomographic (CT) screening for lung cancer should target those at greatest risk “to maximize the benefit-harm ratio” (2). While we concur that a multivariate approach to risk assessment is the best way to achieve this (3), we question whether the validated LLP model maximizes this benefit-harm ratio as suggested.

Although the LLP multivariate model performs better than lung cancer risk models using age and smoking history alone (1), it is less clear that this superior performance translates into improved CT screening outcomes. In a recent opinion piece by Bach and Gould (4), it was strongly argued that screening should be limited to those at greatest risk so that the greatest number of deaths from lung cancer can be averted per person screened. Using the results of the National Lung Screening Trial (NLST), Bach et al. showed that the absolute number of deaths averted by screening is maximized when lung cancer detection rate (or death rate) is maximized. The study by Raji and colleagues (1) does not demonstrate a superior detection rate using the LLP model which the authors themselves have estimated to be only 1.0-1.5 fold that from using the NLST criteria (5). If the LLP model does not substantially increase the detection rate of lung cancer (1), it is difficult to see how it will maximize the benefit-harm ratio (4).

Given the LLP model uses more variables than the Bach model (also externally validated (4)), and achieves a higher area-under-the-curve performance characteristic (AUC), why then might the detection rate be no different to the NLST criteria that uses age and smoking history alone? It may be because the arbitrary cut off of 5% used for selection based on the LLP model is too low. Only validation in a CT screening study will clarify this issue. We have recently shown, using the Pittsburgh CT Screening Study, that when eligible smokers were stratified according to the presence of COPD, the lung cancer detection rate was 5 fold greater than in those without COPD (2,3). It is therefore possible that the absence of COPD in the LLP model may reduce its utility in distinguishing those most at risk of lung cancer among older heavy smokers. The role of “COPD-related” risk factors in improving lung cancer detection rate is currently under investigation in the NLST (2).

References
1. Raji OY, Duffy SW, Agbaje OF, Baker SG, Christianni DC, et al. Predictive accuracy of the Liverpool Lung Project risk model for stratifying patients for computed tomography screening for lung cancer: A case-control and cohort validation study. Ann Int Med 2012; 157: 242-250.

2. Young RP, Hopkins RJ.CT screening for lung cancer. Thorax 2012; 67: 650-651.

3. Young RP, Hopkins RJ. Diagnosing COPD and targeted lung cancer screening. Eur Respir J 2012; doi:10.1183/09031936.0070012.

4. Bach PB, Gould MK. When the average applies to no one: personalized decision making about potential benefits of lung cancer screening. Ann Int Med 2012, August 14.

5. Baldwin DR, Duffy SW, Wald NJ, Page R, Hansell DM, Field JK, et al. UK Lung Screen (UKLS) nodule management protocol: modeling of a single screen randomized control trial of low-dose CT screening for lung cancer. Thorax 2011; 66: 308-313.



Author's Response
Posted on October 12, 2012
John K. Field (PhD, FRCPath)1 ,Olaide Y. Raji (PhD)1, Stephen W. Duffy (MSc)2,
1.Roy Castle Lung Cancer Res Prgm, Univ of Liverpool Cancer Res Ctr, Inst. of Trsltnl Med, Univ of Liverpool. 2.Wolfson Inst of Preventive Med, Barts & London Sch of Med, Queen Mary Univ of Lndn
Conflict of Interest: None Declared

We thank Dr. Young and colleague for their interest and useful comments on our recent article (1). The LLP risk model was demonstrated to perform better than models based on age and smoking history alone. The superior net benefit shown in three independent studies suggested a predicted clinical utility in terms of screening decision at relevant thresholds; however this is a retrospective performance assessment and the actual effect can only be shown with prospective CT screening trials.

The detection rate of 1.5% reported in Baldwin et al 2011 (2) was a conservative estimate used to power the UKLS trial, and quite possibly be an underestimation. The question as to whether the LLP risk model achieves the objective of identifying a ‘high-risk‘ population, will only be proven when the UKLS CT screening trial is completed, with the necessary follow-up and the cost effectiveness of the approach has been demonstrated. Currently the academic community is still in the ‘modelling’ mindset and these discussions will continue on the clinical usefulness of lung cancer risk models, in particular, the LLP risk model until the UKLS trial has been completed.

We agree with Dr. Young and colleagues that COPD is an important independent risk factor for lung cancer, with a two to five fold increase in lung cancer reported among smokers. This strong independent relationship suggests a possible overlap in the biology of the two diseases and possibly sharing similar risk factor in smoking. Whilst, an independent association was found between COPD and lung cancer in the LLP dataset, it was not significant in the multivariable model (3). The lack of significant association of COPD may be due to reporting bias as diagnosis of COPD may have been under reported in the study area.

The LLP risk model has the advantage of simplicity of use and interpretation, and identifies subjects who are at high risk due to risk factors, other than smoking. The model would be most useful in population-based screening, where there is a low COPD prevalence.

References

1. Raji YR, Duffy SW, Agbaje OF, Baker SG, Christiani DC, Cassidy A, et al. Predictive Accuracy of the Liverpool Lung Project Risk Model for Stratifying Patients for Computed Tomography Screening for Lung Cancer. A Case–Control and Cohort Validation Study. Annals of Internal Medicine. 2012;21:128-38.

2. Baldwin DR, Duffy SW, Wald NJ, Page R, Hansell DM, Field JK. UK Lung Screen (UKLS) nodule management protocol: modelling of a single screen randomised controlled trial of low-dose CT screening for lung cancer. Thorax. 2011 Apr;66(4):308-13.

3. Cassidy A, Myles J, van-Tongeren M, Page R, Liloglou T, Duffy S, et al. The LLP risk model: an individual risk prediction model for lung cancer. Br J Cancer. 2008 Epub 2007 Dec 18;98((2)):270-6.

Submit a Comment

Summary for Patients

Clinical Slide Sets

Terms of Use

The In the Clinic® slide sets are owned and copyrighted by the American College of Physicians (ACP). All text, graphics, trademarks, and other intellectual property incorporated into the slide sets remain the sole and exclusive property of the ACP. The slide sets may be used only by the person who downloads or purchases them and only for the purpose of presenting them during not-for-profit educational activities. Users may incorporate the entire slide set or selected individual slides into their own teaching presentations but may not alter the content of the slides in any way or remove the ACP copyright notice. Users may make print copies for use as hand-outs for the audience the user is personally addressing but may not otherwise reproduce or distribute the slides by any means or media, including but not limited to sending them as e-mail attachments, posting them on Internet or Intranet sites, publishing them in meeting proceedings, or making them available for sale or distribution in any unauthorized form, without the express written permission of the ACP. Unauthorized use of the In the Clinic slide sets will constitute copyright infringement.

Toolkit

Buy Now

to gain full access to the content and tools.

Want to Subscribe?

Learn more about subscription options

Advertisement
Related Articles
Related Point of Care
Topic Collections
PubMed Articles
Forgot your password?
Enter your username and email address. We'll send you a reminder to the email address on record.
(Required)
(Required)