Assessment and recalibration of seventeen lung cancer risk prediction models in approximately one million Chinese population utilising healthcare big data: a retrospective cohort analysis

Abstract

Background

A number of lung cancer prediction models have been developed worldwide. However, there have been limited validation studies conducted specifically on Chinese populations. The objective of this study is to evaluate the feasibility and performance of 17 global lung cancer risk prediction models when applied to Chinese healthcare big data.

Methods

The study encompassed individuals whose information was recorded in the Yinzhou Regional Health Care Database (YRHCD) between January 1, 2010 and December 31, 2021. The 17 lung cancer risk prediction models, which comprised the Bach, the Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial 2012 model (PLCO_m2012), the Korean Men, the PLCO_all2014, the Pittsburgh Predictor, Liverpool Lung Project Risk Prediction Model for Lung Cancer Incidence (LLPi), the Lung Cancer Risk Assessment Tool (LCRAT), Constrained LCRAT, the Nord-Trøndelag Health Study (HUNT), the Japan Public Health Center-based study (JPHC), Reduced HUNT, the PLCO_m2012 without information of race (PLCO_m2012-norace), the Liverpool Lung Project version 3 (LLPv3), Lung Cancer Risk Score (LCRS), the Optimized Early Warning Model for Lung Cancer Risk (OWL), the University College London-Incidence (UCL-I), the Shanghai Lung Cancer incidence Model (Shanghai-LCM), were evaluated for their performance in overall population and subgroups stratified by age and sex. The discrimination of the 17 models was assessed using Harrell's C-index and time-dependent area under the curve (AUC). The calibration of the models was evaluated using the expected-to-observed ratio (EOR) and calibration curves. Moreover, the models were recalibrated in the Yinzhou population, and the calibration of the recalibrated models was evaluated. For each model before and after recalibration, we redefined risk thresholds that would select the same number of individuals as the China National Lung Cancer Screening Guideline with Low-dose Computed Tomography 2023 Version (CNLCS 2023) could screen out. The Kaplan–Meier method was used to estimate the incidence and number of cases of lung cancer in individuals screened according to different criteria or models over a five-year follow-up period, and Kaplan–Meier survival curves were plotted.

Findings

A total of 904,667 study participants were included in the analysis, comprising 66,730 ever smokers and 837,937 never smokers. Among the 17 models initially considered, only six (Bach, Pittsburgh Predictor, JPHC, Reduced HUNT, Constrained LCRAT, UCL-I) had complete information of predictive variables available in the YRHCD. Most models showed similar levels of discrimination, with C-indices ranging from 0.78 (95% CI 0.74–0.82) to 0.88 (0.87–0.89) and time-dependent AUCs ranging from 0.74 (95% CI 0.73–0.75) to 0.88 (0.87–0.89). The majority of models showed an overestimation of incidence risk among ever smokers, with EORs ranging from 1.10 (95% CI 1.02–1.19) to 4.37 (4.16–4.58), and an underestimation among never smokers with a few models showing exceptions — EORs ranging from 0.12 (95% CI 0.11–0.14) to 1.30 (1.26–1.35). After recalibration, all models showed improved accuracy of predicted probability. The five-year incidence rates observed in the model-selected population, ranging from 0.81% (95% CI 0.64%–0.96%) to 1.29% (1.08%–1.48%), were consistently higher than that observed in the criteria-selected population (0.75%, 95% CI 0.59%–0.90%). Following recalibration, the five-year incidence rates in the model-selected population improved, ranging from 0.81% (95% CI 0.64%–0.96%) to 1.60% (1.36%–1.82%).

Interpretation

The majority of recalibrated models demonstrated comparable and favorable discrimination and calibration capability, and were capable of identifying individuals at an elevated risk of lung cancer with greater precision than the criteria. Models designed for the general population (such as LLPv3, LLPi, Korean Men, JPHC, and LCRS) are more appropriate for identifying high-risk groups compared to those exclusively for smokers.

Funding

National Natural Science Foundation of China, General Project of Zhejiang Provincial Medical and Health Technology Plan for the Year 2024, Natural Science Foundation of Zhejiang Province.