Interpretable lung cancer risk prediction using ensemble learning and XAI based on lifestyle and demographic data

Abstract

Lung cancer is a leading cause of cancer-related death worldwide. The early and accurate detection of lung cancer is crucial for improving patient outcomes. Traditional predictive models often lack the accuracy and interpretability required in clinical settings. This study aims to enhance lung cancer prediction accuracy using ensemble learning methods while integrating explainable AI (XAI) techniques to ensure model interpretability. Advanced ensemble learning techniques, such as Voting and Stacking, have been implemented to improve the predictive accuracy compared to traditional models. The models are implemented on three real lung cancer datasets, comprising lifestyle data of the patients, and assessed using various performance metrics, highlighting their reliability in clinical diagnosis. XAI methods are incorporated to ensure the models are interpretable, fostering trust among clinicians. SHAP (SHapley Additive exPlanations) values are utilized to identify and prioritize clinical and demographic factors influencing risk predictions. The ensemble models demonstrate superior performance metrics, significantly improving lung cancer prediction accuracy. Specifically, the Stacking ensemble model achieves the average prediction accuracy of 99.59 %, precision of 100 %, recall of 97.64 %, F1-score 98.65 %, AUC of 100 %, Kappa 98.40 %, and MCC of 98.44 % across three datasets. We employed the Friedman aligned ranks test and Holm post hoc analysis to validate performance, showing that the Stacking ensemble consistently outperformed others with higher accuracy and reliable predictions. Feature importance analysis reveals critical risk factors, providing insights into their interconnectivity and enhancing risk assessment frameworks. Integrating XAI techniques ensures the models are interpretable, promoting their potential adoption in clinical practices. The findings support the development of targeted interventions and effective risk management strategies, aiming to improve patient outcomes in lung cancer diagnosis and treatment.