Multiclass classification of leukemia cancer subtypes using gene expression data and Optimized Dueling Double Deep Q-network

Abstract

Microarray technology aids in gene expression tracking, but diagnosing complex conditions like leukemia remains challenging due to multiple clinical factors. Deep reinforcement learning for cancer classification faces challenges related to optimization, handling high-dimensional noisy data, and interpretability. To address these limitations, this study proposes an Optimized Dueling Double Deep Q-Network (DDDQ-N) Framework, integrating advanced feature selection and DRL for robust leukemia subtype prediction. The framework begins with pre-processing, which includes data cleaning, normalization, and addressing class imbalance using the Synthetic Minority Over-sampling Technique (SMOTE). To enhance interpretability and reduce dimensionality, a novel Butterfly Optimization with Chaotic Local Search (BO-CLS) algorithm is introduced for feature selection, efficiently identifying the most discriminative genes. The selected features are then processed by a Dueling Double Deep Q-Network (DDQ-N), combining deep representation learning with reinforcement learning for sequential decision-making. The model employs a custom reward function and episode-based training to handle multi-class imbalance, adapt to tumor heterogeneity, and optimize classification strategies. Experimental results on a multi-class leukemia gene expression dataset demonstrate the model's superiority, achieving 99 % accuracy, 98.8 % precision, 99.2 % recall, and 99 % F1-score, outperforming existing methods such as Machine Learning (ML) Ensemble (94 %), Stacked Autoencoders with Grey Wolf Optimization (SAE-GWO) (98 %), and Feature Selective Neuro Evolution of Augmenting Topologies (FS-NEAT) (93 %). The proposed BO-CLS feature selection also shows significant improvements over ChisIG-SMOTE (95.5 % accuracy) and east Absolute Shrinkage and Selection Operator-Multi-Objective Genetic Algorithm (LASSO-MOGAT) (94.7 % accuracy), confirming its effectiveness in dimensionality reduction. These findings highlight the potential of the proposed framework to revolutionize leukemia diagnosis and provide a more efficient, interpretable, and accurate approach for clinical applications.