Melanoma Detection through Combining Reinforcement Learning, Generative Adversarial Network, and Bayesian Optimization

Abstract

Melanoma, a highly aggressive form of skin cancer, is primarily driven by DNA alterations often linked to environmental factors such as ultraviolet radiation. Addressing the need for improved early detection, this study tackles the key limitations of current methods, which frequently employ convolutional neural networks (CNNs) but struggle with feature selection, class imbalance, hyperparameter tuning, and generalizability. Our strategy leverages dilated convolution (DC) layers trained using reinforcement learning (RL). Unlike other RL-based approaches that handle these challenges in isolation, our method introduces a multi-stage architecture. It integrates RL for feature selection and class balancing. Shapley additive explanations (SHAP) guide feature identification, while augmented rewards for underrepresented classes help mitigate data imbalance. Bayesian optimization hyperband (BOHB) is used for hyperparameter tuning in a unified training process. BOHB combines the predictive strength of Bayesian optimization with the efficiency of hyperband, accelerating model tuning. It also includes an online GAN module for dynamic data augmentation that responds to the evolving output of the RL agent. A novel regularization technique stabilizes GAN training and prevents mode collapse. Importantly, existing RL methods face the challenge of balancing exploration and exploitation. In our RL model, the scope loss function (SLF), integrated with RL, balances exploration and exploitation, thereby ensuring accuracy and generalizability. Collectively, the model jointly tackles four persistent challenges in earlier RL-based approaches: poor exploration–exploitation balance, unstable reward dynamics, static data augmentation, and manual hyperparameter tuning. The model achieved F-measures of 94.3 %, 93.7 %, and 91.5 % on ISIC-2020, HAM10000, and PH2, respectively. This advancement significantly improves early melanoma detection and supports more accurate treatment decisions, contributing valuably to the ongoing effort to combat this lethal cancer.