Empowering early predictions: A paradigm shift in diabetes risk assessment with Deep Active Learning

Abstract

Diabetes is one of the most widespread chronic diseases worldwide, affecting millions and posing significant health risks. Effective management depends on early detection and risk assessment, enabling medical professionals to take timely action and mitigate long-term healthcare consequences. However, there is a critical need for a reliable and accurate detection system to support medical professionals in clinical and computational assessments. Existing detection systems, particularly those based on traditional deep learning models, often fail to address challenges such as class imbalance, the inability to model non-linear patterns, high annotation costs, and the lack of explainability inherent in black-box models. To address these challenges in diabetes prediction, this study applies the proximity weighted synthetic oversampling technique to resolve class imbalance issues in the Behavioral Risk Factor Surveillance System (BRFSS) dataset, ensuring a balanced representation of healthy and diabetic individuals. Subsequently, we propose a novel Diabetic Class-based Sampling Pointer Network (DCSPNetwork) for early diabetes prediction by assessing high-risk factors. The DCSPNetwork effectively captures non-linear patterns in the BRFSS dataset, reduces overfitting risks, and minimizes labeling costs. Experimental results demonstrate the superior performance of DCSPNetwork, achieving an improvement score of 5.88% in accuracy, 8.14% in precision, 9.76% in recall, 8.33% in F1-score, and 4.3% in area under the receiver operating characteristics curve score, and a remarkable decrease of 30.3% in log loss, 72.90% in training time, and 30.30% in inference time compared to its benchmark pointer network model. Using a 10-fold cross-validation approach, we verified the performance and generalizability of our DCSPNetwork, ensuring consistent results across various data splits and demonstrating its robustness. To further support the DCSPNetwork’s efficacy and dependability, statistical validation is carried out utilizing a t-test and a 95% confidence interval. We incorporated explainable artificial intelligence techniques, local interpretable model-agnostic explanations, and Shapley additive explanations to enhance interpretability and transparency in predictions. These techniques improved the reliability of our model and offered insightful information about feature contributions in DCSPNetwork’s predictions. The results indicate that DCSPNetwork is a reliable and effective model for early diabetes risk assessment, combining high performance with interpretability.