A multi-task deep learning pipeline integrating vessel segmentation and radiomics for multiclass retinal disease classification

Abstract

Objective

This study aims to develop a robust, multi-task deep learning framework that integrates vessel segmentation and radiomic analysis for the automated classification of four retinal conditions— diabetic retinopathy (DR), hypertensive retinopathy (HR), papilledema, and normal fundus—using fundus images.

Materials and Methods

A total of 2165 patients from eight medical centers were enrolled. Fundus images underwent standardized preprocessing including histogram equalization, normalization, resizing, and augmentation. Whole vessel and artery-vein segmentations were conducted using five deep learning models: U-Net, Attention U-Net, DeepLabV3+, HRNet, and Swin-Unet. From the segmented vascular maps, 220 radiomic features were extracted using PyRadiomics and Mahotas toolkits. The arteriovenous ratio (AVR) was also computed from artery and vein masks. ICC analysis was used to assess reproducibility across centers, with features below ICC < 0.75 excluded. Feature selection was performed using Least Absolute Shrinkage and Selection Operator (LASSO), Recursive Feature Elimination (RFE), and Mutual Information (MI). The combined AVR and radiomic features were input into four classifiers— Extreme Gradient Boosting (XGBoost), Categorical Boosting (CatBoost), Random Forest (RF), and Ensemble. Models were trained and validated on stratified splits and externally tested on an independent cohort of 769 patients. Evaluation metrics included accuracy, area under curve (AUC), recall, and receiver operating characteristics (ROC) analysis.

Results

Swin-Unet outperformed all models with external Dice Similarity Coefficient (DSC) of 92.4 % for whole vessel and 89.8 % for artery-vein segmentation. Classification using the LASSO-Ensemble combination achieved test accuracy of 93.7 %, external test accuracy of 92.3 %, and AUC of 95.2 %. AVR estimates were consistent with clinical expectations and contributed significantly to class discrimination.

Conclusion

This multi-task pipeline demonstrates the potential of combining transformer-based segmentation with radiomics for accurate, interpretable retinal disease classification, showing strong generalizability for future clinical applications.