Leveraging FastViT based knowledge distillation with EfficientNet-B0 for diabetic retinopathy severity classification

Abstract

Diabetic retinopathy (DR) remains a key contributor to eye impairment worldwide, requiring the development of efficient and accurate deep learning models for automated diagnosis. This study presents FastEffNet, a novel framework that leverages transformer-based knowledge distillation (KD) to enhance DR severity classification while reducing computational complexity. The proposed approach employs FastViT-MA26 as the teacher model and EfficientNet-B0 as the student model, striking the ideal mix between accuracy and computational efficiency. APTOS blindness detection dataset comprising 3662 images across five severity classes is collected, pre-processed, normalized, split and augmented to address class imbalance. The teacher model undergoes training and validation before transferring its knowledge to the student model, enabling the latter to approximate the teacher’s performance while maintaining a lightweight architecture. To comprehensively assess the efficacy of the proposed framework, additional student models—including HGNet, ResNet50, MobileNetV3, and DeiT—are analysed for comparative assessment. Model interpretability is enhanced through Grad-CAM++ visualizations, which highlight critical retinal regions influencing DR severity classification. Several measures are used to evaluate performance, including accuracy, precision, recall, F1-score, Cohen’s Kappa Score (CKS), Weighted Kappa Score (WKS), and Matthews Correlation Coefficient (MCC), ensuring a robust assessment. Among all student models, EfficientNet-B0 achieves the highest classification accuracy of 95.39 %, 95.43 % precision, recall of 95.39 %, F1-score of 95.37 %, CKS of 0.94, WKS of 0.97, MCC of 0.94, AUC of 0.99, and a KD loss of 0.17, with a computational cost of 0.38 G FLOPs. These results demonstrate its effectiveness as an optimized lightweight model for DR detection. The findings emphasize the potential of KD-based lightweight models in attaining high diagnostic accuracy while reducing computational complexity, paving the way for scalable and cost-effective DR screening solutions.