Skin cancer detection using lightweight model souping and ensembling knowledge distillation for memory-constrained devices

Abstract

In contemporary times, the escalating prevalence of skin cancer is a significant concern, impacting numerous individuals. This work comprehensively explores advanced artificial intelligence-based deep learning techniques for skin cancer detection, utilizing the HAM10000 dataset. The experimental study fine-tunes two knowledge distillation teacher models, ResNet50 (25.6M) and DenseNet161 (28.7M), achieving remarkable accuracies of 98.32% and 98.80%, respectively. Despite their notable accuracy, the training and deployment of these large models pose significant challenges for implementation on memory-constrained medical devices. To address this issue, we introduce TinyStudent (0.35M), employing knowledge distillation from ResNet50 and DenseNet161, yielding accuracies of 85.45% and 85.00%, respectively. While TinyStudent may not achieve accuracies comparable to the teacher models, it is 82 and 73 times smaller than DenseNet161 and ResNet50, respectively, implying reduced training time and computational resource requirements. This significant reduction in the number of parameters makes it feasible to deploy the model on memory-constrained edge devices. Multi-teacher distillation, incorporating knowledge from both models, results in a competitive student accuracy of 84.10%. Ensembling methods, such as average ensembling and concatenation, further enhance predictive performances, achieving accuracies of 87.74% and 88.00%, respectively, each with approximately 1.05M parameters. Compared to DenseNet161 and ResNet50, these lightweight ensemble models offer shorter inference times, suitable for medical devices. Additionally, our implementation of the Greedy method in Model Soup establishes an accuracy of 85.70%.