Knowledge distillation approach for skin cancer classification on lightweight deep learning model

Abstract

Over the past decade, there has been a global increase in the incidence of skin cancers. Skin cancer has serious consequences if left untreated, potentially leading to more advanced cancer stages. In recent years, deep learning based convolutional neural network have emerged as powerful tools for skin cancer detection. Generally, deep learning approaches are computationally expensive and require large storage space. Therefore, deploying such a large complex model on resource-constrained devices is challenging. An ultra-light and accurate deep learning model is highly desirable for better inference time and memory in low-power-consuming devices. Knowledge distillation is an approach for transferring knowledge from a large network to a small network. This small network is easily compatible with resource-constrained embedded devices while maintaining accuracy. The main aim of this study is to develop a deep learning-based lightweight network based on knowledge distillation that identifies the presence of skin cancer. Here, different training strategies are implemented for the modified benchmark (Phase 1) and custom-made model (Phase 2) and demonstrated various distillation configurations on two datasets: HAM10000 and ISIC2019. In Phase 1, the student model using knowledge distillation achieved accuracies ranging from 88.69% to 93.24% for HAM10000 and from 82.14% to 84.13% on ISIC2019. In Phase 2, the accuracies ranged from 88.63% to 88.89% on HAM10000 and from 81.39% to 83.42% on ISIC2019. These results highlight the effectiveness of knowledge distillation in improving the classification performance across diverse datasets and enabling the student model to approach the performance of the teacher model. In addition, the distilled student model can be easily deployed on resource-constrained devices for automated skin cancer detection due to its lower computational complexity.