An accurate and efficient self-distillation method with channel-based feature enhancement via feature calibration and attention fusion for Internet of Things

Abstract

With the rise of the Internet of Things (IoT), using convolutional neural networks (CNNs) for image tasks on edge devices has become prevalent, but the increased size and complexity of neural networks for better performance is not ideal for resource-limited embedded devices. Self-distillation, which does not need a pre-trained complex model, has been introduced to utilize knowledge distillation during the model’s own training, thus enhancing performance. However, the model accuracy and efficiency of current self-distillation techniques still need investigation to meet real-world demands in IoT scenarios. Therefore, this paper proposes an improved self-distillation with Channel-Based Feature Enhancement (CBFE) via feature calibration and attention fusion, which improves network performance with minimal extra load. In particular, we first propose a channel-based feature calibration module. This module uses 1x1 convolutions to reduce and then restore the channel dimension of the neural network output feature maps. For each input feature map, it generates a new feature map, which is then element-wise multiplied with the original feature map to enhance representation. Second, we introduce a channel attention-based feature fusion network branch that refines a more accurate feature representation to better guide the training of shallow layers of the network. Experimental results show that our method surpasses the state-of-the-art methods, demonstrating enhanced performance and generalization on various benchmarks.