Contrastive Deep Supervision Meets self-knowledge distillation

Abstract

Self-knowledge distillation (Self-KD) creates teacher–student pairs within the network to enhance performance. However, existing Self-KD methods focus solely on task-related knowledge, neglecting the importance of task-unrelated knowledge crucial for the intermediate layer’s learning. To address this, we propose Contrastive Deep Supervision Meets Self-Knowledge Distillation (CDSKD), a technique enabling the learning of task-unrelated knowledge to aid network training. CDSKD initially incorporates an auxiliary classifier into the neural network for Self-KD. Subsequently, an attention module is introduced before the auxiliary classifier’s feature extractor to fortify original features, facilitating extraction and classification. A projection head follows the extractor, and the auxiliary classifier is trained using contrastive loss to acquire task-unrelated knowledge, i.e., the invariance of diverse data augmentation, thereby boosting the network’s overall performance. Numerous experimental results on six datasets and eight networks have shown that CDSKD outperforms other deep supervision and Self-KD methods.