Layer-wise correlation and attention discrepancy distillation for semantic segmentation

Knowledge distillation (KD) has recently garnered increased attention in segmentation tasks due to its effective balance between accuracy and computational efficiency. Nonetheless, existing methods mainly rely on structured knowledge from a single layer, overlooking the valuable discrepant knowledge that captures the diversity and distinctiveness of features across various layers, which is essential for the KD process. We present Layer-wise Correlation and Attention Discrepancy Distillation (LCADD) to tackle this issue, training compact and accurate semantic segmentation networks by considering layer-wise discrepancy knowledge. Specifically, we employ two distillation schemes: (i) correlation discrepancy distillation, which constructs a pixel-wise correlation discrepancy matrix across various layers to seize more detailed spatial dependencies, and (ii) attention discrepancy self-distillation, which aims to guide the shallower layers of the student network to emulate the attention discrepancy maps of the deeper layers, facilitating self-learning of attention discrepancy knowledge within the student network. Each proposed method is designed to work collaboratively in learning discrepancy knowledge, allowing the student network to better imitate the teacher from the perspective of layer-wise discrepancy. Our method has demonstrated superior performance on various semantic segmentation datasets, including Cityscapes, Pascal VOC 2012, and CamVid, compared to the latest knowledge distillation techniques, thereby validating its effectiveness.