I2CKD : Intra- and inter-class knowledge distillation for semantic segmentation

This paper proposes a new knowledge distillation method tailored for image semantic segmentation, termed Intra- and Inter-Class Knowledge Distillation (I2CKD). The key novelty lies in its dual focus on transferring both intra-class and inter-class knowledge between intermediate layers of the teacher (cumbersome model) and student (compact model). For knowledge extraction, we exploit class prototypes derived from feature maps. To facilitate knowledge transfer, we employ a triplet loss in order to minimize intra-class variances and maximize inter-class variances between teacher and student prototypes. Consequently, I2CKD enables the student to better mimic the feature representation of the teacher for each class, thereby enhancing the segmentation performance of the compact network. Extensive experiments on four segmentation datasets, i.e., Cityscapes, Pascal VOC, CamVid and ADE20K, using various teacher–student network pairs demonstrate the effectiveness of the proposed method.