Self-distillation salient object detection via generalized diversity loss

Classic knowledge distillation (KD) via the Kullback–Leibler loss can improve the performance of small deep classification models effectively, but they are hard to be applied into salient object detection (SOD) models due to the lack of necessary multi-dimension knowledge representations in the logit layer. In this paper, a generalized diversity (GD) loss, inspired by ensemble learning, is proposed to constrain the student and teacher models to hold low diversity. This process drives the student to mimic the teacher’s salient knowledge representations while enhancing the student’s generalization ability. Secondly, a salient self-distillation (SD) framework based on the shared backbone and the salient SD loss is proposed. In a shared backbone network, a lightweight student sub-network and a large parameter teacher sub-network are constructed, respectively, to synchronously achieve coarse but rapid feature extraction, and refined but slow feature extraction. The SD loss is utilized to transfer refined salient knowledge from the teacher sub-network to the student sub-network, so that the performance of the student sub-network is improved. Extensive experimental results on five benchmark datasets demonstrate that the proposed GD loss can achieve salient knowledge transfer and outperforms recent six KD methods, and the proposed student network outperforms recent eleven SOD networks in performance and efficiency.