Cross Knowledge Distillation for Image Super-Resolution

Abstract

Convolutional neural networks (CNNs) have achieved great success in super-resolution (SR) tasks. However, the huge computational requirements and memory footprint limit the practical deployment. Knowledge distillation (KD) allows tiny student networks to obtain performance improvement by learning from over-parameterized teacher networks, thus completing model compression. Previous work has attempted to solve the SR distillation problem by using feature-based distillation while ignoring the supervisory role of the teacher module itself. In this paper, we introduce a novel cross knowledge distillation framework to compress and accelerate super-resolution models. Specifically, we propose to cascade the student into the teacher network for directly utilizing the teacher's well-trained parameters, which avoids manually transforming the features between different networks for alignment. In addition, by instructing the student to learn similarity relations between the teacher layers, we force student learning representation in its own space rather than directly mimic the teacher's intricate features. We also make the teacher's parameters trainable, for alleviating the inefficient distillation problem caused by the representation gap between teacher and student. We conduct experiments at 3 scales on two typical super-resolution networks, EDSR and RCAN. The results show that the generated images obtain better visual improvements and have competitive advantages in metrics such as PSNR and SSIM.