Spatial and frequency information fusion transformer for image super-resolution

Abstract

Previous works have indicated that Transformer-based models bring impressive image reconstruction performance in single image super-resolution (SISR). However, existing Transformer-based approaches utilize self-attention within non-overlapping windows. This restriction hinders the network’s ability to adopt large receptive fields, which are essential for capturing global information and establishing long-distance dependencies, especially in the early layers. To fully leverage global information and activate more pixels during the image reconstruction process, we have developed a Spatial and Frequency Information Fusion Transformer (SFFT) with an expansive receptive field. SFFT concurrently combines spatial and frequency domain information to comprehensively leverage their complementary strengths, capturing both local and global image features while integrating low and high-frequency information. Additionally, we utilize the overlapping cross-attention block (OCAB) to facilitate pixel transmission between adjacent windows, enhancing network performance. During the training stage, we incorporate the Fast Fourier Transform (FFT) loss, thereby fully leveraging the capabilities of our proposed modules and further tapping into the model’s potential. Extensive quantitative and qualitative evaluations on benchmark datasets indicate that the proposed algorithm surpasses state-of-the-art methods in terms of accuracy. Specifically, our method achieves a PSNR score of 32.67 dB on the Manga109 dataset, surpassing SwinIR by 0.64 dB and HAT by 0.19 dB, respectively. The source code and pre-trained models are available at https://github.com/Xufujie/SFFT