Transformer-Inspired Convolutional Network for Image Restoration

Abstract

Image restoration is a critical task in computer vision, as unwanted degradations significantly reduce image quality and adversely affect the robustness of downstream vision systems. Recently, Transformer-based models have demonstrated remarkable performance in image restoration tasks. However, their quadratic computational complexity with respect to input size limits their practical application. In this paper, we develop a computationally efficient convolutional image restoration model inspired by the design philosophy of Transformer blocks. Specifically, we emulate and unify the functionalities of self-attention and feed-forward layers within a single convolution-based module to achieve adaptive and efficient spatial feature learning. Additionally, this module incorporates multiple receptive fields at attention, module, and scale levels, enabling effective management of degradations of varying sizes. Furthermore, to mitigate frequency discrepancies between degraded and sharp image pairs, we widen and deepen the structure of a frequency learning mechanism, which implicitly refines frequency spectra through a simple yet powerful frequency decomposition and calibration approach. Integrating these designs into a U-shaped convolutional backbone results in our efficient and effective network, termed UIRNet, for image restoration. Extensive experiments across single-degradation, all-in-one, and composite degradation restoration tasks validate the effectiveness of our method. Notably, UIRNet achieves state-of-the-art performance on 11 benchmark datasets covering six single-degradation scenarios and performs favorably against leading all-in-one approaches under a five-degradation setting. Importantly, our method also demonstrates robust performance on two challenging composite degradation image restoration benchmarks, as well as in ultra-high-definition (UHD) applications.