CAFIN: cross-attention based face image repair network

Abstract

To address issues such as instability during the training of Generative Adversarial Networks, insufficient clarity in facial structure restoration, inadequate utilization of known information, and lack of attention to color information in images, a Cross-Attention Restoration Network is proposed. Initially, in the decoding part of the basic first-stage U-Net network, a combination of sub-pixel convolution and upsampling modules is employed to remedy the low-quality image restoration issue associated with single upsampling in the image recovery process. Subsequently, the restoration part of the first-stage network and the un-restored images are used to compute cross-attention in both spatial and channel dimensions, recovering the complete facial restoration image from the known repaired information. At the same time, we propose a loss function based on HSV space, assigning appropriate weights within the function to significantly improve the color aspects of the image. Compared to classical methods, this model exhibits good performance in terms of peak signal-to-noise ratio, structural similarity, and FID.