Multiscale contextual joint feature enhancement GAN for semantic image synthesis

Abstract

Semantic image synthesis aims to generate images conditioned on semantic segmentation maps. Existing methods typically employ a generative adversarial framework to combine latent variables with semantic segmentation maps. However, traditional convolutions and feature map complexity often lead to issues such as uneven color, unrealistic textures, and blurred edges in generated images. To address these issues, we propose the Multiscale Contextual Joint Feature Enhancement Generative Adversarial Network, called MSCJ-GAN. Specifically, to capture local details and enhance the global consistency of large-scale objects, a large receptive field feature enhancement module based on Fast Fourier Convolution (FFC) and Transformer is introduced. This module employs an attention mechanism in the frequency domain, enabling neurons in the early layers of the network to access contextual information from the entire image. Furthermore, to ensure clear and realistic textures for objects and their boundaries, a dual-dimensional feature enhancement module based on bias is proposed. This module fully utilizes the statistical features in the feature maps, channel differences, and the detailed expression of the bias matrix to improve the realism of the generated images. Finally, experimental results on three challenging datasets demonstrate that the proposed MSCJ-GAN outperforms state-of-the-art methods, achieving superior performance in generating large-scale objects (e.g., sky and grass) and intricate texture details (e.g., wrinkles and micro-expressions). The code will be released after this work is published: https://github.com/xinxin0312/MSCJ-GAN.