GAN Prior-Enhanced Novel View Synthesis From Monocular Degraded Images

Abstract

With the escalating demand for three-dimensional visual applications such as gaming, virtual reality, and autonomous driving, novel view synthesis has become a critical area of research. Current methods mainly depend on multiple views of the same subject to achieve satisfactory results, but there is often a significant lack of available data. Typically, only a single degraded image is available for reconstruction, which may be affected by occlusion, low resolution, or absence of color information. To overcome this limitation, we propose a two-stage feature matching approach designed specifically for single degraded images, leading to the synthesis of high-quality novel perspective images. This method involves the sequential use of an encoder for feature extraction followed by the fine-tuning of a generator for feature matching. Additionally, the integration of an information filtering module proposed by us during the GAN inversion process helps eliminate misleading information present in degraded images, thereby correcting the inversion direction. Extensive experimental results show that our method outperforms existing state-of-the-art single-view novel view synthesis techniques in handling challenges like occluded, grayscale, and low-resolution images. Moreover, the efficacy of our method remains unparalleled even when aforementioned method integrated with image restoration algorithms.