Spatially-Aware Video Frame Interpolation for Viewpoint-Varying Sequences

Abstract

Video Frame Interpolation (VFI) is a technique for increasing the frame rate by predicting intermediate frames. Current algorithms typically estimate intermediate frame pixels using dense motion fields like optical flow. The rapid development of embodied vision and free-viewpoint video has introduced more viewpoint changes in videos, posing new challenges for traditional VFI algorithms. Viewpoint synthesis algorithms can learn spatial scene representations from multiple viewpoints to render intermediate viewpoint, providing better spatial information than VFI. However, viewpoint synthesis often introduces visual artifacts in occluded and exposed regions, which are unacceptable in VFI tasks. In contrast, the dense motion fields from traditional VFI can maintain global image quality. This paper proposes a viewpoint synthesis reference-based VFI method for viewpoint-varying videos. To the best of our knowledge, this is the first work to leverage the spatial information from viewpoint synthesis to assist VFI. The proposed method uses a viewpoint synthesis model to predict a multi-plane spatial representation of the scene from adjacent frames, thus rendering intermediate viewpoint. The rendered viewpoint then serves as spatial structure References for the VFI network, enabling the learning of high-quality interpolation results. Experimental results demonstrate that the proposed method can effectively handle the VFI for viewpoint-varying sequences. The source code is available at https://github.com/xiaorui0319/View-Synthesis-Reference-based-VFI