Spatial-Temporal Consistency Refinement Network for Dynamic Point Cloud Frame Interpolation

Abstract

Point cloud frame interpolation aims to improve the frame rate of a point cloud sequence by synthesising intermediate frames between consecutive frames. Most of the existing works only use the scene flow or features, not fully exploring their local geometry context or temporal correlation, which results in inaccurate local structural details or motion estimation. In this paper, we organically combine scene flows and features to propose a two-stage network based on residual-learning, which can generate spatially and temporally consistent interpolated frames. At the Stage 1, we propose the spatial-temporal warping module to effectively integrate multi-scale local and global spatial features and temporal correlation into a fusion feature, and then transform it into a coarse interpolated frame. At the Stage 2, we introduce the residual-learning structure to conduct spatial-temporal consistency refinement. A temporal-aware feature aggregation module is proposed, which can facilitate the network adaptively adjusting the contributions of spatial features from input frames, and predict the point-wise offset as the compensations due to coarse estimation errors. The experimental results demonstrate our method achieves the state-of-the-art performance on most benchmarks with various interpolated modes. Code is available at https://github.com/renlancao/SR-Net.