Resampling video super-resolution based on multi-scale guided optical flow

Abstract

Existing video super-resolution (VSR) methods are inadequate for dealing with inter-frame motion and spatial distortion problems, especially in high-motion scenes, which tend to lead to loss of details and degradation of reconstruction quality. To address these challenges, this paper puts forward a resampling video super-resolution algorithm based on multiscale guided optical flow. The method combines multi-scale guided optical flow estimation to address the issue of inter-frame motion and a resampling deformable convolution module to address the issue of spatial distortion. Specifically, features are first extracted from low-quality video frames using a convolutional layer, followed by feature extraction with Residual Swin Transformer Blocks (RSTBs). In the feature alignment module, a multiscale-guided optical flow estimation approach is employed, which addresses the inter-frame motion problem across different video segments and performs video frame interpolation and super-resolution reconstruction simultaneously. Furthermore, spatial alignment is achieved by integrating resampling into the deformable convolution module, mitigating spatial distortion. Finally, multiple Residual Swin Transformer Blocks (RSTBs) are used to extract and fuse features, and pixel rearrangement layers are employed to reconstruct high-quality video frames. The experimental results on the REDS, Vid4, and UDM10 datasets show that our method significantly outperforms current state-of-the-art (SOTA) techniques, with improvements of 0.61 dB in Peak Signal-to-Noise Ratio (PSNR) and 0.0121 in Structural Similarity (SSIM), validating the effectiveness and superiority of the method.