Multiscale motion-aware and spatial–temporal-channel contextual coding network for learned video compression

Abstract

Video compression performance is significantly dependent on accurate motion prediction and efficient entropy coding. However, most current learned video compression methods rely on pre-trained optical flow networks or simplistic lightweight models for motion estimation, which fail to fully leverage the spatial–temporal characteristics of video sequences. This often brings higher bit consumption and distortion in reconstructed frames. Additionally, these methods frequently overlook the rich contextual information present within feature channels that could enhance entropy modeling. To address these issues, we propose a motion-aware and spatial–temporal-channel contextual coding-based video compression network (MASTC-VC). Specifically, we introduce a multiscale motion-aware module (MS-MAM) that estimates effective motion information across both spatial and temporal dimensions in a coarse-to-fine manner. We also propose a spatial–temporal-channel contextual module (STCCM) which optimizes entropy coding by exploiting latent representation correlations, leading to bit savings from spatial, temporal and channel perspectives. On top of it, we further introduce an uneven channel grouping scheme to strike a balance between computational complexity and rate–distortion (RD) performance. Extensive experiments demonstrate that MASTC-VC outperforms previous learned models across three benchmark datasets. Notably, our method achieves an average 10.15% BD-rate savings compared to H.265/HEVC (HM-16.20) using the PSNR metric and average 23.93% BD-rate savings against H.266/VVC (VTM-13.2) using the MS-SSIM metric.