MV-guided deformable convolution network for compressed video action recognition with P-frames

Abstract

Large-scale deep models have driven substantial progress in action recognition, but their heavy computation and the use of full-resolution RGB frames raise latency and privacy concerns. Compressed-domain methods reduce overhead by operating on codec outputs (I-frames, P-frames) but still rely on privacy-sensitive I-frames and incur nontrivial decoding costs. To overcome these limitations, we propose a novel P-frame only framework that (1) employs deformable convolutions to exploit the spatial sparsity of residual maps in P-frames and (2) introduces a Motion Vector–guided Deformable Convolution Network (MV-DCN) that uses motion vectors to predict adaptive sampling offsets. To transfer semantic knowledge from RGB features without decoding I-frames, we further design a Motion-Appearance Mutual Learning (MA-ML) scheme for cross domain distillation. Extensive experiments demonstrate that our model achieves competitive accuracy and speed compared to raw domain and traditional compressed domain approaches, while effectively preserving privacy by utilizing only P-frames.