VC-Mamba: Causal Mamba representation consistency for video implicit understanding

Abstract

Recently, spatiotemporal representation learning based on deep learning has driven the advancement of video understanding. However, existing methods based on convolutional neural networks (CNNs) and Transformers still face limitations in understanding implicit information in complex scenes, particularly in capturing dynamic changes over long-range spatiotemporal data and inferring hidden contextual information in videos. To address these challenges, we propose VC-Mamba, a video implicit understanding model based on causal Mamba representation consistency. By segmenting explicit texture information into token features and leveraging the linear Mamba framework to capture long-range spatiotemporal interactions, we introduce the spatiotemporal motion Mamba block for motion perception. This block includes a multi-head temporal length Mamba to enhance cross-frame motion consistency and a bidirectional gated space Mamba to capture the inter-frame dependencies of feature tokens. Through the analysis of both explicit and implicit spatiotemporal interactions, VC-Mamba effectively captures long-range spatiotemporal representations. Additionally, we design an attention mask perturbation strategy based on causal invariance constraints to optimize the existing selective spatiotemporal mask mechanism. By progressively enhancing the causal strength of related features, this strategy analyzes implicit causal chains in videos, improving the model’s resistance to interference from weakly causal features and enhancing the robustness and stability of implicit information understanding. Finally, we conducted extensive experiments on several datasets, including short-term action recognition and long-term video reasoning tasks. The results demonstrate that VC-Mamba matches or surpasses state-of-the-art models, particularly in capturing long-range spatiotemporal interactions and causal reasoning, proving its effectiveness and generalization in video implicit understanding tasks.