SceneVLP: Augmenting Vision-Language Pre-Training Models for Video Action Recognition With Video Scene Graphs

Abstract

Vision-language pre-training (VLP) models have demonstrated remarkable performance in video action recognition. However, their reliance on global video-text alignment often overlooks fine-grained scene details, leading to mis-recognition in visually similar contexts. To address this, we first explore the utilization of fine-grained scene information from video scene graphs (VSGs) to augment video action recognition. First, we introduce FVSG-LM, a frame-aware VSG generation method, which integrates the large model capabilities including MLLM and LLM and employs an inter-frame entity matching algorithm for unsupervised VSG generation. Second, we propose SceneVLP, an augmentation framework which improves VLP models by capturing key events from VSGs with the incorporation of a video key event generation component VKEG, to improve action recognition accuracy. We generate reusable VSGs for mainstream video action recognition datasets, including Kinetics-400, Something-Something-v2, HMDB51, and UCF101. Experimental results demonstrate that the SceneVLP framework, augmented with VSGs, achieves outstanding video action recognition performance, surpassing state-of-the-art methods of similar scale.