Action-Prompt: A unified visual prompt and fusion network for enhanced video action recognition

Abstract

Video action recognition is a crucial task in video understanding and has garnered significant attention from researchers. However, while most existing methods exploit spatiotemporal and motion features for action recognition, these methods fail to consider that the fusion of different features cannot fully adapt to this task. To address this issue, we designed a prompt block named the Prompt Learning Layer (PLL), which is a plug-and-play module that can be inserted into a backbone to learn visual prompts for action recognition tasks. Additionally, we propose the Spatio-Temporal and Motion Fusion Module (STMF), which utilizes innovative extraction and fusion strategies to enhance the complementarity between the different features. The STMF comprises two main modules: the Bidirectional Motion Difference Module (BiMDM), which deals with bidirectional motion features, and the Spatio-Temporal Adaptive Module (STAM), which deals with spatio-temporal features in an adaptive approach. Finally, the experimental results demonstrate that our proposed method outperforms the state-of-the-art performance on the Kinetics-400, Something–Something V1 and V2 datasets.