Hierarchical chat-based strategies with MLLMs for Spatio-temporal action detection

Abstract

Spatio-temporal action detection (STAD) in football matches is challenging due to the subtle, fast-paced actions involving multiple participants. Multimodal large language models (MLLMs) often fail to capture these nuances with standard prompts, producing results lacking the detailed descriptions needed to improve visual features. To address this issue, we propose a prompt strategy called Hierarchical Chat-Based Strategies (HCBS). Specifically, this strategy enables MLLMs to form a chain of thought (CoT), gradually generating content with increasingly detailed information. We conduct extensive experiments on three datasets: 126 videos from Multisports, 43 videos from J-HMDB, and 147 videos from UCF101-24, all focus on the football sections. Compared to baseline tasks, our method improves performance by 30.3%, 26.1%, and 25.5% on these three datasets, respectively. Through the experiment of Hierarchy Verification, we demonstrate that HCBS effectively guides MLLMs in generating hierarchical descriptions. Additionally, using HCBS to guide MLLMs in content generation, we create a frame-level description dataset with 120,511 frame descriptions across the three datasets. Our code and dataset are available at the following link: https://github.com/TristanAlkaid/HCBS/.