AutoVMR: An autonomous event generation and localization approach for video moment retrieval

Abstract

Video Moment Retrieval (VMR) aims to identify a semantically relevant segment within a video based on a descriptive language query, specifying the segment's boundaries through start and end timestamps. Despite recent advancements, various VMR frameworks still rely heavily on extensive manual annotations, which are resource-intensive and not scalable for large-scale video databases. Besides, although large language model has been applied to VMR, it still suffers from sophisticated prompt design and multi-turn question answering, which is far from being automated. To address these issues, we propose AutoVMR, a novel multimodal large language model framework that employs an autonomous event generation and localization approach for VMR. AutoVMR utilizes an autoregressive architecture, accepting video input and a fixed prompt template, to generate event descriptions of video segments along with their corresponding start and end times. We also introduce a reward model based on Intersection over Union (IoU), trained using reinforcement learning from human feedback. This model is integrated into the Proximal Policy Optimization (PPO) training strategy and includes a query-time boundary generation mechanism to improve AutoVMR's performance. The reward model's modeling approach effectively filters out noise in the VMR dataset, enabling the PPO method to better comprehend video content and generate more accurate temporal localizations. Moreover, the integration of the autoregressive process with PPO training allows the model to be trained on unannotated video data, leading to improved performance in a semi-supervised setting. Experimental results demonstrate that AutoVMR outperforms traditional VMR methods and the latest multimodal large language models, achieving state-of-the-art performance.