Intra-frame scan-free video state spaces model for video moment retrieval

Abstract

With the increasing complexity of video moment retrieval tasks, effectively handling temporal and spatial information in video data has become a central challenge. This paper proposes a novel Intra-frame Scan-free Video State Spaces Model to address the spatiotemporal modeling problem in video moment retrieval. The model eliminates the dependency on the scanning order of intra-frame patches, overcoming the dual temporal limitations of frame order and within-frame patch sequence, which enhances the flexibility and efficiency of video understanding. To better model temporal information, we introduce the concept of video moment boundaries and propose the Weighted Relative Center Difference Loss, which ensures that the predicted center regions are closer to the ground truth, thereby improving retrieval accuracy. Extensive experiments on three public video datasets (ActivityNet Captions, TACoS, and Charades-STA) show that the model achieves superior or near-optimal performance across multiple metrics. The ablation study compares the performance loss when removing different components, the effect of different scanning methods on performance and inference throughput, and the effect of hyperparameters such as the number of SSM layers and the weighted relative centre difference loss threshold on retrieval performance. These results validate the effectiveness and robustness of our approach for video moment retrieval.