Spatio-temporal information mining and fusion feature-guided modal alignment for video-based visible-infrared person re-identification

Abstract

The video-based visible-infrared person re-identification (Re-ID) aims to recognize the same person across modalities through video sequences. The core challenges of this task lie in narrowing the modal differences and deeply mining the rich spatio-temporal information contained in video to enhance model performance. However, existing research primarily focuses on addressing the modality gap, with insufficient utilization of the spatio-temporal information in video sequences. To address this, this paper proposes a novel spatio-temporal information mining and fusion feature-guided modal alignment framework for video-based visible-infrared person Re-ID. Specifically, we propose a spatio-temporal information mining method. This method employs the proposed feature correlation mechanism to enhance the discriminative features of person across different frames, while utilizing a temporal Transformer to mine person motion features. The advantage of this method lies in its ability to alleviate issues such as occlusion and frame misalignment, improving the discriminability of person features. Additionally, we introduce a fusion modality-guided modal alignment strategy, which reduces modality differences between infrared and visible video frames by aligning single-modality features with fusion features. The advantage of this strategy is that each modality not only learns its specific features but also absorbs person information from the other modality, thereby alleviating modality differences and further enhancing the discriminability of person features. Extensive comparative and ablation experiments conducted on the HITSZ-VCM and BUPTCampus datasets confirm the effectiveness and superiority of the proposed framework. The source code is available at https://github.com/lhf12278/SIMFGA.