Temporal Motion and Spatial Enhanced Appearance with Transformer for video-based person ReID

Abstract

For video-based person Re-Identification (Re-ID), how to efficiently extract temporal motion features and spatial appearance features from video sequences is a key issue. Conventional approaches focus on modelling the entire video spatio-temporal features, ignoring the inherent differences between temporal motion features (e.g., gait) that change over time and spatial appearance features (e.g., clothing) that are stable over time in terms of attributes. Because of their different sensitivities in real-world scenarios, conventional approaches often lose critical fine-grained features. To address these issues, we propose a Temporal Motion and spatial Enhanced Appearance with Transformer-based (T${}^2$MEA) framework for modelling spatial–temporal video discriminative representations. Specifically, (1) Dual-Branch Architecture: The content branch emphasises extracting the overall structure of the video using the spatial–temporal aggregation (STA) module from a global view, whereas the fovea branch focuses on gaining local fine-grained spatio-temporal features. (2) Zero-Parameter Design: the [CLS] Token Channel Shift Interaction (TCSI) module captures the dynamic features and static features between adjacent frames without additional parameters; the Spatial Patches Shift Enhancing (SPSE) module is introduced to enhance appearance features within frame to address occlusion and illumination changes without additional parameters. (3) Spatial–Temporal Interaction: The Cross-Attention Aggregation (CAA) module is proposed to interact between temporal and spatial features and further enrich the spatial–temporal feature representation for video sequences. Extensive experiments on three public Re-ID benchmarks (MARS, iLIDS-VID, and PRID-2011) demonstrate that the proposed framework outperforms several state-of-the-art methods.