Beyond Static Prompts: Dynamic Token Fusion for Temporal-Align Visual--Language Tracking

Abstract

Visual-language (VL) tracking leverages natural language descriptions to enhance visual object tracking. However, a critical yet overlooked issue is the inherent temporal heterogeneity between static language descriptions and dynamic visual sequences, which limits the efficacy of cross-modal fusion. To bridge this gap, we propose a novel Temporal-Align Visual-Language Tracker (TAVLT). Unlike prior works that simply concatenate multimodal tokens, TAVLT processes template, search, and language features separately to explicitly model their interrelations. Our framework introduces two core components: 1) a Temporal Semantic Enrichment Module (TSEM) that constructs a Spatio-Temporal Semantic Affinity Matrix to dynamically recalibrate language token weights, ensuring temporal alignment with the visual sequence; and 2) a lightweight Cross-Modal Feature Fusion Module (CMFFM) that leverages the affinity matrices for efficient feature integration. Furthermore, we introduce a Consistency Learning Balance (CLB) strategy to mitigate performance disparities across different perceptual constancy tasks (e.g., shape vs. color). Extensive experiments on LaSOT, LaSOT-ext, TNL2K, and OTB99-Lang demonstrate that TAVLT achieves state-of-the-art performance while maintaining high inference efficiency (49 FPS). Our approach effectively addresses the temporal misalignment in VL tracking, setting a new standard for robust and efficient cross-modal learning.