IAMTrack: interframe appearance and modality tokens propagation with temporal modeling for RGBT tracking

Abstract

RGBT tracking has emerged as a robust solution for various applications, including surveillance, autonomous driving, and robotics, owing to its resilience in challenging environments. However, existing RGBT tracking approaches often overlook target appearance changes, location shifts, and the dynamic significance of modality features, limiting long-term tracking accuracy. To address these limitations, we propose IAMTrack, a novel transformer-based framework that achieves sequential tracking by propagating modality and appearance tokens across frames. The method compresses the discriminative features of each modality into modality tokens to transmit modality quality and target location information in real time, allowing the model to focus more on features with high modality quality and features with high target probability, while suppressing noise and redundant information. It also compresses the appearance features of objects similar in appearance across frames into appearance tokens to convey changes in appearance. To further enhance the token learning capability, we design a temporal generalized relation modelling approach that guides future predictions based on past information. The experimental results show that IAMTrack outperforms existing methods in various RGBT tracking scenarios, especially in UAV tracking tasks. Compared with those of previous methods, the MPRs and MSRs of the VTUAV short-term and long-term subdatasets are improved by \(1.7\%/2.1\%\) and \(2.5\%/2.2\%\), respectively.