Incorporating prior knowledge and temporal memory transformer network for satellite video object tracking

Abstract

Satellite video object tracking (SVOT) faces more challenges compared to general video tracking, such as sparse target features, cluttered backgrounds, and frequent occlusion. Although numerous researchers have proposed solutions to address these challenges, SVOT still encounters three major issues. (1) Insufficient mining of temporal information: Most methods only utilize motion cues and dynamic templates as sources of temporal information. (2) Lack of robust solutions for frequent occlusion: Existing methods typically rely on threshold hyperparameters and employ Kalman filtering as the motion model, making it challenging to handle complex and long-term occlusion scenarios. (3) Underutilization of prior knowledge: Current methods typically employ cosine windows to suppress excessive displacement, but they neglect the kinematic patterns of targets in satellite videos. In order to address the above issues, we propose a method that incorporates prior knowledge and memory transformer network, namely MemTrack. The proposed memory module adaptively extracts and stores the relevant discriminative features of the target during the tracking phase, thereby further mining target-related temporal information and enhancing the model’s perception of the target. Based on prior knowledge and motion cues, we introduce an adaptive judgment strategy that identifies occlusion scenarios according to target size without relying on threshold hyperparameters, and we employ a linear regression approach as the motion model, which is both simple and effective in mitigating frequent occlusion issues. Additionally, we develop a biased 2D Gaussian window that indicates the target’s motion trend, thereby boosting tracker performance. MemTrack experiment in four large satellite video datasets, namely SatSOT, SV248S, OOTB and VISO respectively, achieving the best performance compared to the state-of-the-art (SOTA) trackers. On the SatSOT dataset, our tracker achieves an AUC score of 57.0%, marking the first time, to the best of our knowledge, that an AUC value has surpassed 55 without satellite video training on this dataset. The results demonstrate effectiveness and superiority of proposed method in SVOT. The project is available in https://github.com/jiawei-zhou/MemTrack.git, boosting progress of the SVOT.