A visual prompt learning network for hyperspectral object tracking

Abstract

Hyperspectral object tracking aims to achieve continuous tracking and localization of targets in a series of Hyperspectral images (HSIs) by analyzing and comparing the spectral and spatial features of the targets. Due to the relatively small size of hyperspectral object tracking datasets, existing strategies mainly rely on fine-tuning models initially trained on RGB images and then adapted them to hyperspectral data. However, the transferability of this comprehensive fine-tuning strategy is limited by the deficiencies in the data, resulting in suboptimal performance and limited results in hyperspectral object tracking. To address these challenges, we propose a visual prompt learning network for hyperspectral object tracking (VPH). In this approach, we freeze all the parameters of the model trained on RGB images and introduce a hyperspectral prompt module to efficiently transfer data-related information within HSIs to the RGB modality at a lower computational cost. In addition, we introduce an adapter module to adjust the frozen parameters of the RGB branch, ensuring fast adaptation to the hyperspectral tracking task. Our proposed network achieves the best performance in benchmark tests, validating the effectiveness of the proposed method. Our code and additional results are available at: https://github.com/972821054/VPH.git.