Prompt-Based Granularity-Unified Representation Network for Remote Sensing Image-Text Matching

Abstract

Remote sensing (RS) image–text matching has gained significant attention for its promising potential. Despite great advancements, accurately matching RS images (RSIs) and captions remains challenging due to the significant multimodal gap and inherent characteristics of RS data. Many approaches use complex models to extract global features to handle semantic redundancy and varying scales in RSIs, but losing important details in RSIs and captions. While some methods align between fine-grained local features, but overlooking the semantic granularity differences between fine-grained features. Fine-grained features in RSIs typically capture only a small fraction of the overall semantics, whereas those in captions convey more comprehensive and abstract semantics. Therefore, we propose the prompt-based granularity-unified representation network, an end-to-end framework designed to mitigate the multimodal semantic granularity difference and achieve comprehensive alignment. Our approach includes two key modules: 1) the prompt-based feature aggregator, which dynamically aggregates fine-grained features into several granularity-unified tokens with fully semantic, and 2) the text-guided vision modulation, which further enhances visual representations by modulating the visual features with RS captions as language typically contains more precise semantic than visual data. Furthermore, to address the challenges posed by high similarity in RS datasets, we introduce an effective hybrid cross-modal loss that facilitates comprehensive multimodal feature alignment within a unified structure. We conduct extensive experiments on three benchmark datasets, achieving state-of-the-art performance, which validates the effectiveness and superiority of our method.