Improving visual grounding in remote sensing images with adaptive modality guidance

Abstract

This study explores the visual grounding (VG) task using data from remote sensing sensors, focusing on grounding phrases or prompts. We achieve this by employing masked feature encoding to create enriched, target-specific, and semantically aware visual representations. We aim to develop an efficient grounding architecture that uses language expressions as prompts for aerial images. Despite significant progress in phrase grounding within natural scenes, the vision community has yet to effectively translate these advancements to aerial imagery. This gap is mainly due to the lack of finely annotated remote sensing datasets and the substantial variation in the dimensional attributes of instances observed on Earth’s surface. We propose a novel framework, AMVG, for visual grounding, designed to enhance object localisation by leveraging multi-modal deformable attention encoding and adaptive decoding guided by language. In contrast to traditional methods that utilise region proposals or anchor-box-based predictions, our approach dynamically adapts attention to emphasise key spatial details and contextually relevant areas, enabling more accurate alignment of visual and textual features. Additionally, we propose a new training objective, Attention Alignment Loss (AAL), to address inconsistencies in attention allocation that can lead to suboptimal grounding accuracy. Experiments demonstrate that our method, Adaptive Modality-guided Visual Grounding (AMVG), significantly improves grounding accuracy compared to baseline methods, especially in challenging scenarios involving multiple textual and instance descriptions or ambiguous visual contexts. Our methodology outperforms recent methods, including MGVLF, CrossVG, VSMR, LPVA, and LQVG, achieving marginal improvements of 7.14%, 4.06%, 5.74%, 2.27%, and 3.35% in mean IoU, respectively. Similarly, for cumulative IoU, our approach surpasses MGVLF, VSMR, LPVA, and LQVG by 11.79%, 10.5%, 5.31%, and 3.14%, respectively. Similar trends are observed on the RSVG-HR and OPT-RSVG datasets, where AMVG demonstrates superior grounding accuracy in challenging scenarios involving complex textual prompts, ambiguous visual contexts, and diverse object attributes. These results reaffirm the effectiveness of our proposed method and its potential to address critical challenges in remote sensing VG tasks, setting a new benchmark in this domain. The full implementation of AMVG is publicly available https://github.com/Shabnamchoudhury/AMVG.git.