TACMT: Text-aware cross-modal transformer for visual grounding on high-resolution SAR images

Abstract

This paper introduces a novel task of visual grounding for high-resolution synthetic aperture radar images (SARVG). SARVG aims to identify the referred object in images through natural language instructions. While object detection on SAR images has been extensively investigated, identifying objects based on natural language remains under-explored. Due to the unique satellite view and side-look geometry, substantial expertise is often required to interpret objects, making it challenging to generalize across different sensors. Therefore, we propose to construct a dataset and develop multimodal deep learning models for the SARVG task. Our contributions can be summarized as follows. Using power transmission tower detection as an example, we have built a new benchmark of SARVG based on images from different SAR sensors to fully promote SARVG research. Subsequently, a novel text-aware cross-modal Transformer (TACMT) is proposed which follows DETR’s architecture. We develop a cross-modal encoder to enhance the visual features associated with the textual descriptions. Next, a text-aware query selection module is devised to select relevant context features as the decoder query. To retrieve the object from various scenes, we further design a cross-scale fusion module to fuse features from different levels for accurate target localization. Finally, extensive experiments on our dataset and widely used public datasets have demonstrated the effectiveness of our proposed model. This work provides valuable insights for SAR image interpretation. The code and dataset are available at https://github.com/CAESAR-Radi/TACMT.