HI4HC and AAAAD: Exploring a hierarchical method and dataset using hybrid intelligence for remote sensing scene captioning

Abstract

Remote sensing scene captioning is crucial for the deep understanding and intelligent analysis of Earth observation data. Many existing methods and datasets lack a fine-grained description of key geographical elements, fail to capture the full diversity of spatial relations, and are limited in their applicability to real-world geospatial scenarios. To address these shortcomings, we propose HI4HC (hybrid intelligence for remote sensing scene hierarchical captioning), a novel method that combines deep learning algorithms with expert knowledge to generate hierarchical captions for remote sensing scenes. This approach comprehensively describes scenes across three dimensions: geographical elements, spatial relations, and scene concepts, resulting in more accurate, detailed, and comprehensive captions. Leveraging HI4HC, we have constructed and made public a high-quality hierarchical caption dataset named AAAAD (adopt-amend-annihilate-add dataset). Extensive experiments show that AAAAD outperforms traditional single-level caption datasets in terms of the richness of geographical elements, the precision of spatial relations, and overall caption diversity, with improvements observed across 11 out of 13 evaluation metrics. Moreover, the hierarchical captions generated by HI4HC offer users the flexibility to organize information according to specific application needs such as imagery classification, change detection, multimodal understanding and cross-modal retrieval. This adaptability not only alleviates the semantic gap in imagery understanding but also plays an important role in advancing intelligent analysis of remote sensing imagery. AAAAD can be accessed through https://github.com/jaycecd/HI4HC.