KE-RSIC: Remote Sensing Image Captioning Based on Knowledge Embedding

Abstract

Current remote sensing image captioning methods often struggle to provide accurate and comprehensive descriptions due to their reliance on networks designed for natural images. Due to limited domain-specific knowledge in remote sensing, these networks often fail to accurately reflect the intrinsic semantic information of remote sensing categories. This article proposes a novel knowledge-embedded remote sensing image captioning model. We first define two types of remote sensing knowledge: general knowledge within the field of remote sensing, and specific knowledge that is relevant to the input image. To acquire general knowledge, we construct a remote sensing knowledge graph and propose a general knowledge embedding method, enabling semantic correlations between entities and relationships in remote sensing knowledge graphs. The generated entity embeddings and relationship embeddings can effectively capture the intrinsic semantic information of remote sensing categories. To acquire specific knowledge, we also propose a specific knowledge embedding method. We retrieve reports with similar label distributions to the input and then extract entities and relationships from the retrieved reports using a relation extractor. Embedding specific knowledge can alleviate to some extent the issue of poor matching between visual features and semantic features due to the lack of relevant knowledge. Subsequently, to integrate entity embeddings, relationship embeddings, and visual features, we propose a visual feature and knowledge information dynamic fusion module. This module can efficiently combine the visual features of remote sensing images with structural information on embedded knowledge. Numerous experimental findings attest to the superiority and effectiveness of the proposed method.