FewRelEx: Exploring multimodal few-shot relation extraction with enhanced visual–textual mapping

Abstract

Few-shot relation extraction is a fundamental process for building knowledge graphs. However, existing methods often experience a notable degradation in performance when dealing with concise and noisy textual data, mainly due to the scarcity of ample contextual information within these texts. To overcome this limitation, we have designed a new multimodal few-shot relation extraction method that uses visual information to assist in identifying entity relationships within the text. The FewRelEx model consists of two key components: a semantic feature extraction module and a graph structure alignment module. The semantic feature extraction module is responsible for extracting semantic information from both text and images, including the global features of the image and the local features of the objects within the image. The graph structure alignment module is responsible for mapping the visual relationships between local objects to the textual relationships between entities in the sentence. We conducted in-depth experiments on two public datasets, and the results show that by introducing visual information, the FewRelEx model significantly improves the accuracy of relationship prediction in few-shot scenarios, effectively complementing the deficiencies in textual information.