Beyond expression: Comprehensive visualization of knowledge triplet facts

Abstract

Multi-modal Knowledge Graphs (KGs) enhance traditional KGs by incorporating multi-modal data to bridge the information gap in natural language processing (NLP) tasks. One direct method to incorporate multi-modal data is to associate structured KG with corresponding image modalities, thereby visualizing entities and triplet facts. However, existing visualization methods for triplet facts often exclude triplet facts containing abstract entities and non-visual relations, resulting in their disassociation from corresponding image modalities. This exclusion compromises the completeness and utility of multi-modal KGs. In this paper, we aim to construct a comprehensive multi-modal KG that includes abstract entities and non-visual relations, ensuring complete visualization of every triplet fact. To achieve this purpose, we propose a method for the integration of image Retrieval-Generation-Editing (RGE) to completely and accurately visualize each triplet fact. Initially, we correct the triplet facts by integrating a Large Language Model (LLM) with a retrieved knowledge database about triplet facts. Subsequently, by providing appropriate contextual examples to the LLM, we generate visual elements of relations, enriching the semantics of the triplet facts. We then employ image retrieval to obtain images that reflect the semantics of each triplet fact. For those triplet facts for which images cannot be directly retrieved, we utilize image generation and editing to create and modify images that can express the semantics of the triplet facts. Through the RGE method, we construct a multi-modal KG named DB15kFact, which includes 86,722 triplet facts, 274 relations, 12,842 entities, and 387,096 images. The construction of DB15kFact has resulted in a fourfold increase in the number of relations compared to the previous multi-modal KG, ImgFact. In experiments, both automatic and manual evaluations confirm the quality of DB15kFact. The results demonstrate that the DB15kFact significantly enhances model performance in link prediction and relation classification. Notably, in link prediction, the model optimized with DB15kFact achieves a 7.12% improvement in the H@10 metric compared to existing solutions.