OTMKGRL: a universal multimodal knowledge graph representation learning framework using optimal transport and cross-modal relation

Abstract

The demand for integrating multimodal information, such as text and images, has grown significantly as it enables richer and more comprehensive knowledge representations. Most existing multimodal knowledge graph representation learning (KGRL) methods focus primarily on fusing multimodal entity information, directly applying multimodal entities and single-modal relations to downstream tasks. However, these methods face challenges related to the heterogeneity of multi-source entity data, which amplifies the differences in feature distributions between entity and relation representations. To address these challenges, we propose a universal multimodal KGRL framework, OTMKGRL, which seamlessly incorporates multimodal information into three types of single-modal KGRL methods. First, OTMKGRL employs Tucker decomposition to project entity text and image data into a shared space, thereby generating multimodal entity representations. It then uses optimal transport to integrate multimodal entity information into the original single-modal entity representations. Second, OTMKGRL introduces a cross-modal relation attention mechanism that fuses effective multimodal entity features into the original single-modal relations, yielding cross-modal relation representations. Extensive experiments across three multimodal datasets demonstrate the effectiveness and versatility of our approach. The OTMKGRL framework significantly enhances the performance of existing single-modal KGRL models in multimodal settings.