ReranKGC: A cooperative retrieve-and-rerank framework for multi-modal knowledge graph completion

Abstract

Multi-modal knowledge graph completion (MMKGC) aims to predict missing links using entity’s multi-modal attributes. Embedding-based methods excel in leveraging structural knowledge, making them robust to entity ambiguity, yet their performance is constrained by the underutilization of multi-modal knowledge. Conversely, fine-tune-based (FT-based) approaches excel in extracting multi-modal knowledge but are hindered by ambiguity issues. To harness the complementary strengths of both methods for MMKGC, this paper introduces an ensemble framework ReranKGC, which decomposes KGC to a retrieve-and-rerank pipeline. The retriever employs embedding-based methods for initial retrieval. The re-ranker adopts our proposed KGC-CLIP, an FT-based method that utilizes CLIP to extract multi-modal knowledge from attributes for candidate re-ranking. By leveraging a more comprehensive knowledge source, the retriever generates a candidate pool containing entities not only semantically, but also structurally related to the query entity. Within this higher-quality candidate pool, the re-ranker can better discern candidates’ semantics to further refine the initial ranking, thereby enhancing precision. Through cooperation, each method maximizes its strengths while mitigating the weaknesses of others to a certain extent, leading to superior performance that surpasses individual capabilities. Extensive experiments conducted on link prediction tasks demonstrate that our framework ReranKGC consistently enhances baseline performance, outperforming state-of-the-art models.