MgHiSal: MLLM-guided hierarchical semantic alignment for multimodal knowledge graph completion

Abstract

Multimodal Knowledge Graph Completion (MMKGC) aims to improve link prediction and empower downstream applications like intelligent question answering, reasoning, and recommendation by integrating multimodal information. However, existing methods commonly face semantic fragmentation between visual and textual modalities. This issue largely stems from the reliance on pre-trained vision models, which often struggle to model deep entity semantics, resulting in noisy visual features and the neglect of key semantic information. To address this, we propose MgHiSal, an MLLM-guided Hierarchical Semantic Alignment framework for MMKGC. MgHiSal first generates context-aware visual descriptions by conditioning MLLM generation on existing entity text, ensuring low-noise, relevant representations for initial semantic alignment. The framework then utilizes a hierarchical gated attention mechanism that progressively unifies multimodal representations by dynamically selecting and optimizing key cross-modal features via regularization. Finally, a neighbor-aware module enhances entity representations by aggregating multimodal neighbor information. Experiments on DB15K and MKG-W show MgHiSal significantly improves MRR by approximately 13.1 % and 12.5 % over respective runner-ups. The source code is publicly available at https://github.com/wyZhang016/MgHiSal.