Dynamic link prediction: Using language models and graph structures for temporal knowledge graph completion with emerging entities and relations

Abstract

Knowledge graphs (KGs) represent real-world facts through entities and relations. However, static KGs fail to capture continuously emerging entities and relations over time. Temporal knowledge graphs address this by incorporating time information or providing multiple sequential snapshots of a static knowledge graph. Most existing work focuses on static KGs with fixed sets of entities and relations, meaning existing methods still struggle to encode emerging entities and relations. Therefore, we propose a novel methodology of combining language models and graph structure to enable the encoding of unseen entities and relations for temporal KG completion. Specifically, we encode relations with RoBERTa and entities using neighbouring relations alongside the entity’s relation type to provide contextual information. We evaluate our methodology on three datasets with emerging entities and relations over temporal snapshots: LKGE-Hybrid, FB-MBE, and the mergers and acquisitions domain TKGQA dataset. Our experiments show that our model achieves new state-of-the-art results on FB-MBE and LKGE-Hybrid while providing strong benchmark results for the TKGQA dataset. Our ablation studies show us that graph structure information is only beneficial if there is sufficient connectivity with the knowledge graph since sparser knowledge graphs can lead to noisy signals. We also explore the performance of Llama v2 on temporal link prediction, and the results show that current LLMs struggle with domain-specific temporal link prediction. Overall, our work provides an essential advance around effectively encoding continuously emerging entities and relations for temporal link prediction across evolving knowledge graphs over time.