CITR: Context-driven implicit triple reasoning for joint multimodal entity-relation extraction

Abstract

Joint Multimodal Entity-Relation Extraction (JMERE) jointly models Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE), aiming to extract valuable structured information from multimodal input. However, existing JMERE methods struggle to fully leverage bidirectional semantic interactions between tasks. To this end, this paper proposes a context-driven implicit triple reasoning framework (CITR), which takes type triples composed of entity and relation types as the foundation. Specifically, CITR first uses context generated by large multimodal models (LMMs) as semantic guidance cues to enhance modality representations, and prevent excessive semantic bias through a constraint module. Subsequently, CITR models the complex dependencies of different type triples to iteratively refine the representations associated with implicit triples. Finally, this paper reformulates the JMERE task as a type triple-centric sequence labeling problem and designs a dual-sequence joint tagging scheme, which reduces the computational complexity and label sparsity compared to previous schemes. Experimental results show that CITR achieves F1 score of 58.02% on the JMERE (Joint) task, significantly outperforming the state-of-the-art methods by 0.99%. Compared to methods with LMMs, CITR using LLaVA-1.5 achieves a superior F1 score of 58.49%.