Bridging demonstration and multi-source attention fusion using LLMs for grounded multimodal named entity recognition

Abstract

Grounded multimodal named entity recognition (GMNER) is a challenging and emerging task that aims to identify all entity-type-region triplets from multimodal image-text pairs. Existing approaches often struggle with insufficient interaction between named entities and visual regions, leading to difficulties in accurate triplet alignment, cross-modal entity disambiguation, and visual semantic grounding. To tackle these challenges, we present a novel two-stage GMNER framework that integrates demonstration retrieval and multi-source cross-layer attention fusion. The initial stage for MNER employs entity-aware attention mechanism to select task-relevant demonstration examples, enabling large language models (LLMs) to generate high-quality external knowledge. The subsequent stage for visual grounding implements a sufficient cross-modal semantic interaction by introducing the multi-source multi-head cross-layer attention fusion (MMCAF) module, which integrates multi-source inputs (raw text, named and visual entity expressions, and image captions). Meanwhile, within this two-stage framework, we adopt a dual-LLM architecture using both text and vision LLMs, aiming to separate the generation of semantic priors from visual-language alignment and bridge gaps in cross-modal understanding. Our model achieves state-of-the-art performance across two GMNER datasets (Twitter-GMNER and Twitter-FMNERG) with different granularity, and further demonstrates superiority in ablation experiments and cross-domain evaluation.