MINIGE-MNER: A multi-stage interaction network inspired by gene editing for multimodal named entity recognition

Abstract

Multimodal Named Entity Recognition (MNER) integrates complementary information from both text and images to identify named entities within text. However, existing methods face three key issues: imbalanced handling of modality noise, the cascading effect of semantic mismatch, and information loss resulting from the lack of text dominance. To address these issues, this paper proposes a Multi-stage Interaction Network Inspired by Gene Editing for MNER(MINIGE-MNER). The core innovations of this method include: A gene knockout module based on the variational information bottleneck, which removes inferior genes (modality noise) from the text, raw image, and generated image features. This approach retains the superior genes, achieving balanced filtering of modality noise. A determination of gene recombination sites module that maximizes the mutual information between superior genes across modalities, reducing the spatial distance between them and ensuring precise, fine-grained semantic alignment. This helps to prevent the cascading effect of semantic mismatch. A text-guided gene recombination module that implements a “text-dominant, vision-supplementary” cross-modal fusion paradigm. This module dynamically filters out visual noise unrelated to the text while avoiding excessive reliance on visual information that could obscure the unique contextual information of the text, effectively mitigating information loss. Experimental results show that MINIGE-MNER achieves F1 scores of 76.45 % and 88.67 % on the Twitter-2015 and Twitter-2017 datasets, respectively, outperforming existing state-of-the-art methods by 0.83 % and 0.42 %. In addition, this paper presents comprehensive experiments that demonstrate the superiority of MINIGE-MNER and the effectiveness of its individual modules.