An effective Multi-Modality Feature Synergy and Feature Enhancer for multimodal intent recognition

Abstract

Multimodal intent recognition is a critical task that aims to accurately capture and interpret a user’s true intentions by integrating various sensory inputs such as facial expressions, body language, and vocal emotions. In complex and dynamic real-world multimodal interaction scenarios, deepening the understanding of human language and behavior becomes essential. Although multimodal data is rich in information, enhancing the representation of data features and efficiently integrating multimodal information to improve intent recognition performance remains a significant technical challenge. To address the aforementioned issue, a Video Feature Enhancer (VFE) module, combined with a Multi-Modality Feature Synergy (MFS) method, is proposed. The Video Feature Enhancer module employs a feature-weighting strategy based on energy optimization, along with an attention mechanism across channel spaces, to enhance the representational capability of video features. The Multi-Modality Feature Synergy method uses multi-level textual feature guidance and multimodal association learning to effectively integrate and optimize the feature representations of video and audio modalities. The Multi-Modality Feature Synergy method also suppresses non-essential information, facilitating the fusion of complementary information across different modalities, ultimately improving multimodal intent recognition performance. In the experimental evaluation, significant performance improvements are demonstrated over existing state-of-the-art methods on two benchmark datasets. On the MIntRec dataset, accuracy (ACC) is improved by 0.6%, weighted F1 score (WF1) by 1.21%, and weighted precision (WP) by 1.7%, while recall (R) increases by 1.8%. On the MELD-DA dataset, a 0.9% improvement in ACC is achieved, a significant increase of 1.15% in WF1 and 1.34% in WP, and also a 0.21% improvement in R is shown. Furthermore, through ablation studies, the substantial contributions of both the Video Feature Enhancer module and the Multi-Modality Feature Synergy method are validated in enhancing modality-specific feature representations and improving intent recognition accuracy.