Multimodal Affect Perception With Large Language Model Enhancement Network

Abstract

Multimodal Sentiment Analysis (MSA) plays a vital role in understanding emotional content from social media and multimedia data. However, existing methods often rely on large-scale labeled datasets, leading to high annotation costs and poor adaptability. They also suffer from modality imbalance and suboptimal feature fusion. To address these issues, we propose MapleNet—a Multimodal Affect Perception framework enhanced by Large Language Models. MapleNet integrates a prototype-guided fusion strategy and a dynamic modality balancing mechanism to improve alignment and collaboration between text and image features. Specifically, a shared-space encoder combined with prompt optimization ensures semantic consistency across modalities. Within the prototype learning framework, the model dynamically adjusts modality-specific learning by aligning features with class prototypes, thus mitigating imbalance and uncovering complementary affective cues. In addition, MapleNet employs a similarity-based sample retrieval module to construct contextual prompts, enriching sentiment understanding in few-shot settings. Experiments on six benchmark datasets show that MapleNet consistently outperforms state-of-the-art methods, especially under few-shot conditions, achieving superior accuracy and generalization.