Prototype-oriented multimodal emotion contrast-enhancer

Abstract

Prototype learning has been proven effective and reliable for few-shot learning. Therefore, prototype learning can also do data enhancement work. Simultaneously, although CL(Contrastive Learning)-based methods can alleviate the data sparsity problem, they may amplify the noise in the original features. Recently, a series of outstanding models have emerged in multimodal sentiment analysis. However, the limited size of benchmark datasets in this field presents significant challenges for training models. To address this, we propose a prototype-contrast-enhanced approach for multimodal sentiment analysis. Our method combines contrastive learning with prototype learning, using improved contrastive learning to supervise the effectiveness of prototype learning and ensure the effectiveness of data augmentation. This method utilizes prototype learning to denoise features in contrastive and contrastive learning to supervise prototype performance. During the training phase, we generate prototyped representations as base classes. At the same time, the prototype representation of the training phase is supervised by contrastive loss. In the testing phase, these base classes augment samples, thereby assisting the model in accurately recognizing emotions. To evaluate our proposed method, we conduct experiments on widely used multimodal sentiment datasets, namely MOSI and MOSEI. The outcome of our extensive experiments confirms the significant effectiveness of our approach. We are making the code public at https://github.com/925151505/MyCode