Enhancing multi-modal aspect-based sentiment classification via emotional semantic-aware cross-modal relation inference

Abstract

Multi-modal Aspect-based Sentiment Classification (MASC) determines the sentiment polarity of specific aspects in text–image pairs. Recent research has explored leveraging image–text relevance to improve MASC performance. However, existing approaches primarily focus on explicit alignments between textual aspects and visual objects or on the global relevance between entire texts and images, often overlooking the implicit emotional connections specific to aspects. In this work, we propose an aspect-level emotional cross-modal relation scheme that captures both explicit alignments and implicit emotional connections between text and image. Based on this scheme, we construct a new dataset, the Aspect-level Emotional Cross-modal Relevance dataset (AECR-Twitter), which contains 3,562 image–text pairs. We also introduce several methods for integrating cross-modal relevance into MASC. Experimental results across eight different model architectures consistently demonstrate the effectiveness of our aspect-level emotional cross-modal relation scheme in enhancing MASC performance, with F1 scores increasing by an average of 1.26% on Twitter-15 and 1.28% on Twitter-17. We release our data and code at https://github.com/li9527yu/AECR-Twitter.