Multimodal sentiment analysis based on multi-head self-attention and convolutional block attention module

Abstract

Sarcasm is a type of emotional expression. Sarcasm is commonly used on social media to express the inverse of what appears to be a statement and what is said. Previous automatic sarcasm detection mainly focused on text. With the rise of image sharing mode on social media platforms, text cannot fully reveal users’ emotions, so people begin to study multimodal sentiment analysis by combining text and images. Previous researches on sarcasm detection have used Bidirectional Long Short-term Memory Network (Bi-LSTM) and Residual Network (ResNet) to extract text and image feature vectors, respectively. While Multi-Head Self-Attention (MH-SA) is added to the Bi-LSTM model to perform relation extraction, which can effectively avoid complex feature engineering in traditional tasks. In the process of image extraction, the channel attention module (CAM) and the spatial attention module (SAM) are used to weight different spatial and channel features and focus on different regions and features of the image. The two complement each other, greatly improving the network’s ability to express features. On the Twitter dataset, our proposed model has a sarcasm detection accuracy of 87.55 %, which outperforms most models proposed in current papers.