HIAN: A hybrid interactive attention network for multimodal sarcasm detection

Abstract

Multimodal sarcasm detection aims to use various modalities of data, such as text, images, etc., to identify whether they contain sarcastic meanings. Both images and texts contain rich sarcastic clues, but there are differences in dimension between them, and the quality of the sarcastic information they contain is very different. Therefore, seeking an appropriate feature fusion strategy to align modal features to maximize the utilization of inconsistent relationships between modalities is a significant challenge in this task. To this end, we introduce a novel sarcasm detection fusion model based on multimodal hybrid interactive attention (HIAN). We concatenate class words obtained from images with text and use the proposed bidirectional long short-term memory network with an interactive attention layer to enhance the extraction of text features. The text features obtained in this way can fully capture the contextual information of the text and the supplementary information in the image. To further enhance the feature fusion between modalities, we propose a multimodal interactive attention network and a fusion-enhanced transformer to promote the sharing of high-order complementary information, which represents the complementary non-linear semantic relationship between the three modalities and captures more inconsistencies between modalities. Extensive experiments conducted on publicly available multimodal sarcasm detection benchmark datasets show that our results surpass those of the baseline model and current state-of-the-art methods for the case of using the base BERT model.