RCLMuFN: Relational context learning and multiplex fusion network for multimodal sarcasm detection

Abstract

Sarcasm typically conveys emotions of contempt or criticism by expressing a meaning that is contrary to the speaker’s true intent. Accurately detecting sarcasm aids in identifying and filtering undesirable information on the Internet, thereby mitigating malicious defamation and rumor-mongering. Nonetheless, automatic sarcasm detection remains a challenging task for machines, as it critically depends on intricate factors such as relational context. Existing multimodal sarcasm detection methods focus on introducing graph structures to establish entity relationships between text and image while neglecting to learn the relational context between text and image, which is crucial evidence for understanding the meaning of sarcasm. In addition, the meaning of sarcasm evolves across different contexts, but current methods may struggle to accurately model such dynamic changes, thereby limiting the generalization ability of the models. To address the aforementioned issues, we propose a relational context learning and multiplex fusion network (RCLMuFN) for multimodal sarcasm detection. First, we employ four feature extractors to comprehensively extract features from raw text and images, aiming to excavate potential features that may have been previously overlooked. Second, we propose a relational context learning module to learn the contextual information of text and images and capture the dynamic properties through shallow and deep interactions. Finally, we propose a multiplex feature fusion module to enhance the model’s generalization by effectively integrating multimodal features derived from diverse interaction contexts. Extensive experiments on two multimodal sarcasm detection datasets show that RCLMuFN achieves state-of-the-art performance.