Multimodal Hateful Memes Detection via Image Caption Supervision

A large amount of hateful speech exist on the Internet in the form of text and images uploaded by social media users. Recently, multimodal hateful speech detection task has attracted more and more researchers to invest, producing some representative work for perceiving the negative samples. For this special multimodal task, the ability of multimodal semantic information understanding is particularly crucial. However, the existing models have insufficient understanding ability of image modality semantic compared with the text modality, due to the appearance complexity of each image. Therefore, this paper utilizes the text modality which is well understood by the model to improve understanding ability of image modality semantic. Specifically, this paper proposes an image caption supervision (ICS) auxiliary method for multimodal hateful speech detection, where the image caption is designed to supervise the feature learning of images for further understanding the semantic information. On the Facebook Hateful Memes dataset, the proposed ICS method outperforms some state-of-the-art unimodal and multimodal baselines, demonstrating the effectiveness of ICS.