Differently processed modality and appropriate model selection lead to richer representation of the multimodal input

Abstract

We aim to effectively solve and improvise the Meta Meme Challenge for the binary classification of hateful memes detection on a multimodal dataset launched by Meta. This problem has its challenges in terms of individual modality processing and its impact on the final classification of hateful memes. We focus on feature-level fusion methodologies in proposing the solutions for hateful memes detection in comparison with the decision-level fusion as feature-level fusion generates richer features’ representation for further processing. Appropriate model selection in multimodal data processing plays an important role in the downstream tasks. Moreover, inherent negativity associated with the visual modality may not be detected completely through the visual processing models, necessitating the differently processed visual data through some other techniques. Specifically, we propose two feature-level fusion-based methodologies for the aforesaid classification problem, employing VisualBERT for the effective representation of textual and visual modality. Additionally, we employ image captioning generating the textual captions from the visual modality of the multimodal input which is further fused with the actual text associated with the input through the Tensor Fusion Networks. Our proposed model considerably outperforms the state of the arts on accuracy and AuROC performance metrics.