Hateful Meme Prediction Model Using Multimodal Deep Learning

Abstract

With the emergence of deep neural networks along with high-end computers that can process deep architectures, there has been a lot of research when Computer Vision and Natural Language Processing has been fused into a single problem. To enable students and researchers to deep dive into multimodal deep learning Facebook AI Research team published a dataset on hateful meme classification “The Hateful Meme Challenge Dataset” in May 2020 that gave us the motivation to test ourselves and an opportunity to learn more about the dataset. The rise of communication on the internet with memes as a medium, they have been used to convey incorrect information, political agendas and also has led to cyberbullying, trolling etc. This results in the need of creating an automated tool that can detect such hateful content published on the internet and remove it at the root level before it does any harm. This paper intends to adopt Unimodal Text and Image models using Bert, LSTM and VGG16, Resnet50, SE-Resnet50, XSE-Resnet architectures and combining them into Multimodal models for effective prediction of a hateful meme. The paper compares various architectures both unimodal models and multimodal models on the evaluation metrics AUC-ROC score, F1 score and accuracy score.)