An Empirical Framework for Identifying Sentiment from Multimodal Memes using Fusion Approach

Advances in social media platforms led to the widespread adoption of memes, making them a powerful communication tool on the internet. Memes’ visual aspect gives them a remarkable ability to influence users’ opinions. However, individuals misemploy this popularity to foment animosity. The spread of these hostile memes can have a detrimental effect on people, causing depression and suicidal thoughts. Therefore, stopping inappropriate memes from spreading on the internet is crucial. However, identifying memes is di cult due to their multimodal nature. This paper proposes a deep-learning-based framework to classify sentiment (into ‘positive’ or ‘negative’) from multimodal memes in Bengali. Due to the unavailability of standard corpora, a Bengali meme corpus consisting of 1671 memes is developed to perform the memes’ sentiment classification task. Five popular deep learning models (CNN, BiLSTM) and pre-trained models (VGG16, VGG19, InceptionV3) are investigated for textual and visual features. The framework is developed by combining visual and textual models. The comparative analysis confirms that the proposed model (BiLSTM + VGG19) achieved the highest f1-score (0.68) compared to other multimodal methods.