Deep Learning approach for text, image, and GIF multimodal sentiment analysis

Abstract

Recently, social media users are increasingly using visual media like GIFs, videos and images as powerful means of expressing their emotions and sentiments. We created a multimodal sentiment analysis tool for different forms of tweets in Python to compute the sentiment score not only base on the tweet's text, but also to consider GIFs and images to gain better accuracy for the tweet's overall sentiment score. For image sentiment, we use fine-tuned CNN, for texts we use VADER, and for the GIFs we apply both image sentiment and facial expression analysis in every frame of the file. In our work, we demonstrate that by utilizing both textual and image features we can achieve better results compared to other models that only rely on either images or text features. The final sentiment score for the incoming tweets will be calculated by aggregating the output scores obtained from each of our text, image, and GIF modules.