Visual Sentiment Prediction by Merging Hand-Craft and CNN Features

Abstract

Nowadays, more and more people are getting used to social media such as Instagram, Facebook, Twitter, and Flickr to post images and texts to express their sentiment and emotions on almost all events and subjects. In consequence, analyzing sentiment of the huge number of images and texts on social networks has become more indispensable. Most of current research has focused on analyzing sentiment of textual data, while only few research has focused on sentiment analysis of image data. Some of these research has considered handcraft image features, the others has utilized Convolutional Neural Network (CNN) features. However, no research to our knowledge has considered mixing both hand-craft and CNN features. In this paper, we attempt to merge CNN which has shown remarkable achievements in Computer Vision recently, with handcraft features such as Color Histogram (CH) and Bag-of-Visual Words (BoVW) with some local features such as SURF and SIFT to predict sentiment of images. Furthermore, because it is often the case that the large amount of training data may not be easily obtained in the area of visual sentiment, we employ both data augmentation and transfer learning from a pre-trained CNN such as VGG16 trained with ImageNet dataset. With the handshake of hand-craft and End-to-End features from CNN, we attempt to attain the improvement of the performance of the proposed visual sentiment prediction framework. We conducted experiments on an image dataset from Twitter with polarity labels ("positive" and "negative"). The results of experiments demonstrate that our proposed visual sentimental prediction framework outperforms the current state-of-the-art methods.