Hybrid Model for Visual Sentiment Classification Using Content-Based Image Retrieval and Multi-Input Convolutional Neural Network

With the exponential growth of multimedia content, visual sentiment classification has emerged as a significant research area. However, it poses unique challenges due to the complexity and subjective nature of the visual information. This can be attributed to the significant presence of semantically ambiguous images within the current benchmark datasets, which enhances the performance of sentiment analysis but ignores the differences between various annotators. Moreover, most current methods concentrate on improving local emotional representations that focus on object extraction procedures rather than utilizing robust features that can effectively indicate the relevance of objects within an image through color information. Motivated by these observations, this paper addresses the need for efficient algorithms for labeling and classifying sentiment from visual images by introducing a novel hybrid model, which combines content-based image retrieval (CBIR) and a multi-input convolutional neural network (CNN). The CBIR model extracts color features from all dataset images, creating a numerical representation for each. It compares a query image to dataset images’ features to find similar features. This process continues until the images are grouped according to color similarity, which allows accurate sentimental categories based on similar features and feelings. Then, a multi-input CNN model is utilized to extract and efficiently incorporate high-level contextual visual information. This model comprises 70 layers, with six branches, each containing 11 layers. It seeks to facilitate the fusion of complementary information by incorporating multiple input categories that differ according to the color features extracted by the CBIR technique. This feature enables the model to understand the target and generate more precise predictions fully. The proposed model demonstrates significant improvements over existing algorithms, as evidenced by evaluations of six benchmark datasets of varying sizes. Also, it outperforms the state of the art in sentiment classification accuracy, getting 87.88%, 84.62%, 84.1%, 83.7%, 80.7%, and 91.2% accuracy for the EmotionROI, ArtPhoto, Twitter I, Twitter II, Abstract, and FI datasets, respectively. Furthermore, the model is evaluated on two newly collected large datasets, which confirm its scalability and robustness in handling large-scale sentiment classification tasks, and thus achieves a significant accuracy of 85.21% and 83.72% with the BGETTY and Twitter datasets, respectively. This paper contributes to the advancement of visual sentiment classification by offering a comprehensive solution for analyzing sentiment from images and laying the foundation for further research.