Affective Computing of Image Emotion Perceptions

Abstract

Images can convey rich semantics and evoke strong emotions in viewers. The research of my PhD thesis focuses on image emotion computing (IEC), which aims to predict the emotion perceptions of given images. The development of IEC is greatly constrained by two main challenges: affective gap and subjective evaluation [5]. Previous works mainly focused on finding features that can express emotions better to bridge the affective gap, such as elements-of-art based features [2] and shape features [1]. Based on the emotion representation models, including categorical emotion states (CES) and dimensional emotion space (DES) [5], three different tasks are traditionally performed on IEC: affective image classification, regression and retrieval. The state-of-the-art methods on the three above tasks are image-centric, focusing on the dominant emotions for the majority of viewers. For my PhD thesis, I plan to answer the following questions: 1. Compared to the low-level elements-of-art based features, can we find some higher level features that are more interpretable and have stronger link to emotions? 2. Are the emotions that are evoked in viewers by an image subjective and different? If they are, how can we tackle the user-centric emotion prediction? 3. For imagecentric emotion computing, can we predict the emotion distribution instead of the dominant emotion category? 1. The artistic elements must be carefully arranged and orchestrated into meaningful regions and images to describe specific semantics and emotions. The rules, tools or guidelines of arranging and orchestrating the elements-of-art in an artwork are known as the principles-of-art, which consider various artistic aspects, including balance, emphasis, harmony, variety, gradation, movement, rhythm, and proportion [5]. We systematically study and formulize the former 6 artistic principles, explaining the concepts and translating these concepts into mathematical formulae. 2. The images in Abstract dataset [2] were labeled by 14 people on average. 81% images are assigned with 5 to 8 emotions. So the perceived emotions of different viewers may vary. To further demonstrate this observation, we set up a large-scale dataset, named Image-Emotion-Social-Net dataset, with over 1 million images downloaded from Flickr. To get the personalized emotion labels, firstly we use traditional lexicon-based methods as in [4] to obtain the text segmentation results of the title, tags and descrip-