Extracting emotions from speech using a bag-of-visual-words approach

Recognition of humans' emotions may be crucial in certain applications involving e.g., human-computer interaction, monitoring of elderly, understanding the affective state of learners during a course etc. To this goal and depending on the application and the environment, one may use physiological parameters (e.g., heart rate, brain activity etc.) which are typically obtrusive, or analyze other modalities that may be extracted by simply observing a human, such as visual (e.g., her/his facial expressions, gestures, skeletal motion etc.) or audio (e.g., speech). In many applications the only available modality is the latter one, i.e., the human's voice. In this work we aim to analyze a speaker's emotions by relying only on paralinguistic information, extracted by her/his voice, thus discarding the linguistic aspect of speech (i.e., the spoken words). To this goal, we propose a novel emotion classification approach that has been inspired by computer vision tasks. We use a spectrogram, which is a visual representation of the spectrum of an audio segment. We then extract features and code them using a visual vocabulary and represent a spectrogram as a “bag-of-visual words.” This representation is used for classifying an audio segment to an emotion class. We evaluate our approach on 3 datasets that contain speech from different languages and compare it to baseline methods.