On multimodality in the perception of emotions from materials of the HuComTech corpus

Abstract

Emotions are important constituents of human behavior. The production and perception of cues of emotions is a complex task involving both verbal and nonverbal aspects of behavior. This complexity is further enhanced by the fact that emotions are subject to interpretation; a resulting emotion cannot be compositionally derived from its constituent building blocks. Even though we commonly associate an emotion with a given modality (most often with visuality), it will be argued that emotions are essentially multimodal. Multimodality in turn involves both the temporal alignment and sequential organization of cues across a number of modalities with virtually no primary modality of expression. These assumptions will be tested and elaborated using the extensively annotated multimodal HuComTech corpus by considering the frequency, alignment and sequence of the annotations of the basic emotions across three perception conditions: video only, audio only and video+audio.