Cognitive Multimodal Processing: from Signal to Behavior

Abstract

Affective computing, social and behavioral signal processing are emerging research disciplines that attempt to automatically label the emotional, social and cognitive state of humans using features extracted from audio-visual streams. I argue that this monumental task cannot succeed unless the particularities of the human cognitive processing are incorporated into our models, especially given that often the quantities we are called to model are either biased cognitive abstractions of the real world or altogether fictional creations of our cognition. A variety of cognitive processes that make computational modeling especially challenging are outlined herein, notably: 1) (joint) attention and saliency, 2) common ground, conceptual semantic spaces and representation learning, 3) fusion across time, modalities and cognitive representation layers, and 4) dual-system processing (system one vs. system two) and cognitive decision non-linearities. The grand challenges are outlined and examples are given illustrating how to design models that are both high-performing and respect basic cognitive organization principles. It is shown that such models can achieve good generalization and representation power, as well as model cognitive biases, a prerequisite for modeling and predicting human behavior.