CNN-based stress and emotion recognition in ambulatory settings

Abstract

Stress has been recognized as a major contributor in a number of mental, psychological or physical conditions which reduce the quality of human life. The monitoring of affective states through readily available wearables and unobtrusive sensors can allow to recognize early signs of stress and burn-out, thereby develop prevention policies to combat psychosocial risks. This study analyzes data from diverse sensing modalities with signal processing techniques and advanced machine learning approaches in order to unobtrusively recognize stress and negative emotions. We investigate the performance of - easy to obtain in ambulatory settings - heart rate signal and juxtapose it against multi-modal information from electrophysiological signals, facial expression features and body posture. For the former, we introduce 2D spectrograms into a convolutional neural network (CNN) and use the obtained activation maps as frequency patterns differentiating stressful conditions. For the rest of the sensors, we compare different classifiers (SVM, KNN, Random Forest, Neural Networks) and data fusion schemes. In addition, a second phase assessment is conceptualized for emotion recognition reflected in facial expressions using images from a smartphone’s camera. Our CNN implementation in Android platform enables near real-time estimation of the instantaneous emotional expressions, which, when combined with stress-indicators, can help elucidating the relationship between stress and negative affective states.