I Know How you Feel Now, and Here's why!: Demystifying Time-Continuous High Resolution Text-Based Affect Predictions in the Wild

Affective computing 'in the wild' is of huge relevance to the healthcare field, like it is for many industries today. Applications of direct relevance are patient monitoring (e.g., emotional state, depression and pain monitoring), health information mining, diagnosis and opinion mining (e.g., from medical reports and drug reviews). The prevalence of the text modality in the medical field for various reasons – e.g., privacy laws, high costs and prohibitory memory requirements for audio and video data – has made the text modality the most popular. Deviating away from traditionally a classification task at a sample-level, the promising baseline results for the Audio/Visual Emotion Challenge (AVEC) 2017 make a strong case for the suitability of text data for a 'time-continuous' affect estimation. For the very first time, we present insights into the inner workings of deep learning, 'in the wild' affect-predicting, time-continuous regression model. We compute relevance of the sparse text-based bag-of-words features (BoTW) of the AVEC 2017 challenge in estimating the three affect labels, viz. arousal, valence and liking, by using a layerwise relevance propagation method(LRP). Interestingly, the trained models are found to rely more on adjectives and adverbs such as 'schlecht', 'gut', 'genau' with positive or negative connotations, and action descriptors such as and – quite analogous to the human perception of emotion expression.