Machine Learning Identifies the Emotion Climate During Naturalistic Conversations Using Speech Features and Affect Dynamics

Abstract

Emotion recognition in conversations (ERC) is of high importance, especially when it relates with human behavior assessment. Nevertheless, ERC so far has mainly focused on the identification of each interlocutor’s emotions. Here, for the first time, we consider the concept of emotion climate (EC), that is, the emotion reciprocally established by the peers during a naturalistic conversation, and we introduce machine learning (ML) models that efficiently perform emotion climate recognition (ECR). The latter is explored in the cases where the EC is (a) perceived within a conversational group, (b) conveyed from interlocutors involved in a conversation to the external observers, and (c) felt by the external observer. Features from conversational speech and affect dynamics (AD) data (n = 4685), drawn from three open datasets (i.e., K-EmoCon, IEMOCAP, and SEWA), were inputted to the ML-based ECR, achieving maximum accuracy of 96% and 83% in the K-EmoCon and IEMOCAP datasets, respectively. Cross-lingual validation was performed on SEWA dataset, justifying the generalization potential of the proposed approach. These results show that efficient ML-based ECR can identify how the EC is jointly built, perceived, and felt by others, providing a new approach in assessing emotional aspects in naturalistic conversations.