Detection of Conversational Health in a Multimodal Conversation Graph by Measuring Emotional Concordance

Abstract

With the advent of social media and technology, the increased connections between individuals and organizations have led to a similar increase in the number of conversations. These conversations, in most cases are bimodal in nature, consisting of both images and text. Existing work in multimodal conversation typically focuses on individual utterances rather than the overall dialogue. The aspect of conversational health is important in many real world conversational uses cases including the emerging world of Metaverse. The work described in this paper investigates conversational health from the viewpoint of emotional concordance in bimodal conversations modelled as graphs. Using this framework, an existing multimodal dialogue dataset has been reformatted as a graph dataset that is labelled with the emotional concordance score. In this work, determination of conversational health has been framed as a graph classification problem. A graph neural network based model using algorithms such as Graph Convolution Network and Graph Attention Network is then used to detect the emotional concordance or discordance based upon the multimodal conversation that is provided. The model proposed in this paper achieves an overall F1 Score of 0.71 for equally sized class training and testing size, which offers improved results compared to previous models using the same benchmark dataset.