Analytical review of methods for automatic detection of user engagement in virtual communication

Abstract

Introduction: The solution of the task of the recognition and assessment of user engagement in the acts of human-machine interaction or telecommunication, achieved through the use of automatic means, is highly important in computer recognition of human psycho-emotional states. This is necessary for e-learning, business and entertainment applications design. Purpose: To conduct a comparative analysis of the current information support in the field of automatic recognition and assessment of user involvement in human-machine interaction or virtual communication, as well as to establish a methodology for building a data body based on the idea of multimodal communication. Results: The conducted analysis of research papers has shown that in most existing databases there is a substantial lack of data for natural online communication. Moreover, not all databases contain different modalities in “human-machine-human” communication system. Text and audio modalities turn out to be important for a multilevel engagement classification task, aimed at the determination of engagement intensity. It is also promising to take into account “body language” features, such as facial expressions, movements of the body and the head, gestures. For the correct assessment of involvement, an engagement database must contain meta-data on the psycho-emotional states of communicants. Neural network-based approaches to the automatic detection of user engagement show the best performance. Practical relevance: Based on the obtained analytical conclusions, the authors of the paper are going to elaborate an original software system for automatic recognition of user engagement, and to collect a data set for machine learning purposes. The presented review formulates basic requirements for such systems and contributes to the solution of the problem of automatic recognition of psycho-emotional states. Discussion: The survey leads to the conclusion that the notion of engagement as understood in studies on automatic emotion recognition differs from that used in psychology. User (or communicant) engagement in terms of info- and communicative sphere implies the manifestation of a person's mental activity level (emotional, cognitive, and behavioral components) changing dynamically while interacting with another person or computer system.