A pilot case study for developing a software for human emotion recognition using multimodal data

Abstract

In developing software to analyze the emotions of patients with schizophrenia using multimodal data, a pilot case study was conducted in order to examine its accuracy in a healthy subject. This study shows a low agreement and reliability between the MTCNN and the subjective evaluation of the three examiners based on the result of the ICC and Cronbach's alpha coefficient. However, it can be revealed that the Multi-Task Cascaded Convolutional Networks (MTCNN) facial expression recognition and the Heart Rate Variability (HRV) analysis showed consistent results. Subject who experienced anticipated feelings of happiness when the conversation was focused in the subject's favorite food, which showed an increased in HFnu, indicating increased parasympathetic activity. It was considered that the subject felt that the conversation with the robot was lively and empathetic. Findings suggest that MTCNN can be used in combination with HRV analysis in determining the facial expression of an individual. However, further research should be done involving additional subjects in order to ascertain the validity and reliability of the MTCNN.