Expression Tracking with OpenCV Deep Learning for a Development of Emotionally Aware Chatbots

Abstract

Affective computing explores the development of systems and devices that can perceive, translate, process, and reproduce human emotion. It is an interdisciplinary field which includes computer science, psychology and cognitive science. An inspiration for the research is the ability to simulate empathy when communicating with computers or in the future robots. This paper explored the potential of facial expression tracking with deep learning to make chatbots more emotionally aware through developing a post-therapy session survey chatbot which responds depending on two inputs, interactant’s response and facial expression. The developed chatbot summarizes emotional state of the user during the survey through percentages of the tracked facial expressions throughout the conversation with the chatbot. Facial expression tracking for happy, neutral, and hurt had 66.7%, 16.7%, and 56.7% tracking accuracy, respectively. Moreover, the developed program was tested to track expressions simultaneously per second. It can track 17 expressions with stationary subject and 14 expressions with non-stationary subject in a span of 30 seconds.