Deep learning of facial embeddings and facial landmark points for the detection of academic emotions

Automatic emotion recognition is an actively researched area as emotion plays a pivotal role in effective human communications. Equipping a computer to understand and respond to human emotions has potential applications in many fields including education, medicine, transport and hospitality. In a classroom or online learning context, the basic emotions do not occur frequently and do not influence the learning process itself. The academic emotions such as engagement, frustration, confusion and boredom are the ones which are pivotal to sustaining the motivation of learners. In this study, we evaluated the use of deep learning on FaceNet embeddings and facial landmark points for academic emotion detection on a publicly available dataset - DAiSEE that has been annotated with the emotional states of engagement, boredom, frustration and confusion. By modeling both the spatial and temporal dimensions, the results demonstrated that both models are able to detect incidences of boredom and frustration and can be used in the moment-by-moment monitoring of boredom and frustration of learners using a tutoring system either online or in a classroom.