Gesture-Based Affective and Cognitive States Recognition Using Kinect for Effective Feedback during e-Learning

Abstract

With the growth of online education, there have been many attempts by educators to identify the learner's emotions and attention so as to improve feedback during the learning process. Such systems have mostly used the learner's interaction with the system, audio and video monitoring, and profiling to identify the user's empathic state and provide feedback accordingly. Facial expressions, eye tracking as well as eye PERCLOS have been used to identify both alertness and emotions. However, identification of cognitive as well as affective states using gestures is a relatively neglected area, though in the field of gaming and pedagogy, gesture recognition is an important area of research for interaction with computers. In this paper, we report a work in progress where we have been able to determine some of the user's empathic states through her gestures using Kinect, and have proposed to create an accurate system for cognitive state and affective gesture recognition by first developing a database of gestures signifying user's emotional and affect states related to e-learning context, and then by calibrating the system for accurate detection of emotions and allied states through gestures. This can be used independently or with other multimedia inputs for accurate feedback in e-learning environments.