Automatic Instructional Pointing Gesture Recognition by Machine Learning in the Intelligent Learning Environment

Abstract

The recorded video in an intelligent learning environment is a key source for analyzing participants' behavior and receiving instructional feedbacks. Teachers and students are the main participants in school education. According to recent studies, the teacher's instructional pointing gesture to the learning content can gain students' attention as well as improve students' learning performance. We aim to automatically recognize these pointing gestures by a series of compound methods. Our main contributions are threefold: first, we collected and labeled our own dataset for instructional pointing gesture recognition in the intelligent learning environment; second, we applied non-linear neural networks to learn the pointing gesture based on the joint points extracted by OpenPose[1]; Third, we proposed a novel framework to recognize the teacher's pointing gesture on a target area, which we named as the instructional pointing gesture, whose contents are frequently changing in real time. From the experimental results, we have proven that our proposed method allows us to recognize the pointing gesture with 90% accuracy on average.