Behavior capture guided engagement recognition

Abstract

Engagement recognition aims to assess an individual’s involvement in various activities, which is essential in fields like education, healthcare, and driving. However, existing methods often suffer from performance degradation due to excessive data and distractions. In this paper, we introduce a novel model, the Behavior Capture-guided Transformer (BCTR). One of its key innovations lies in the proposed architecture for extracting regional features. Specifically, BCTR employs three independent class tokens to capture regional features – ocular, head, and trunk – from image sequences. These features are then used to model the dynamic streams of these regions for video-based engagement recognition. Another unique innovation of BCTR is its ability to mimic the observational techniques used by human teachers. By leveraging both frame-level and video-level class tokens, the model uses dual branches to detect both static and dynamic disengagement behaviors. This approach not only enables BCTR to achieve superior performance – 64.51% accuracy on the DAiSEE dataset and 0.0602 MSE loss on the EmotiW-EP dataset – but also enhances the interpretability of engagement levels by identifying these disengagements.