Fusing YOLOv5s-MediaPipe-HRV to classify engagement in E-learning: From the perspective of external observations and internal factors

The rapid advancements in computer vision technology present significant potential for the automatic recognition of learner engagement in E-learning. We conducted a two-stage experiment to assess learner engagement based on behavioural (external observations) and physiological (internal factors) cues. Using computer vision technology and wearable sensors, we extracted three feature sets: action, head posture and heart rate variability (HRV). Subsequently, we integrated our constructed YOLOv5s–MediaPipe behaviour detection model with a physiological detection model based on HRV to comprehensively evaluate learners’ behavioural, affective and cognitive engagement. Additionally, we developed a method and criteria for assessing distraction based on behaviour, ultimately creating a comprehensive, efficient, low-cost and easy-to-use system for the automatic recognition of learner engagement. Experimental results showed that our improved YOLOv5s model achieved a mean average precision of 92.2 %, while halving both the number of parameters and model size. Unlike other deep learning-based methods, using MediaPipe–OpenCV for head posture analysis offers advantages in real-time performance, making it lightweight and easy to deploy. Our proposed long short-term memory classifier, based on sensitive HRV metrics and their normalisation, demonstrated satisfactory performance on the test set, with an accuracy = 80 %, precision = 81 %, recall = 80 % and an F1 score = 80 %.