Classification of Student Affective States in Online Learning using Neural Networks

The ongoing pandemic moved many classes online, and disrupted the classroom teaching experience and the feedback loop between teachers and students. One key challenge is to detect the engagement and other affective states exhibited by students during online learning. This paper investigates the capabilities and limitations of neural networks to distinguish between different affective states (i.e., boredom, engagement, confusion, and frustration), and their intensity level (i.e., very low, low, high, and very high). Several models are built using a hybrid ResNet+TCN neural network architecture. The models are trained using a large dataset, DAiSEE, that contains short 10 second video recordings of students as they watch educational content ‘in the wild’. A second dataset consisting of longer videos, EmotiW2020, is used to cross validate the engagement level classification model. The affective state classification model outperforms prior models. Boredom, confusion and frustration level classification models outperform or are on par with prior models. The engagement level classification model achieved similar performance with other baseline models and was outperformed by some SOTA models, but those models used 5 times more frames and 5 to 10 times more training epochs. The engagement level classification model was validated and achieved similar performance on both the DAiSEE and EmotiW2020 datasets.