Lightweight expression recognition combined attention fusion network with hybrid knowledge distillation for occluded e-learner facial images

In online education, it is crucial to monitor the students’ learning status timely and accurately. Facial expression recognition serves as the main tool for assessing their engagement levels. Existing algorithms still have some issues when directly applied in online learning. These issues primarily involve the loss of facial features due to occlusion, which directly affects the accuracy of expression recognition and at the same time, the expression recognition models require a large number of parameters and significant computational power, making them difficult to deploy and apply effectively on mobile devices with limited hardware resources. We aims to address the two issues by a two-stage framework: training process of an occluded facial expression recognition model, and the compression process of the occlusion facial expression recognition. To be more specific, in the first stage, we propose an occlusion facial expression recognition model based on attention fusion (AFNet), which adopts a multi-branch spatial attention network, extracts local facial features, automatically perceives facial occluded regions, reduces the weight of the occluded areas, and enhance robustness to occlusion by combining with a randomly masked channel network. Meanwhile, a feature pyramid network is introduced to extract global multi-scale features. In the second stage, we propose a hybrid model compression algorithm based on multi-layer knowledge distillation (MKD). We introduce a spatial attention network to focus on the important knowledge, reducing the information loss during knowledge distillation. Experimental results on five datasets show that the AFNet and MKD outperformed the baseline.