Joint utilization of positive and negative pseudo-labels in semi-supervised facial expression recognition

Abstract

Facial expression recognition has obtained significant attention due to the abundance of unlabeled expressions, and semi-supervised learning aims to leverage unlabeled samples sufficiently. Recent approaches primarily focus on combining an adaptive margin and pseudo-labels to extract hard samples and boost performance. However, the instability of pseudo-labels and the utilization of the rest unlabeled samples remain critical challenges. We introduce a stable-positive-single and negative-multiple pseudo-labels (SPS-NM) method to solve the above two challenges. All unlabeled samples are categorized into three groups properly by adaptive confidence margins. When the maximum confidence scores are high and stable enough, the unlabeled samples are attached with positive pseudo-labels. On the contrary, when the confidence scores of unlabeled samples are low enough, the related negative-multi pseudo-labels are attached to these samples. The quality and quantity of classes in negative pseudo-labels are balanced by $top$-$k$. Eventually, the remaining unlabeled samples are ambiguous and fail to match their pseudo-labels, but they can still be used to extract valuable features by contrastive learning. We conduct comparative experiments and ablation study on RAF-DB, AffectNet and SFEW datasets to demonstrate that SPS-NM achieves improvement and becomes the state-of-the-art method in facial expression recognition.