Label distribution learning for compound facial expression recognition in‐the‐wild: A comparative study

Abstract

Human emotional states encompass both basic and compound facial expressions. However, current works primarily focus on basic expressions, consequently neglecting the broad spectrum of human emotions encountered in practical scenarios. Compound facial expressions involve the simultaneous manifestation of multiple emotions on an individual's face. This phenomenon reflects the complexity and richness of human states, where facial features dynamically convey a combination of feelings. This study embarks on a pioneering exploration of Compound Facial Expression Recognition (CFER), with a distinctive emphasis on leveraging the Label Distribution Learning (LDL) paradigm. This strategic application of LDL aims to address the ambiguity and complexity inherent in compound expressions, marking a significant departure from the dominant Single Label Learning (SLL) and Multi‐Label Learning (MLL) paradigms. Within this framework, we rigorously investigate the potential of LDL for a critical challenge in Facial Expression Recognition (FER): recognizing compound facial expressions in uncontrolled environments. We utilize the recently introduced RAF‐CE dataset, meticulously designed for compound expression assessment. By conducting a comprehensive comparative analysis pitting LDL against conventional SLL and MLL approaches on RAF‐CE, we aim to definitively establish LDL's superiority in handling this complex task. Furthermore, we assess the generalizability of LDL models trained on RAF‐CE by evaluating their performance on the EmotioNet and RAF‐DB Compound datasets. This demonstrates their effectiveness without domain adaptation. To solidify these findings, we conduct a comprehensive comparative analysis of 12 cutting‐edge LDL algorithms on RAF‐CE, S‐BU3DFE, and S‐JAFFE datasets, providing valuable insights into the most effective LDL techniques for FER in‐the‐wild.