In-the-wild facial emotion recognition using relation-aware geometric features and CapsNet

Occlusions and pose variations are key challenges in Facial Emotion Recognition (FER), affecting recognition accuracy, especially in uncontrolled environments. This paper presents a robust FER method, FMR-CapsNet (includes Facemesh mediapipe, ResNet50, and Capsule Neural Network), designed to address these issues. The proposed model employs the FaceMesh model for geometric feature extraction, utilizing facial blendshape scores to capture expression-related features even in side-facing and occluded images. A Euclidean Distance metric constructs a relation-aware distance matrix to encode spatial relationships between blendshape scores. To further refine features, transfer learning is applied using a pretrained Residual Network (ResNet50), followed by a Capsule Neural Network (CapsNet) to capture directional and spatial information, improving feature differentiation. Extensive experiments on three in-the-wild datasets— Real-world Affective Faces Database (RAF-DB), AffectNet, and FERPlus demonstrate that FMR-CapsNet significantly enhances FER performance, achieving 97.01% accuracy on RAF-DB, 71.12% on AffectNet, and 91.82% on FERPlus, outperforming state-of-the-art (SOTA) methods in handling occlusions and pose variations.