Context-aware emotion recognition through agent-scene interactions

In real-world scenarios, context-aware emotion recognition (CAER) is a key problem in affective computing with broad application prospects. Most current CAER methods primarily rely on image-level contextual features. However, the interactive relationships between the agent and other objects within the scene are often overlooked or only partially modeled, which limits emotion recognition accuracy. To address this, we proposed a spatial interactive context-aware emotion network (ICENet) that consists of an agent feature extraction branch and a scene-context interaction branch. Specifically, the agent feature extraction branch aims to extract facial and posture features from the target agent and fuse them. In the facial feature extraction network named FaceNet, pure Convolutional Neural Network (ConvNeXt) is used as the backbone to extract global features, and a self-attention-based fine-grained feature extraction (FGFE) module is designed to capture more discriminative local features. In the posture feature extraction network, semantic segmentation is used to extract human silhouettes, which are then processed by Vision Transformer to obtain posture-related features. Meanwhile, the scene-context interaction branch named ObjNet integrates agent’s gaze angle and global depth maps to construct target agent-objects relationship in three-dimensional (TAR3D). Subsequently, a Graph Convolutional Network is employed to model the TAR3D and extract scene-context interaction features. Subsequently, a multiplicative fusion strategy is adopted to integrate agent features with scene-context interaction features, and emotion classification is performed based on the fused representation. Finally, experiments on EMOTIC and CAER-S datasets show that our approach outperforms current state-of-the-art methods in classification accuracy. The code is available at https://github.com/Cyx336/ICENet.git.