Emotion Recognition Method based on Guided Fusion of Facial Expression and Bodily Posture

Abstract

Aiming at single modality video emotion recognition of face or body is easily affected by occlusion, angle deflection, and low emotional intensity, we propose an emotion recognition method based on guided fusion of facial expression and bodily posture (GF-FB). Firstly, Resnet50 and DNN are used to obtain intra-frame facial texture vector and bodily skeleton vector. Meanwhile, the whole-body geometric feature captured by the transformer encoder, is guided to obtain facial enhancement vector and bodily enhancement vector by the vectors of two modalities, respectively. Then, an inter-frame time encoder is designed to describe spatio-temporal features of facial enhancement sequence and bodily enhancement sequence. Finally, the heterogeneous features adaptive fusion module is constructed to realize the weight allocation of facial enhancement branch and bodily enhancement branch. Experimental results on the BabyRobot Emotion Dataset show that the accuracy of proposed method reaches 78.22%, which is 6.22% higher than baseline network.