U-Shaped Distribution Guided Sign Language Emotion Recognition With Semantic and Movement Features

Abstract

Emotional expression is a bridge to human communication, especially for the hearing impaired. This paper proposes a sign language emotion recognition method based on semantic and movement features by exploring the relationship between emotion valence and arousal in-depth, called SeMER. The SeMER framework includes a semantic extractor, a movement feature extractor, and an emotion classifier. The contextual relations obtained from the sign language recognition task are added to the semantic extractor as prior knowledge using a transfer learning approach to better acquire the affective polarity of semantics. In the movement feature extractor based on graph convolutional networks, a spatial-temporal adjacency matrix of gestures and node attention matrix are developed to aggregate the emotion-related movement features of intra- and inter-gestures. The proposed emotion classifier maps semantic and movement features to the emotion space. The validated U-shaped distributions of valance and arousal are then used to guide the relationship between them, and improve the accuracy of emotion prediction. In addition, a sign language emotion dataset containing 5 emotions from 18 participants, SE-Sentence, is collected through armbands with built-in surface electromyograph and inertial measurement unit sensors. Experimental results showed that SeMER achieved an accuracy and f1 value of 88% on SE-Sentence.