A mobile emotion recognition system based on speech signals and facial images

Abstract

Smartphones are used daily for personal and business communications, and they have become a primary medium to capture human emotions. By recognizing the emotions of speakers during a conversation, one can deliver or understand messages better, make successful negotiations, and provide more personal services. Therefore, we developed an emotion recognition system on a mobile platform based on speech signals and facial images. This research has two phases, a training phase and a testing phase. In the first phase, speech signals and facial images are processed through data preprocessing, feature extraction, and SVM classifier construction steps. In the second phase, the participants generated video recordings as test data. These data were transformed for feature extraction and classified into four emotion classes by using the generated classifiers. Feature selection methods were exploited to choose useful features. We proposed an adjustable weighted segmentation method to determine the final results of emotion recognition. Various experiments were performed using real world simulations to evaluate the proposed system. The result showed an average accuracy rate of 87 percent with the highest accuracy rate at 91 percent. Facial images were also used to improve emotion recognition especially during periods of silence in conversations.