Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Abstract

Music, as an important part of the culture, occupies a significant position and can be easily accessed. The research on the sentiment represented by music and its effect on the listener’s emotion is increasing gradually, but the existing research is often subjective and neglects the real-time expression of emotion. In this article, two labeled datasets are established. The deep learning method is used to classify music sentiment while the decision-level fusion method is used for real-time listener multimodal sentiment. We combine the sentiment analysis with a traditional online music playback system and propose innovatively a human-music emotional interaction system, using multimodal sentiment analysis based on the deep learning method. By means of individual observation and questionnaire survey, the interaction between human-music sentiments is proved to have a positive influence on listeners’ negative emotions.