Spatiotemporal Features Learning from Song for Emotions Recognition with Time Distributed CNN

Abstract

Building a system that can naturally interact with humans has been one of the ultimate goals for researchers in the computer science field. The system should be able to interpret both verbal and non-verbal meanings from the messages conveyed by the interlocutors. A song can also be a vehicle to express a message to the listeners, and capturing the emotions from the song automatically can provide a system that can have the digital feeling when they are listening to the song. Emotions can be automatically captured and processed through several modalities via sensors. Deep learning has been the golden standard of learning architecture in many fields. The emotions recognition model can be trained well with some of the deep learning architectures. Convolution Neural Networks (CNN) is famous to train models that have multi-dimensional input features. However, it has a limitation when dealing with features that have temporal information. This research aims to use Time Distributed layers to CNN architecture to learn Spatio-temporal features from the songs (audio signals). Eight architectures were proposed in this research to explore the potential of learning Spatio-temporal features from songs with CNN architecture. The best model presented in this paper achieved 99.95%, 93.41 %, 1.84, 2.03 in training accuracy, testing accuracy, training loss and testing loss, respectively.