Real time speech emotion recognition using RGB image classification and transfer learning

This paper describes a real-time Speech Emotion Recognition (SER) task formulated as an image classification problem. The shift to an image classification paradigm provided the advantage of using an existing Deep Neural Network (AlexNet) pre-trained on a very large number of images, and thus eliminating the need for a lengthy network training process. Two alternative multi-class SER systems, AlexNet-SVM and FTAlexNet, were investigated. Both systems were shown to achieve state-of-the-art results when tested on a popular Berlin Emotional Speech (EMO-DB) database. Transformation from speech to image classification was achieved by creating RGB images depicting speech spectrograms. The ALEXNet-SVM method passes the spectrogram images as inputs to a pre-trained Convolutional Neural Network (AlexNet) to provide features for the Support Vector Machine (SVM) classifier, whereas the FTAlexNet method simply applies the images to a fine tuned AlexNet to provide emotional class labels. The FTAlexNet offers slightly higher accuracy compared to the AlexNet-SVM, while the AlexNet-SVM requires a lower number of computations due to the elimination of the neural network training procedure. A real-time demo is given on: https://www.youtube.com/watch?v=fuMpF3cUqDU&t=6s.