Augmenting Drama: A Speech Emotion - Controlled Stage Lighting Framework

Abstract

Lighting can play a key role in the aesthetic concept of a theatrical production. This paper explores the field of augmented interaction with stage lighting, providing a synesthetic approach to emotion perception. In the audio driven framework that is presented, the actors' speech is captured by stage microphones. The signals are led to a Speech Emotion Recognition system that classifies them by emotion, which are thereafter matched to different colors. Thus, stage lighting color can change in real-time in accordance with the actor's recognized speech emotion. The system is described in a generic form, suitable for different implementations of the main idea. For the purpose of this paper, 5 classes that represent different emotions were defined. Several audio features and classifiers were tested with audio data from different datasets of emotional speech to train a speech emotion recognition model. The final evaluation results are presented for the logistic regression classifier. Accuracy results and confusion matrix are presented for logistic regression classification. A wheel of emotions model for emotion visualization and color selection was adopted to render and simulate the colored lighting results.