Sentiment-based robot sonification can enhance comprehension, emotional meaning and engagement in the fairy tale listening: A preliminary study

Abstract

Sonification provides highly informative data, but its interpretation can vary depending on the context and the audience’s capability. Specifically, sonification can play a key role in storytelling experience. The current study examined how sentiment-based sonification vs. classical music vs. no sonification influence the experience of robot storytelling, especially in terms of audience members’ comprehension, emotional meaning, and engagement in such an environment. We analyzed three fairy tales using narrative analysis principles and a sentiment analysis algorithm to design sentiment-based sonification based on classical music. Thirty-four participants experienced three stories with emotion-reflecting sonification, classical music, or no sonification (within-subjects design) with a humanoid robot, NAO, as a storyteller. The results showed that the ratings for comprehension, emotional meaning, and engagement were significantly improved in the sonification condition compared to the other conditions. Implications are discussed with various models and theories, including the ITPRA theory of expectation, the capacity model of attention, and the limited capacity theory, among others.