Visualization of affective information in music using chironomie

The purpose of this study is to visualize affective information that cannot be conveyed by symbolic notation alone, to enhance the musical experience of the hearing impaired. To represent the rhythm of music effectively and uniquely, we focused on Chironomie, which represents the structure of rhythm with emotional impression. In general, Chironomie is drawn by a curve corresponding to the score, which is determined by whether a short segment of the score represents one of two classes, Arsis or Thesis. First, we utilized the machine learning technique to classify Arsis and Thesis from the score as input. We conducted experiments to confirm the accuracy of the classification, and the usefulness of the estimated Chironomie in conveying the rhythm of music. In the latter experiment, four types of stimuli combining visual and sound information were used to confirm the effects of Chironome: score only, Chironomie only, score and Chironomie, and score and sound. Results showed that Chironomie has certain usefulness in conveying the rhythmic structure of a piece of music. This paper mainly focuses on evaluation experiments and discusses experimental and analytical methods under these experimental conditions.