Learning Emotion Information for Expressive Speech Synthesis Using Multi-resolution Modulation-filtered Cochleagram

Abstract

Emotion information plays an important role in improving the expressiveness of synthesized speech. At present, researchers mainly use style or emotion encoder to extract emotion information from mel-spectrogram extracted by mel fil-terbank. The mel filterbank does not consider the masking effect in the human auditory system, which results in mel-spectrogram not modeling complete auditory information. The multi-resolution modulation-filtered cochleagram (MMCG) simulates the auditory signal processing mechanism and reflects the function of the human auditory system. It can extract high-level auditory representations and significantly improve the emotion recognition performance. Therefore, we propose extracting emotion information from MMCG rather than mel-spectrogram to improve the expressiveness of synthesized speech. We propose three different kinds of MMCG encoders based on the characteristics of MMCG. Subjective and objective experiments demonstrate that using MMCG as an input feature can not only improve the naturalness and style transfer performance of synthesized speech but also reduce the fundamental frequency error. Our proposed MMCG encoders can extract more complete and rich emotion information from MMCG to further improve the expressiveness of synthesized speech.