Multi-Feature Cross-Lingual Transfer Learning Approach for Low-Resource Vietnamese Speech Synthesis

Abstract—Based on neural network end-to-end speech synthesis systems, high-quality speech can be synthesized when there is sufficient training data. However, it is difficult for languages with small datasets to synthesize speech with high quality and naturalness. Vietnamese is a tonal language, belonging to the Vietic branch of the Austroasiatic language family, which uses a spelling system. To improve the quality and naturalness of speech synthesis with limited dataset resources, we first use transfer learning to improve the acoustic model of Vietnamese by leveraging the similarities in pronunciation and grammar between Mandarin Chinese and Vietnamese. Secondly, based on the prosodic characteristics of Vietnamese, we use a "speech-text" alignment tool to extract prosodic boundary information and supplement it to the training text sequence. Using FastSpeech2 as the experimental baseline system, we designed and added a prosody embedding layer. The experimental results show that the model trained with prosodic markers has better prosody expression compared to the original text. Furthermore, compared to the baseline system, adding the prosody embedding layer improved the prosody expression of the synthesized speech and eliminated the need for marked text during speech synthesis.