A Study on Low-Latency Recognition-Synthesis-Based Any-to-One Voice Conversion

Some application scenarios of voice conversion, such as identity disguise in voice communication, require low-latency generation of converted speech. In traditional conversion methods, both history and future information in input speech are utilized to predict the converted acoustic features at each frame, which leads to long latency of voice conversion. Therefore, this paper proposes a low-latency recognition-synthesis-based any-to-one voice conversion method. Bottleneck (BN) features are extracted by an automatic speech recognition (ASR) acoustic model for frame-by-frame phoneme classification. A minimum mutual information (MMI) loss is introduced to reduce the speaker information in BNs caused by the low-latency configuration. The BN features are sent into a speaker-dependent low-latency LSTM-based acoustic feature predictor and the speech waveforms are reconstructed by an LPCNet vocoder from predicted acoustic features. The total latency of our proposed voice conversion method is 190ms, which is less than the delay requirement for comfortable communication in ITU-T G.114. The naturalness of converted speech is comparable with the upper-bound model trained without low-latency constraints.