A streaming variable neural speech codec

Abstract

This paper presents a variable bit rate streaming neural speech codec designed for ultra-low bit rate scenarios, based on the SoundStream network framework. The codec employs the vector quantized variational auto-encoder (VQ-VAE) algorithm to capture the temporal structure and spectral characteristics of the speech signal, and constructs a latent space codebook to facilitate the effective mapping of feature vectors to discrete vectors. Based on the harmonic characteristics of speech signals and the inherent defects of single-scale discriminators, we introduce multi-period discriminators and multi-scale discriminators. The training process uses a balanced training strategy to ensure the balance between codebook utilization and training weights, and utilizes the Short-Time Fourier Transform (STFT) spectrum that can provide more accurate time–frequency resolution to compute the reconstruction loss. We introduce codebook loss to improve the utilization rate of the codebook and accelerate the convergence of the model. In the inference process, we use a quantizer selection strategy to achieve adaptive adjustment of variable bitrate. Objective and subjective experiments demonstrate that our proposed new neural speech codec outperforms traditional classical speech codecs and existing neural speech codecs in terms of reconstructed speech naturalness and quality while maintaining the low latency characteristic of neural speech codecs. With a multi-stimulus test with hidden reference and anchor (MUSHRA) score of 87, it is highly suitable for ultra-low bit rate speech compression applications such as satellite speech communication and narrowband instant messaging. The demo has been publicly released at https://svcodec.github.io/.