Single channel speech denoising by DDPG reinforcement learning agent

Abstract

Speech denoising (SD) covers the algorithms that suppress the background noise from the contaminated speech and improve its clarity. In this paper, a novel SD algorithm is presented based on the deep deterministic policy gradient (DDPG) agent; an off-policy reinforcement learning (RL) agent with a continuous action space. The noisy speech is first converted from the time domain to the time–frequency (TF) domain by taking its short-time Fourier transform (STFT), and then two separate DDPG agents are trained on the magnitude and phase components of the STFT. The reward function used for training these agents is the relative perceptual quality score of speech. After training, the DDPG agents generate the magnitude and phase soft masks, when noisy speech is given as input to them. These masks are then applied to the complex STFT matrix of the noisy speech to obtain the denoised speech. For matched testing data, the proposed system offers an improvement of 1.55 points in the perceptual evaluation of speech quality (PESQ) over the unprocessed speech, the highest among the other recent state-of-the-art models used for comparison in this paper. It achieves this performance by utilizing data that is 7 times smaller than that required by other models. Also, its learnable parameters are the lowest among all models, almost 12 times less than the next most compact model, based on another continuous RL agent (policy gradient (PG)) for estimating its convolutional kernels. When cascaded with a coloured spectrogram-based SD model, the proposed model further improves the PESQ by 0.07, CSIG by 1.23, and COVL by 1.4 points; the metrics estimating respectively the perceived quality of speech, its composite distortion, and its composite overall quality, surpassing all other baseline systems compared here. In the cascaded configuration, our proposed model offers the highest gain in PESQ (by 0.78 points) at 50 times fewer episodes, compared to the already trained speech enhancement and recognition models utilizing discrete agents for their performance improvement.