Multiple Transformer-Based Models for Phase-Sensitive Masking-Based Single-Channel Speech Enhancement

Abstract

Leveraging both global contextual dependencies and local temporal-spectral patterns can further enhance speech quality and intelligibility, motivating the integration of diverse neural network structures for improved mask estimation. Furthermore, due to the limitations of existing time-frequency phase-aware masks, a new constrained phase-sensitive mask is introduced and estimated using the proposed architectures. In this paper, we first propose a modified transformer model for constrained phase-sensitive ideal magnitude ratio mask (cPSIRM) estimation (TRF-cPSIRM) to incorporate both magnitude and phase information and improve speech enhancement quality. The transformer model can extract global information, whereas the convolutional neural network (CNN) and convolutional recurrent network (CRN) are effective in capturing local content. CNNs have feature extraction capability due to their convolutional layers, and CRNs benefit from the temporal modelling strength of LSTM recurrent layers, which is useful for enhancement. Therefore, in this paper, to exploit both transformer (TRF) and CNN/CRN capabilities, we propose the cascaded structures (models) of CNN and TRF layers (CNN-TRF-cPSIRM) and also TRF and CRN layers (TRF-CRN-cPSIRM) for cPSIRM estimation and, consequently, speech enhancement. The CNN is used for feature extraction in CNN-TRF-cPSIRM and CRN for enhancement in TRF-CRN-cPSIRM. Moreover, considering the harmonic property of speech in the frequency spectrum, we present the transposed transformer-based model (TTRF), in which the neighbourhood relationship between the different frequency sub-bands is used as a sequence in the model. Then, to model both the long-term and short-term dependencies, the cascaded model of TTRF (intra-transformer) and TRF (inter-transformer) is proposed for cPSIRM estimation (TTRF-TRF-cPSIRM). The experimental results show that among these transformer-based models, the CNN-TRF-cPSIRM has the best performance, achieving up to about 0.44 perceptual evaluation of speech quality (PESQ) improvement over the baseline and 1.5 over the noisy speech.