Multimodal and multichannel speech separation using location-guided speech feature mapping network

In reality, the audio and visual signals of sound sources are closely aligned, working collaboratively to isolate the desired speech signal from overlapping voices of simultaneous talkers. To leverage the complementarity and utilize all available information from both auditory and visual sources in speech separation, we propose a novel robust multimodal and multichannel speech separation method, without requiring known camera parameters. The proposed method exploits the complementarity of audio and visual modalities to estimate the speaker’s location and adopts a location-guided speech feature mapping strategy, wherein the attention mechanism fusion method combines the high-level semantic information of auditory and visual sources, aiding in the separation of target speech with its corresponding directional features. Experimental results suggest that the proposed multimodal and multichannel speech separation system outperforms the baselines, demonstrating improvements of 0.64 dB in SI-SDR and 0.17 in PESQ, respectively. The proposed system consistently outperformed the baselines by achieving a 10.14 % absolute (26.05 % relative) word error rate (WER) reduction.