Exploiting spatial information and target speaker phoneme loss for multichannel directional speech enhancement and recognition

Abstract

Directional speech extraction catches increasing attention recently in multichannel speech separation, as it focuses solely on extracting the target speech to make real-time communication (RTC) and automatic speech recognition (ASR) more productive. This work investigates a real-time multichannel neural framework for directional speech enhancement and recognition by exploiting the explicit spatial information derived from the microphone array geometry, and the implicit spatial information learned from a dedicated narrow-band network. In addition to the traditional signal-based loss functions, we further introduce a loss inspired by the ASR phoneme mismatch to guide the framework training towards the distortion-less target speech signals. Experimental results with simulated datasets show that the proposed framework significantly improves the speech quality of the target speaker locating at the specific direction in noisy and reverberant environments with interfering speakers. The improved ASR results with the real-recorded dataset of live conversations from the CHiME8 MMCSG Challenge further verify the effectiveness of our system for practical applications.