Synthesis-to-real robust training for enhanced sound event localization and detection using dynamic kernel convolution networks

Abstract

Deep learning-based methods have shown high performance in sound event localization and detection (SELD). In real-world spatial sound environments, the presence of reverberation and the uneven distribution of different sound events increase the complexity of the SELD task. In this paper, we propose an effective SELD system in real spatial scenes. We first introduce a dynamic kernel convolution module with the convolution blocks to adaptively model the channel-wise features with different receptive fields. Secondly, we integrate two mainstream networks into the proposed SELD system with the multi-track activity-coupled Cartesian direction of arrival (ACCDOA). Moreover, two synthesis-to-real robust training strategies are introduced into the training stage to improve the system's generalization in realistic spatial sound scenes. Finally, we use data augmentation methods to extend the dataset using channel rotation, and spatial data synthesis. Four joint metrics are used to evaluate the performance of the SELD system on the Sony-TAu Realistic Spatial Soundscapes dataset. Experimental results show that the proposed systems outperform the fixed-kernel convolution SELD systems. In addition, the ensemble system achieves a SELD score of 0.348 in the DCASE SELD task and outperforms the SOTA methods.