Frame-Level Multiple Sound Sources Localization Based on Visual Understanding

Sound source localization is an important field of audio and visual research. In the dynamic performance stage, finding the positions of multiple sounding objects in real time can give the audience an immersive feeling. Due to the complexity of the performance scene, it is a challenge to perform audio-visual recognition and localization because of the audio overlapping and visual object masking. To address this problem, we propose a novel two-stream learning framework that disentangles different classes of audio-visual representations from complex scenes, then maps the audio area of each visual in multi-instance labels learning through adaptive multi-stream fusion, and localizes sounding instrument from coarse to fine. We have obtained the state-of-the-art results on the public dataset. Experiment results show that our method can effectively realize frame-level multiple sound sources location.