Environmental sound classification using two-stream deep neural network with interactive time-frequency attention

Abstract

Environmental sound classification (ESC) is crucial for understanding urban acoustic environments, and current classification methods cannot strike a balance between model performance and computational efficiency. In this paper, we propose a novel approach for modeling interactive time-frequency attention at different network levels using a two-stream deep neural network, namely ITFA-DNN. It uses two branches to capture time and frequency global dependencies at different layers of the network, respectively, introduces an inter-branch information interaction mechanism to enhance learning efficiency, and leverages depthwise separable convolutions to reduce model size. On two datasets, ESC-50 and UrbanSound8K, the proposed model achieves classification accuracy of 94.2% and 95.3%. It outperforms other models in ESC-50, and is comparable to state-of-the-art models in UrbanSound8K while reducing computational complexity by over 90%. Results from ablation experiments and visualizations further demonstrate that the proposed method effectively captures the global time-frequency dependencies of sounds in complex environments while improving computational efficiency. These highlight the superiority of the proposed method in environmental sound classification.