Self-supervised random forests for robust voice activity detection with limited labeled data

Voice activity detection is essential for various downstream speech-related applications. Existing deep learning models for voice activity detection and speech recognition are available, but they often require substantial annotated data and assume noise-free environments. This limitation hinders their application to the vast but sparsely labeled audio datasets available. To address this gap, we propose a novel approach: self-supervised random forest voice activity detection (SSRF-VAD), designed for noisy environments and limited labeled data. We integrate a set of five handcrafted features to optimize performance under mixed signal-to-noise ratios (SNRs). The study incorporates various noise classes covering diverse environmental sounds such as urban sounds, water sounds, indoor appliances, and animals. Our SSRF-VAD approach achieves an improvement of 3 % in F1-score using only 20 % of the labeled training data compared to state-of-the-art MarbleNet model trained on the complete training dataset. Feature selection, implemented using two distinct feature importance techniques, SHAP and GINI, reduces the feature vector dimensionality by 75 % while preserving accuracy. Further, a novel three-class classification for separating clean speech, noisy speech, and non-speech audio segments with the proposed technique achieves 98.74 % accuracy with 0.982 F1-score. This framework enhances speech analysis and noise characterization, contributing to efficient speech enhancement. Thus, the proposed SSRF-VAD method reduces the requirement for labeled data and can be implemented on resource-constrained devices such as smart hearing aids and smart home assistants.