Multivariate time series approach integrating cross-temporal and cross-channel attention for dysarthria detection from speech

Abstract

Speech analysis offers a non-invasive, low-cost approach to dysarthria detection. Studies have shown that the temporal correlations within speech signals and the interactions among the multidimensional feature variables derived from them can facilitate dysarthria detection. However, current studies either rely on pre-designed feature sets, which depend heavily on cumbersome feature engineering, or focus solely on spectral or high-dimensional audio vectors that capture temporal dependencies while neglecting the interactions between internal multivariate features. We propose an end-to-end method that utilizes audio pre-trained models as multivariate time series feature extractors, combined with InceptionTime and cross-temporal and cross-channel attention mechanisms, to fully capture temporal dependencies and interactions among variables within speech for accurate dysarthria detection. Results show that the proposed method achieves a detection accuracy of 92.06 % on a local Mandarin dysarthria dataset, which is at least 2.17 percentage points higher than previous studies, with the highest stability and the lowest time cost. Furthermore, it achieves an accuracy of 87.73 % on an external English dataset, demonstrating good cross-linguistic adaptability and generalizability. Additionally, experiments show that in connected speech tasks, structured tasks outperform unstructured ones in leveraging interactions, leading to more effective dysarthria detection. These findings validate the effectiveness of the proposed end-to-end dysarthria detection method, further advancing the development of speech analysis as a promising tool for dysarthria screening.