Speech Disfluency Detection with Contextual Representation and Data Distillation

Abstract

Stuttering affects almost 1\% of the world's population. It has a deep sociological impact and hinders the people who stutter from taking advantage of voice-assisted services. Automatic stutter detection based on deep learning can help voice assistants to adapt themselves to atypical speech. However, disfluency data is very limited and expensive to generate. We propose a set of preprocessing techniques: (1) using data with high inter-annotator agreement, (2) balancing different classes, and (3) using contextual embeddings from a pretrained network. We then design a disfluency classification network (DisfluencyNet) for automated speech disfluency detection that takes these contextual embeddings as an input. We empirically demonstrate high performance using only a quarter of the data for training. We conduct experiments with different training data size, evaluate the model trained on the lowest amount of training data with SEP-28k baseline results, and evaluate the same model on the FluencyBank dataset baseline results. We observe that, even by using a quarter of the original size of the dataset, our F1 score is greater than 0.7 for all types of disfluencies except one,\textit{ blocks}. Previous works also reported lower performance with \textit{blocks} type of disfluency owing to its large diversity amongst speakers and events. Overall, with our approach using only a few minutes of data, we can train a robust network that outperforms the baseline results for all disfluencies by at least 5\%. Such a result is important to stress the fact that we can now reduce the required amount of training data and are able to improve the quality of the dataset by appointing more than two annotators for labeling speech disfluency within a constrained labeling budget.