Gauging the validity of machine learning-based temporal feature annotation to measure fluency in speech automatically

Abstract

Machine learning (ML) techniques allow for automatically annotating various temporal speech features, particularly by the cascade connection of ML-based modules. Although such systems are expected to enhance scalability of second language (L2) speech research, their annotation accuracy is potentially moderated by speaking tasks and proficiency levels due to the mismatch between training and real-world data. Accordingly, we developed and validated an ML-based temporal feature annotation system on L2 English datasets split by speaking tasks (monologic vs. dialogic tasks) and proficiency levels, operationalized as overall fluency levels (low, mid vs. high). We compared the annotations by experts and the system in terms of the agreement between manual and automatic annotations, correlations between manual and automatic measures, and the predictive power for listener-based fluency judgments. Results showed a substantial degree of agreement in the annotations for monologic tasks and a general tendency of strong correlations between manual and automatic measures regardless of tasks and overall fluency levels. Furthermore, automatic measures yielded substantial predictive power of fluency scores in monologic tasks. These findings suggest the substantial applicability of ML-based annotation systems to monologic tasks possibly without biases by holistic levels of fluency.