Predicting the CEFR level of English listening texts with machine learning methods

Comprehension in listening texts is often judged by lexical coverage. However, this might not be easily interpretable for language teachers. The CEFR is becoming increasingly influential due to its standardized descriptors across languages. Learners are often placed into classes based on proficiency level, therefore a CEFR level is likely more interpretable than lexical coverage when judging listening text difficulty. Machine learning methods have been used to predict the CEFR level of English reading texts and learner writing, but no such studies exist for listening. The current study hopes to bridge this gap by investigating the potential to predict the CEFR level of listening texts. A corpus of CEFR-labelled listening texts (728 texts, 345,104 words) was compiled for text classification. Three types of variables were created from the corpus data to evaluate comparative predictive accuracy. The first method used linguistic and acoustic features. The others used text embeddings, which represent semantic meaning. The data was split into four classes: A1, A2, B1, and B2+. The accuracy of each method was evaluated by comparing the predicted label in the test data with the label from the original text. The most accurate method used OpenAI embeddings and Support Vector Machines. The overall accuracy was 0.81, with macro averages of precision = 0.75, recall = 0.78, and f-score = 0.76, indicating balanced classification performance across CEFR levels. This method has the potential to predict the CEFR level of listening texts, which could help practitioners and researchers match learners and participants to appropriate listening texts.