What makes listening comprehension difficult?: A feature-based machine learning approach to understanding item difficulty

Abstract

Understanding what makes second language (L2) listening comprehension difficult is crucial for advancing language learning and assessment. In L2 listening assessment, a key challenge is developing items with targeted difficulty levels. This difficulty can be influenced by textual and acoustic features from different item segments (i.e. stimuli, stems, and options) embedded in a multi-layered structure, along with task-related features. This study explores a feature-based machine learning (ML) approach to predicting difficulty of multiple-choice listening items on a local language proficiency test. We extracted construct-relevant textual and acoustic features from item segments across five dimensions: lexical complexity, syntactic complexity, fluency, pronunciation, and similarities among item segments. Incorporating these features, we compared traditional and mixed-effects ML models for predictive accuracy and interpretability. The best-performing model—a mixed-effects Ridge model with twenty-three features—achieved high accuracy (R2 = 0.860) and showed meaningful feature-difficulty relationships. This study presents methodological innovations for item difficulty modeling and offers practical implications for human- and machine-mediated item development. It also demonstrates potential of incorporating computational linguistics and ML in enhancing L2 listening assessment.