Rating reliability and rater effects in scoring the Gaokao English story continuation writing task: Evidence from city-level mock examinations

Abstract

The scoring of integrated writing tasks in high-stakes examinations is inherently rater-mediated, raising persistent concerns about score reliability. Focusing on the Story Continuation Writing Task (SCWT) in China’s National College Entrance Examination (Gaokao), this study examines rating reliability and rater effects under near-operational marking conditions. A stratified sample of 180 SCWT scripts from two city-level mock examinations was rated in two scoring rounds by 30 teacher raters recruited from the provincial Gaokao marking pool, with construct-oriented calibration conducted between the two rounds. Two expert raters provided external reference ratings. Rating quality was examined using classical agreement indices and many-facet Rasch measurement (MFRM), supplemented by questionnaire evidence on raters’ perceptions of scale clarity and scoring stability. Results showed moderate-to-high inter-rater reliability across both rounds, alongside systematic severity differences among raters. MFRM results further indicated a small estimated difference between the two rounds, although rater- and panel-related variation remained detectable. Overall, the findings provide evidence relevant to the feasibility and monitoring of SCWT scoring under Gaokao-like conditions, while highlighting the value of combining classical and facet-based approaches for monitoring rater-mediated variability in large-scale writing assessment.