Effects of script presentation mode on rater scoring: A cross-classified multilevel modelling approach

Abstract

This study examined rater scoring in L2 writing assessment under two conditions of onscreen marking: scanned handwritten scripts and word-processed versions of the same scripts. Fifty-three raters marked both versions of 30 scripts presented to them on a computer screen, using an analytical rating scale. A cross-classified multilevel modelling approach was employed to analyse both the overall score and four subscores. The findings indicated that the word-processed scripts were, on average, scored significantly lower than their handwritten counterparts by 0.05–0.11 SDs. Given that the desired mode effect is 0 SD, these differences could be considered substantial in high-stakes examination contexts. Lower scores for word-processed scripts were not attributable to specific scripts being penalised, but rather reflected variability in individual rater scoring behaviour. There was significant variation among raters in the degree to which they scored word-processed scripts more severely than handwritten scripts across both the overall and four subscale scores, with the grammar criterion showing the widest spread in rater-specific mode effects. Practical implications of the findings are discussed in relation to rating conditions, rater training, and automated scoring.