AI tools vs human in english narrative writing scoring: A comparative study of scoring agreement among human raters, DeepSeek and ChatGPT in exam context

Abstract

A growing body of literature is examining the agreement between artificial intelligence (AI) and humans in scoring. However, empirical findings remain inconclusive, ranging from a strong correlation to a large difference in the scores assigned by the two media. In response, this mixed-methods study compared scoring agreement and evaluation rationales across three sources, including ChatGPT-4, DeepSeek-R1 and human ratings from a city-level standarised English narrative writing examination (n = 212). Quantitatively, the findings revealed strong relative agreement between the two AI models, while only moderate alignment was observed between each AI tool and the human ratings. Qualitative analyses of a focus group interview and scoring rationales indicated that these discrepancies reflect systematic differences in evaluation orientation. Human raters demonstrated more flexibility and sensitivity to contextual and presentation-related factors in evaluating writing quality, whereas AI tools prioritised linguistic accuracy and structural regularity and remained unaffected by surface features such as handwriting or erasures. Rather than suggesting simple scoring errors, the results reveal systematic differences between human and AI judgment, with implications for the principled integration of AI into high-stakes assessment.