When AI meets source use: Exploring ChatGPT's potential in L2 summary writing assessment

Integrated writing assessments, which require students to incorporate source material into their writing, pose unique challenges for human raters. Understanding how AI tools like ChatGPT perform in assessing such tasks has become critical. This study investigates ChatGPT's ability to score L2 summary writing compared to human raters, focusing on the differences in rating results across various writing criteria, and their decision-making process in assessing source use. Using Many-Facet Rasch Measurement (MFRM) analysis, ratings of 90 student essays by GPT_original, GPT_calibrated, and two human raters were analyzed. Results indicated that GPT_original was the strictest rater overall, particularly in language-focused criteria. While GPT_calibrated aligned more closely with human raters, it still exhibited significant gaps in assessing a source-use-related criterion. Qualitative analyses of raters' think-aloud protocols revealed ChatGPT's detailed, rule-based approach to identifying source use strategies but also its lack of contextual flexibility, often misjudging legitimate paraphrasing attempts and over-relying on surface-level cues. These findings highlight ChatGPT's potential as a supplementary rating tool for L2 integrated writing while underscoring its limitations in addressing the developmental and contextual aspects of assessing source use. Implications point to the need for further refinement through L2-specific training to better align ChatGPT's judgments with human standards.