Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-12DOI: 10.1016/j.asw.2026.101132
Yao Fang, Haomin Zhang
{"title":"Rating reliability and rater effects in scoring the Gaokao English story continuation writing task: Evidence from city-level mock examinations","authors":"Yao Fang, Haomin Zhang","doi":"10.1016/j.asw.2026.101132","DOIUrl":"10.1016/j.asw.2026.101132","url":null,"abstract":"<div><div>The scoring of integrated writing tasks in high-stakes examinations is inherently rater-mediated, raising persistent concerns about score reliability. Focusing on the Story Continuation Writing Task (SCWT) in China’s National College Entrance Examination (Gaokao), this study examines rating reliability and rater effects under near-operational marking conditions. A stratified sample of 180 SCWT scripts from two city-level mock examinations was rated in two scoring rounds by 30 teacher raters recruited from the provincial Gaokao marking pool, with construct-oriented calibration conducted between the two rounds. Two expert raters provided external reference ratings. Rating quality was examined using classical agreement indices and many-facet Rasch measurement (MFRM), supplemented by questionnaire evidence on raters’ perceptions of scale clarity and scoring stability. Results showed moderate-to-high inter-rater reliability across both rounds, alongside systematic severity differences among raters. MFRM results further indicated a small estimated difference between the two rounds, although rater- and panel-related variation remained detectable. Overall, the findings provide evidence relevant to the feasibility and monitoring of SCWT scoring under Gaokao-like conditions, while highlighting the value of combining classical and facet-based approaches for monitoring rater-mediated variability in large-scale writing assessment.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101132"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199643","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-13DOI: 10.1016/j.asw.2026.101130
Daniel Yu-Sheng Chang, Guoxing Yu, George Leckie
{"title":"Effects of script presentation mode on rater scoring: A cross-classified multilevel modelling approach","authors":"Daniel Yu-Sheng Chang, Guoxing Yu, George Leckie","doi":"10.1016/j.asw.2026.101130","DOIUrl":"10.1016/j.asw.2026.101130","url":null,"abstract":"<div><div>This study examined rater scoring in L2 writing assessment under two conditions of onscreen marking: scanned handwritten scripts and word-processed versions of the same scripts. Fifty-three raters marked both versions of 30 scripts presented to them on a computer screen, using an analytical rating scale. A cross-classified multilevel modelling approach was employed to analyse both the overall score and four subscores. The findings indicated that the word-processed scripts were, on average, scored significantly lower than their handwritten counterparts by 0.05–0.11 SDs. Given that the desired mode effect is 0 SD, these differences could be considered substantial in high-stakes examination contexts. Lower scores for word-processed scripts were not attributable to specific scripts being penalised, but rather reflected variability in individual rater scoring behaviour. There was significant variation among raters in the degree to which they scored word-processed scripts more severely than handwritten scripts across both the overall and four subscale scores, with the grammar criterion showing the widest spread in rater-specific mode effects. Practical implications of the findings are discussed in relation to rating conditions, rater training, and automated scoring.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101130"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199646","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-17DOI: 10.1016/j.asw.2026.101135
Burcu Gündoğdu, Ali Merç
{"title":"Exploring teachers’ dispositions toward AI feedback accuracy and the refinement process and their enhancement practices in EFL writing","authors":"Burcu Gündoğdu, Ali Merç","doi":"10.1016/j.asw.2026.101135","DOIUrl":"10.1016/j.asw.2026.101135","url":null,"abstract":"<div><div>AI chatbots are increasingly used as writing aids and feedback providers; however, their classroom implementation remains limited due to inconsistent feedback accuracy, highlighting the need for teachers to act as feedback curators who refine and contextualize AI-generated feedback. However, no study has systematically examined the scope, nature, and focal areas of teachers’ refinements of AI-generated feedback. The present study documents teacher interventions using color-coded annotations and structured review forms, examines teachers’ evaluations of feedback accuracy through accuracy ratings, and explores their perceptions of classroom usability through interviews. The findings indicated that AI frequently overlooked meaning-level problems and instead focused on surface-level errors; in densely errored papers, major meaning-related issues often remained unresolved. While error-coded feedback was dominated by surface-level corrections, neglecting clarity and structural concerns, content feedback showed limited ability to detect repetitive ideas and organizational flaws, which were readily identified by teachers. Consequently, teachers described the filtering process of error-coded feedback as burdensome, whereas content feedback was perceived as more useful and easier to enhance through teacher-led prioritization adjustments addressing students’ actual needs. The findings highlighted the key areas of teacher refinements and offer insights for improving the accuracy and classroom applicability of AI-generated feedback through teacher prioritization strategies.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101135"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199649","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-09-28DOI: 10.1016/j.asw.2026.101126
Bayan Alkhathlan
{"title":"Collaborative agency in rating scale development for L2 academic writing assessment","authors":"Bayan Alkhathlan","doi":"10.1016/j.asw.2026.101126","DOIUrl":"10.1016/j.asw.2026.101126","url":null,"abstract":"<div><div>Calls for greater learner involvement in assessment have highlighted a need for rating scales that are not only technically sound but also pedagogically meaningful. However, in second language (L2) writing, students are largely absent from scale design, with most instruments being developed by teachers or test developers. Previous co-construction studies have often positioned students as the primary agents, with comparative underuse of teacher expertise and limited curricular alignment. This study addresses this gap by investigating how collaborative agency between teachers and students can inform the development of a rating scale for academic writing in a Saudi higher education context.</div><div>Drawing on a multi-source framework, four meshed teacher–student focus groups and a follow-up focus group were conducted to negotiate the scale criteria, properties, and level descriptors. Qualitative content analysis revealed both convergence and divergence: while grammar, organization, and development were prioritized, students argued for a stronger recognition of ideas and discourse-level features, whereas teachers emphasized accuracy and feasibility. Through cycles of debate and compromise, the process yielded a five-level analytic rating scale with six criteria grounded in curricular objectives and stakeholder perspectives.</div><div>This study shows how collaborative agency was enacted through a dialogic process of negotiating and integrating teacher and student perspectives in rating scale development. It extends participatory assessment research in L2 writing by illustrating a context-sensitive, theoretically informed, and curriculum-aligned approach to analytic rating scale co-construction for classroom use.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101126"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149140576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-17DOI: 10.1016/j.asw.2026.101133
Junju Wang, Changmin Liu, Shengnan Kong
{"title":"Tracing the development of graduate students’ peer feedback literacy: A longitudinal multiple-case study","authors":"Junju Wang, Changmin Liu, Shengnan Kong","doi":"10.1016/j.asw.2026.101133","DOIUrl":"10.1016/j.asw.2026.101133","url":null,"abstract":"<div><div>Recent research has highlighted the importance of peer feedback literacy in academic writing, yet its longitudinal development among graduate students remains underexplored. This longitudinal multiple-case study traced the development of peer feedback literacy among four Chinese graduate students working in two fixed provider–receiver dyads across two peer feedback cycles. Drawing on drafts, peer comments, multimodal interaction records, and interviews, the analysis identified cross-case commonalities and role-related variation. The findings revealed a shared three-stage trajectory, moving from initial constraints on productive participation in peer feedback, through the recalibration of feedback orientations and practices, to more coordinated, strategic, and collaborative feedback practices. Within this trajectory, development took different forms across cases: one receiver showed growing confidence in evaluating feedback, whereas the other developed greater trust in the feedback relationship; among the providers, one produced increasingly explicit and actionable feedback, whereas the other adopted a more adaptive and relationally sensitive evaluative stance. The findings suggest that peer feedback literacy is a role-differentiated and socially co-constructed configuration of capacities, dispositions, and practices shaped by affective, relational, and interactional conditions. They offer pedagogical implications for creating emotionally safe peer-feedback environments, scaffolding feedback provision, and structuring opportunities for digitally mediated dialogue.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101133"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199648","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-17DOI: 10.1016/j.asw.2026.101134
Seohyeon Choi, Jechun An, Manjary Guha, Emily A. Reno, Emma Shanahan, Adrea J. Truckenmiller, Kristen L. McMaster
{"title":"Generative artificial intelligence and automated scoring in curriculum-based measurement for elementary students with early writing needs","authors":"Seohyeon Choi, Jechun An, Manjary Guha, Emily A. Reno, Emma Shanahan, Adrea J. Truckenmiller, Kristen L. McMaster","doi":"10.1016/j.asw.2026.101134","DOIUrl":"10.1016/j.asw.2026.101134","url":null,"abstract":"<div><div>Writing assessments, including curriculum-based measurement (CBM), face feasibility challenges because manual scoring becomes time- and resource-intensive when large numbers of writing samples must be evaluated. Automated scoring offers a promising complement, and rapid advances in generative artificial intelligence (AI) further expand its potential. However, empirical evidence on automated scoring of brief CBM writing samples remains limited, and potential scoring bias remains underexplored. We evaluated automated CBM scores generated by a feature-based system (writeAlizer) and a generative AI system (Gemini) using three-minute passage-level CBM Story Prompt responses produced by elementary students identified as having intensive early writing needs. We examined (a) consistency and agreement between automated scores and trained raters’ hand-calculated scores, (b) criterion-related validity in relation to a standardized writing assessment, and (c) subgroup bias. Overall, scoring performance varied across CBM metrics, with stronger agreement observed for simpler metrics. Automated and hand-calculated scores demonstrated comparable criterion-related validity, though correlations with the standardized measure were modest. No systematic differences in scoring accuracy were found across student race/ethnicity, sex, or English Learner status. Findings highlight the potential of automated scoring and the need for continued research before it can be recommended for scoring brief CBM writing samples in beginning writers.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101134"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199645","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-15DOI: 10.1016/j.asw.2026.101131
Pauline Mak
{"title":"Harnessing writing portfolios to foster student feedback literacy and text revision in second language writing","authors":"Pauline Mak","doi":"10.1016/j.asw.2026.101131","DOIUrl":"10.1016/j.asw.2026.101131","url":null,"abstract":"<div><div>Feedback is a crucial driver of student learning. To maximize the efficacy of feedback, students must possess feedback literacy or the capacities to actively engage in feedback processes. In second language writing contexts, conventional feedback practice appears to limit opportunities for students to develop their evaluation capacities to use feedback productively and hinders their development of feedback literacy. Against this background, this study, involving four Grade 5 classes of one elementary school in Hong Kong, examines the development of student feedback literacy through the use of writing portfolios and investigates how, and to what extent such development may influence text revision. Drawing on data from individual interviews with teachers and students, stimulated recall with students, classroom observations, students’ writing portfolios and students’ drafts with feedback over the course of one academic year, results indicate that the students developed better understanding of feedback, possessed enhanced capacities to make productive use of feedback and held a more positive attitude towards their active role in the feedback process. Such enhanced writing feedback literacy may have facilitated text revisions and contribute to writing improvement. Implications for student feedback literacy development will be discussed.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101131"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149199644","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-07DOI: 10.1016/j.asw.2026.101128
Sophia Minnillo, Dana R. Ferris
{"title":"It takes a village: Feedback providers and multilingual graduate students’ writing self-perceptions","authors":"Sophia Minnillo, Dana R. Ferris","doi":"10.1016/j.asw.2026.101128","DOIUrl":"10.1016/j.asw.2026.101128","url":null,"abstract":"<div><div>Despite the importance of feedback for writing development and academic socialization, little is known about multilingual graduate students’ experiences with different feedback providers during their early years of graduate study. This article charts students’ responses to various feedback providers and examines how students characterize the impact of feedback from different sources on their perceptions of themselves as academic writers. We interviewed 12 multilingual STEM graduate students over the course of their first year in a graduate program and continued interviewing three of the students in their second year. Inductive and deductive qualitative analysis revealed that students varied in the extent to which they received feedback from supervisors, disciplinary instructors, and peers. Many students felt dissatisfied that they lacked a “village” of feedback providers, leading them to elicit comments from GenAI tools. Students prioritized relationships, expertise, availability, and helpfulness in feedback providers, and they connected their reactions to comments with previous feedback experiences. This study illustrates the applicability of Social Cognitive Theory as a lens for understanding how both internal and external factors, including past performance, shape the relationship between feedback experiences and students' writing self-perceptions. Implications for L2 graduate writing education and mentorship are discussed.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101128"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149140579","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-10-08DOI: 10.1016/j.asw.2026.101129
Apichat Khamboonruang, Ute Knoch
{"title":"Assessing the assessors: GenAI and human ratings of second language writing using analytic and holistic rubrics","authors":"Apichat Khamboonruang, Ute Knoch","doi":"10.1016/j.asw.2026.101129","DOIUrl":"10.1016/j.asw.2026.101129","url":null,"abstract":"<div><div>With the evolution of generative artificial intelligence (GenAI) technology, there has been a surge in research exploring its potential for automated essay scoring (AES). However, few studies have investigated the AES capabilities of multiple GenAI models within language classrooms nor have studies compared GenAI performances when scoring different types of rating scales. Therefore, this study aims to compare the analytic and holistic ratings of GenAI models (e.g., ChatGPT, Claude, Gemini, Copilot, DeepSeek) and nine experienced teachers on 104 students’ opinion paragraphs in a Thai EFL classroom context. GenAI models scored each paragraph following structured, rubric-aligned prompting. Data were analysed using Spearman’s rank-order correlation and many-facet Rasch measurement. Findings revealed that overall, experienced teachers demonstrated relatively less extreme severity estimates, better-fitting ratings, and broadly satisfactory score differentiation across rubrics. GenAI models showed better alignment with human ratings for holistic scoring and language than for task completion and content, and organisation. While approaching human rating behaviours, GenAI models’ rating performances varied across scoring rounds, rubrics, and traits. The findings support supervised application of GenAI models as a supplementary scoring assistant in classroom assessment, with teacher oversight remaining essential. This study offers implications for validating and integrating GenAI-assisted AES into L2 writing classrooms.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101129"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149140577","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Assessing WritingPub Date : 2026-10-01Epub Date: 2026-09-29DOI: 10.1016/j.asw.2026.101125
Chenze Wu, Ricky Lam
{"title":"Generative artificial intelligence in writing assessment: University professors’ perspectives and recommendations","authors":"Chenze Wu, Ricky Lam","doi":"10.1016/j.asw.2026.101125","DOIUrl":"10.1016/j.asw.2026.101125","url":null,"abstract":"<div><div>Research has recognized the potential of Generative AI (GenAI) to support writing assessment. However, the responsible and effective application of GenAI in writing assessment remains underexplored, and consequently, university instructors are uncertain about how to integrate these tools into their assessment practices. Drawing on Mills et al.’s (2024) three-dimensional GenAI literacy framework, this qualitative study examined professors’ beliefs about incorporating GenAI into L2 writing assessment and their recommendations for responsible and effective integration. Twenty-four professors from universities in Chinese Mainland, Hong Kong, Macau, and Singapore were recruited. Data were collected through semi-structured interviews and stimulated recall tasks. The findings revealed that while professors recognized benefits of incorporating GenAI in writing assessment, they expressed concerns about certain critical issues. They emphasized the essential role of human agency and offered practical recommendations for effective, responsible, and supportive use. Based on these findings, the study proposes seven evidence-based and actionable recommendations, including enhancing instructors’ GenAI literacy, adopting multiple sources of feedback, prioritizing teacher judgment and modification for reliability, validity, and discriminative power, ensuring transparent and ethical use of GenAI, foster socio-emotional awareness, as well as establishing and implementing institutional policies and training for GenAI adoption in writing assessment. These recommendations help guide the development of institutional policies and enable university instructors of L2 writing and other disciplines to integrate GenAI into assessment practices with responsibility, ethics, and effectiveness.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"70 ","pages":"Article 101125"},"PeriodicalIF":7.6,"publicationDate":"2026-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149140578","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}