Generative artificial intelligence and automated scoring in curriculum-based measurement for elementary students with early writing needs

Abstract

Writing assessments, including curriculum-based measurement (CBM), face feasibility challenges because manual scoring becomes time- and resource-intensive when large numbers of writing samples must be evaluated. Automated scoring offers a promising complement, and rapid advances in generative artificial intelligence (AI) further expand its potential. However, empirical evidence on automated scoring of brief CBM writing samples remains limited, and potential scoring bias remains underexplored. We evaluated automated CBM scores generated by a feature-based system (writeAlizer) and a generative AI system (Gemini) using three-minute passage-level CBM Story Prompt responses produced by elementary students identified as having intensive early writing needs. We examined (a) consistency and agreement between automated scores and trained raters’ hand-calculated scores, (b) criterion-related validity in relation to a standardized writing assessment, and (c) subgroup bias. Overall, scoring performance varied across CBM metrics, with stronger agreement observed for simpler metrics. Automated and hand-calculated scores demonstrated comparable criterion-related validity, though correlations with the standardized measure were modest. No systematic differences in scoring accuracy were found across student race/ethnicity, sex, or English Learner status. Findings highlight the potential of automated scoring and the need for continued research before it can be recommended for scoring brief CBM writing samples in beginning writers.