Assessing Writing最新文献_第7页

Judgment accuracy in primary school EFL writing assessment: Do text characteristics matter? 小学英语写作评价的判断准确性：文本特征重要吗？

IF 4.2 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-06-26 DOI: 10.1016/j.asw.2025.100957

Ruth Trüb , Jens Möller , Julian Lohmann , Thorben Jansen , Stefan D. Keller

{"title":"Judgment accuracy in primary school EFL writing assessment: Do text characteristics matter?","authors":"Ruth Trüb , Jens Möller , Julian Lohmann , Thorben Jansen , Stefan D. Keller","doi":"10.1016/j.asw.2025.100957","DOIUrl":"10.1016/j.asw.2025.100957","url":null,"abstract":"<div><div>Assessing the writing competence of pupils learning English as a foreign language (EFL) at primary school is challenging. This study aimed at examining a largely unexplored topic, namely the role of text characteristics in writing assessment, and analysed judgment accuracy differentiated by nine aspects of text quality (communicative effect, level of detail, coherence, cohesion, complexity of syntax and grammar, correctness of syntax and grammar, vocabulary, orthography and punctuation). Two hundred pre-service teachers assessed four randomly assigned texts from learners in grade six. Their assessment was compared to the existing ratings of two experts from a previous study. We found a relative judgment accuracy between <em>r</em> = .34 and .60 for the nine assessment criteria, with vocabulary being assessed significantly more accurately than almost all other criteria. Orthography, complexity and correctness of syntax and grammar and punctuation were rated with significantly more accuracy than cohesion, level of detail, communicative effect and coherence. The pre-service teachers assessed most criteria more strictly and with higher variability than the experts. The results suggest that teacher education should offer pre-service teachers concrete opportunities to practise writing assessment, implement activities to strengthen the assessment of content- and structure-related criteria, and help them adjust their assessment rigour.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100957"},"PeriodicalIF":4.2,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144490215","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Using ChatGPT to score essays and short-form constructed responses 使用ChatGPT对文章和短文进行评分

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-10-13 DOI: 10.1016/j.asw.2025.100988

Mark D. Shermis

{"title":"Using ChatGPT to score essays and short-form constructed responses","authors":"Mark D. Shermis","doi":"10.1016/j.asw.2025.100988","DOIUrl":"10.1016/j.asw.2025.100988","url":null,"abstract":"<div><div>This study evaluates the effectiveness of ChatGPT-4o in scoring essays and short-form constructed responses compared to human raters and traditional machine learning models. Using data from the Automated Student Assessment Prize (ASAP), ChatGPT’s performance was assessed across multiple predictive models, including linear regression, random forest, gradient boost, and XGBoost. Results indicate that while ChatGPT’s gradient boost model achieved quadratic weighted kappa (QWK) scores close to human raters for some datasets, overall performance remained inconsistent, particularly for short-form responses. The study highlights key challenges, including variability in scoring accuracy, potential biases, and limitations in aligning ChatGPT’s predictions with human scoring standards. While ChatGPT demonstrated efficiency and scalability, its leniency and variability suggest that it should not yet replace human raters in high-stakes assessments. Instead, a hybrid approach combining AI with empirical scoring models may improve reliability. Future research should focus on refining AI-driven scoring models through enhanced fine-tuning, bias mitigation, and validation with broader datasets. Ethical considerations, including fairness in automated scoring and data security, must also be addressed. This study concludes that ChatGPT holds promise as a supplementary tool in educational assessment but requires further development to ensure validity and fairness.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100988"},"PeriodicalIF":5.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145320230","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Response time for English learners on large-scale writing assessments 英语学习者在大规模写作评估中的反应时间

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-08-13 DOI: 10.1016/j.asw.2025.100979

Catherine Welch, Stephen Dunbar, Jeongmin Ji, Annette Vernon, Junhee Park

引用次数: 0

Comparing GPT-based approaches in automated writing evaluation 比较基于gpt的自动写作评价方法

IF 4.2 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-06-24 DOI: 10.1016/j.asw.2025.100961

Yingying Liu , Xiaofei Lu , Huilei Qi

{"title":"Comparing GPT-based approaches in automated writing evaluation","authors":"Yingying Liu , Xiaofei Lu , Huilei Qi","doi":"10.1016/j.asw.2025.100961","DOIUrl":"10.1016/j.asw.2025.100961","url":null,"abstract":"<div><div>Large language models (LLMs) like OpenAI’s GPT models show significant promise in automated writing evaluation (AWE). However, recent research has mainly focused on non-fine-tuned GPT models, with limited attention to fine-tuned models as well as potential factors influencing performance, such as model type, prompting strategy, and dataset characteristics. This study compares six GPT-based approaches for evaluating TOFEL argumentative writing, namely, GPT-3.5 zero-shot, GPT-3.5 few-shot, GPT-4 zero-shot, GPT-4 few-shot, and two fine-tuning methods. We assess the impact of model type (GPT-3.5 vs. GPT-4), prompting strategy (zero-shot vs. few-shot), fine-tuning, class imbalance and dataset shift on performance. Our findings reveal that fine-tuned GPT models consistently outperform non-fine-tuned GPT-4 models, which in turn outperform GPT-3.5 models. Few-shot prompting does not show clear advantages over zero-shot prompting in this study. Additionally, class imbalance and dataset shift negatively affect model accuracy and reliability. These results offer valuable insights into the effectiveness of different GPT-based approaches and the factors that influence their performance in AWE.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100961"},"PeriodicalIF":4.2,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144471912","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Exploring the scoring validity of holistic and dimension-based Comparative Judgements of young learners’ EFL writing 探讨青少年英语写作整体与维度比较判断的评分效度

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-10-15 DOI: 10.1016/j.asw.2025.100986

Rebecca Sickinger , John Pill , Tineke Brunfaut

{"title":"Exploring the scoring validity of holistic and dimension-based Comparative Judgements of young learners’ EFL writing","authors":"Rebecca Sickinger , John Pill , Tineke Brunfaut","doi":"10.1016/j.asw.2025.100986","DOIUrl":"10.1016/j.asw.2025.100986","url":null,"abstract":"<div><div>Comparative Judgement (CJ) is a pairwise comparison evaluation method, typically conducted online. Multiple judges each compare the quality of a series of paired performances and, from their decisions, a rank order is constructed and scores calculated. Research across different educational contexts supports CJ’s reliability for evaluating written performances, permitting more precise scoring of scripts and for dimension-focused evaluation. However, scant insights are available about the basis of judges’ evaluations. This issue is important because argument-based approaches to validation (common in the field of language testing and adopted in this study) require evidence to support claims about how scores are appropriate for test purpose. Therefore, we investigate the scoring validity of CJ, both when used holistically (the standard application of CJ) and when evaluating scripts by individual criteria (termed dimensions in the research context). Twenty-seven judges evaluated 300 scripts addressing two writing task types in a national English as a Foreign Language examination for young learners in Austria. Judges reported via questionnaires what they had focused on while judging. Subsequently, eight judges provided think-aloud data while evaluating 157 scripts, offering further insight into the writing features they considered and their decision-making during CJ. Findings showed that while most judges adapted a decision-making process similar to traditional rating methods, some adapted their method to accommodate the nature of CJ evaluation. Furthermore, results indicated that the judges considered construct-relevant criteria when using CJ, both holistically and by dimension, thus offering support to an argument for the appropriateness of using CJ in this context.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100986"},"PeriodicalIF":5.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145320231","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Linguistic predictors of L2 writing performance: Variations across genres 二语写作表现的语言预测因素：不同体裁的差异

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-09-30 DOI: 10.1016/j.asw.2025.100985

Weiwei Yang , Sara T. Cushing , Guoxing Yu

{"title":"Linguistic predictors of L2 writing performance: Variations across genres","authors":"Weiwei Yang , Sara T. Cushing , Guoxing Yu","doi":"10.1016/j.asw.2025.100985","DOIUrl":"10.1016/j.asw.2025.100985","url":null,"abstract":"<div><div>This study investigated how linguistic complexity (including lexical and syntactic complexity), accuracy, and fluency (CAF) predicted second language (L2) writing scores across four essay genres: narration, exposition, expo-argumentation and argumentation. Approximately 60 essays were collected on each of these genres on the same subject matter and were scored using a holistic rubric. Eight measures of complexity, accuracy and fluency were examined. Forward stepwise regression analysis based on Akaike Information Criterion Corrected (AICC) was conducted for each genre. The findings revealed a large amount of score variance explained by CAF: 61 % for the argumentative task and about 70 % for the other three tasks. Fluency was found to be a highly important score predictor for the narrative and expository tasks, while lexical sophistication was equally important or more important than fluency for the expo-argumentative and argumentative tasks. The regression model for the narrative task also differed from those for the expository, argumentative task types, regarding syntactic complexity predictors. Lexical diversity was generally less important in predicting scores than lexical sophistication. The implications of the findings for L2 writing scoring and automated essay scoring are discussed.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100985"},"PeriodicalIF":5.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145219003","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving writing feedback quality and self-efficacy of pre-service teachers in Gen-AI contexts: An experimental mixed-method design 提高Gen-AI情境下职前教师写作反馈质量和自我效能：一种实验性混合方法设计

IF 4.2 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-06-19 DOI: 10.1016/j.asw.2025.100960

Siyu Zhu , Qingyang Li , Yuan Yao , Jialin Li , Xinhua Zhu

{"title":"Improving writing feedback quality and self-efficacy of pre-service teachers in Gen-AI contexts: An experimental mixed-method design","authors":"Siyu Zhu , Qingyang Li , Yuan Yao , Jialin Li , Xinhua Zhu","doi":"10.1016/j.asw.2025.100960","DOIUrl":"10.1016/j.asw.2025.100960","url":null,"abstract":"<div><div>The rapid advancement of Generative AI (Gen-AI), such as ChatGPT, presents both opportunities and challenges for teacher education. For pre-service teachers (PSTs), Gen-AI offers new tools to enhance the efficiency and quality of writing feedback. However, it also raises concerns, as many PSTs lack classroom experience, confidence in giving feedback, and knowledge of how to effectively integrate AI-generated content into instructional practice. To address these issues, this study adopted a pre-post experimental design to examine the effects of targeted training on PSTs’ provision of writing feedback, with a focus on feedback quality, self-efficacy, and their relationship in ChatGPT-supported contexts. Over a two-week training program with 30 PSTs, Wilcoxon signed-rank test results from the content analysis showed significant improvements in feedback quality and self-efficacy. Semi-structured interviews with eight participants identified cognitive changes and enhanced ChatGPT operational skills as key drivers of these improvements. We reaffirmed that mastery and vicarious experiences are crucial for enhancing teacher self-efficacy. Furthermore, a reciprocal relationship was observed between the quality and self-efficacy in providing ChatGPT-assisted feedback. This study contributes to the broader discourse on ChatGPT in education and offers specific strategies for effectively incorporating new technology into teacher training.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100960"},"PeriodicalIF":4.2,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144321635","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Challenges and opportunities of automated essay scoring for low-proficient L2 English writers 对低熟练程度的第二语言英语作家的自动作文评分的挑战和机遇

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-09-20 DOI: 10.1016/j.asw.2025.100982

Vanessa De Wilde , Orphée De Clercq

{"title":"Challenges and opportunities of automated essay scoring for low-proficient L2 English writers","authors":"Vanessa De Wilde , Orphée De Clercq","doi":"10.1016/j.asw.2025.100982","DOIUrl":"10.1016/j.asw.2025.100982","url":null,"abstract":"<div><div>Assessing students’ writing can be a challenging activity. To make writing assessment more feasible, researchers have investigated the possibilities of automated essay scoring (AES). Most studies investigating AES have focused on L1 writing or intermediate to advanced L2 writing. In this study we explored the possibilities of using AES with low proficiency L2 English writers. We used a dataset which comprised writing samples from 3166 young L2 English learners who were at the very start of L2 English instruction. All tasks received a score assigned by humans.</div><div>For automated scoring we experimented with two machine learning methods. First, a feature-based approach for which the dataset was linguistically preprocessed using natural language processing tools. The second approach employed deep learning by fine-tuning various large language models. Because we were particularly interested in the influence of spelling errors, we also created a corrected, spell-checked version of our dataset.</div><div>Models trained on the uncorrected samples yield the best results. Especially the deep learning approach leads to a satisfying performance with a quadratic weighted kappa above .70. The model which was fine-tuned on an underlying Dutch large language model was superior, which might be linked to the low L2 English proficiency of the young L1 Dutch writers in our sample.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100982"},"PeriodicalIF":5.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145105005","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Growth mindset and writing engagement: The roles of motivation regulation and engagement with teacher’s written corrective feedback 成长心态与写作投入：动机调节与参与对教师书面纠正反馈的作用

IF 5.5 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-08-20 DOI: 10.1016/j.asw.2025.100980

Mahdieh Darvari , S. Yahya Hejazi , Majid Sadoughi

{"title":"Growth mindset and writing engagement: The roles of motivation regulation and engagement with teacher’s written corrective feedback","authors":"Mahdieh Darvari , S. Yahya Hejazi , Majid Sadoughi","doi":"10.1016/j.asw.2025.100980","DOIUrl":"10.1016/j.asw.2025.100980","url":null,"abstract":"<div><div>Although engagement is widely recognized as a key and desirable outcome in Foreign/Second Language (L2) education and has garnered increasing research attention, its skill-specific manifestation, particularly in areas such as writing, remains underexplored. To address this gap, this study is among the first attempts to examine L2 writing engagement and its three potential predictors. To this end, motivated by Dweck’s (2017) mindsets theory and Lou and Noels’ (2019) language mindset meaning system, the present study investigated the link between growth L2 writing mindset, i.e., beliefs about improvability and changeability of L2 writing skills and abilities, and engagement by taking into account the parallel mediating roles of learners’ motivation regulation and engagement with teacher’s written corrective feedback (WCF). A total of 343 Iranian learners at intermediate proficiency level were selected through convenience sampling and responded to questionnaires. Structural Equation Modelling (SEM) analyses indicated that growth writing mindset could positively predict engagement, and this link was mediated by learners’ motivation regulation and engagement with teacher’s WCF. Implications and suggestions for fostering growth mindset and promoting learners’ feedback engagement as well as motivation regulation are presented.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100980"},"PeriodicalIF":5.5,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144867216","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Assessing L2 writing formality using syntactic complexity indices: A fuzzy evaluation approach 用句法复杂性指标评价二语写作的正式性：一种模糊评价方法

IF 4.2 1区文学

Assessing Writing Pub Date : 2025-10-01 Epub Date: 2025-07-12 DOI: 10.1016/j.asw.2025.100973

Zhiyun Huang , Guangyao Chen , Zhanhao Jiang

{"title":"Assessing L2 writing formality using syntactic complexity indices: A fuzzy evaluation approach","authors":"Zhiyun Huang , Guangyao Chen , Zhanhao Jiang","doi":"10.1016/j.asw.2025.100973","DOIUrl":"10.1016/j.asw.2025.100973","url":null,"abstract":"<div><div>Addressing the ambiguity in formality standards, this study introduces a cutting-edge Multi-dimensional Connection Cloud Model (MCCM) that leverages syntactic complexity indices to develop a fuzzy assessment model for formality in L2 writing. Employing Elastic Net Regression (ENR), the results revealed that four large-grained indices (mean length of sentence, mean length of T-unit, complex nominals per T-unit and complex nominals per clause), and one fine-grained index (average number of dependents per direct object) were significant in predicting the level of formality in L2 writing. To evaluate the model’s predictive power, 45 essays were used as a validation set. The MCCM model achieved a prediction accuracy of 91.1 % (41 out of 45 cases) in matching human ratings, with connection degrees effectively capturing classification uncertainty and boundary transitions. This pioneering framework effectively navigates the complexities and variable distributions of indicators, offering a more objective solution compared to conventional expert evaluations and introducing a novel methodological approach to assessing formality in academic writing.</div></div>","PeriodicalId":46865,"journal":{"name":"Assessing Writing","volume":"66 ","pages":"Article 100973"},"PeriodicalIF":4.2,"publicationDate":"2025-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144604347","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"文学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0