Reliability, validity, and strictness of adult learner online self-rating based on teacher and achievement criteria for different individual characteristics and academic achievement

IF 2.6 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Studies in Educational Evaluation Pub Date : 2024-12-06 DOI:10.1016/j.stueduc.2024.101431

Chi-Cheng Chang, Ju-Shih Tseng

{"title":"Reliability, validity, and strictness of adult learner online self-rating based on teacher and achievement criteria for different individual characteristics and academic achievement","authors":"Chi-Cheng Chang, Ju-Shih Tseng","doi":"10.1016/j.stueduc.2024.101431","DOIUrl":null,"url":null,"abstract":"<div><div>Although many studies have shown that student self-assessment/rating is reliable or valid, many teachers still hesitate to implement student self-assessment/rating in their teaching. Therefore, whether or not student self-rating is reliable or valid has been a crucial concern. The study aimed to investigate the reliability, validity, and strictness of adult students’ online self-ratings by examining the consistency and difference between self-rating and teacher rating, and the consistency between self-rating and term project grade. There were 368 graduate students who used an online assessment platform to rate their own presentations. The results showed that adult students’ self-ratings had significant but low inter-rater consistency reliability and teacher-referenced validity, but inadequate teacher-referenced strictness, compared with teacher/expert criteria. Self-rating had significant but low achievement-referenced validity, compared with academic achievement criteria. Age, work experience, paper-pencil self-assessment experience, and online self-assessment experience affected the inter-rater consistency reliability and teacher-referenced validity of student self-rating. Individual characteristics and academic achievement had no effect on self-rating strictness. Implications and recommendations for academic theory and educational practice are proposed.</div></div>","PeriodicalId":47539,"journal":{"name":"Studies in Educational Evaluation","volume":"84 ","pages":"Article 101431"},"PeriodicalIF":2.6000,"publicationDate":"2024-12-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Studies in Educational Evaluation","FirstCategoryId":"95","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0191491X24001172","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Although many studies have shown that student self-assessment/rating is reliable or valid, many teachers still hesitate to implement student self-assessment/rating in their teaching. Therefore, whether or not student self-rating is reliable or valid has been a crucial concern. The study aimed to investigate the reliability, validity, and strictness of adult students’ online self-ratings by examining the consistency and difference between self-rating and teacher rating, and the consistency between self-rating and term project grade. There were 368 graduate students who used an online assessment platform to rate their own presentations. The results showed that adult students’ self-ratings had significant but low inter-rater consistency reliability and teacher-referenced validity, but inadequate teacher-referenced strictness, compared with teacher/expert criteria. Self-rating had significant but low achievement-referenced validity, compared with academic achievement criteria. Age, work experience, paper-pencil self-assessment experience, and online self-assessment experience affected the inter-rater consistency reliability and teacher-referenced validity of student self-rating. Individual characteristics and academic achievement had no effect on self-rating strictness. Implications and recommendations for academic theory and educational practice are proposed.

查看原文本刊更多论文

基于教师和成就标准的成人学习者在线自评的信度、效度和严格性

尽管许多研究表明学生自评是可靠或有效的，但许多教师仍然对在教学中实施学生自评犹豫不决。因此，学生自评是否可靠或有效一直是一个重要的问题。本研究旨在通过检验成人学生在线自评与教师自评的一致性和差异性，以及自评与学期课题成绩的一致性，来考察成人学生在线自评的信度、效度和严格性。共有368名研究生使用在线评估平台对自己的演讲进行评分。结果表明，与教师/专家标准相比，成人学生自评具有显著但较低的一致性信度和教师参考效度，但教师参考严格性不足。与学业成就标准相比，自评具有显著但较低的成就参照效度。年龄、工作经历、纸笔自评经历和在线自评经历影响学生自评的一致性信度和教师参考效度。个体特征和学业成绩对自评严格程度无影响。提出了对学术理论和教育实践的启示和建议。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Studies in Educational Evaluation Multiple-

CiteScore

6.90

自引率

6.50%

发文量

审稿时长

62 days

期刊介绍： Studies in Educational Evaluation publishes original reports of evaluation studies. Four types of articles are published by the journal: (a) Empirical evaluation studies representing evaluation practice in educational systems around the world; (b) Theoretical reflections and empirical studies related to issues involved in the evaluation of educational programs, educational institutions, educational personnel and student assessment; (c) Articles summarizing the state-of-the-art concerning specific topics in evaluation in general or in a particular country or group of countries; (d) Book reviews and brief abstracts of evaluation studies.