人类评分者的对立面:使用自动化(AES 系统)进行能力测试评估的心理测量应对转变

Mohammad Idhom, I. A. Buditjahjanto, Munoto, Trimono, P. Riyantoko
{"title":"人类评分者的对立面:使用自动化(AES 系统)进行能力测试评估的心理测量应对转变","authors":"Mohammad Idhom, I. A. Buditjahjanto, Munoto, Trimono, P. Riyantoko","doi":"10.46627/silet.v4i2.291","DOIUrl":null,"url":null,"abstract":"This research is part of proof tests to a combination of statistical processing methods, collecting assessment rubrics in vocational education by comparing two systems, automated essay scoring and human rater. It aims to analyze the final assessment score of essays in Akademi Komunitas Negeri (AKN) Pacitan (Pacitan’s State Community College) and Akademi Komunitas Negeri (AKN) Blitar (Blitar’s State Community College) in East Java, Indonesia. The provisional assumption is that the results show an antithesis to the assessment of human feedback with an automated system due to the conversion of scores between the rubric and the algorithm design. As the hypothesis, algorithm-based score conversion affects automated essay scoring and human rater methods, which led to antithesis feedback. The validity and reliability of the measurement maintain the scoring consistency between the two methods and the accuracy of the answers. The novelty of this article is comparing between AES system and Human Rater using statistical methods. The research shows that there is a similar result using the psychometrics approach, which indicates different metaphor expressions and language systems. Thus, the objective of this study is to provide assistance in the advancement of an information technology system that utilizes a scoring mechanism merging computer and human evaluations, employing a psychological approach known as psychometric leads.","PeriodicalId":34708,"journal":{"name":"Studies in Learning and Teaching","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Antithesis of Human Rater: Psychometric Responding to Shifts Competency Test Assessment Using Automation (AES System)\",\"authors\":\"Mohammad Idhom, I. A. Buditjahjanto, Munoto, Trimono, P. Riyantoko\",\"doi\":\"10.46627/silet.v4i2.291\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"This research is part of proof tests to a combination of statistical processing methods, collecting assessment rubrics in vocational education by comparing two systems, automated essay scoring and human rater. It aims to analyze the final assessment score of essays in Akademi Komunitas Negeri (AKN) Pacitan (Pacitan’s State Community College) and Akademi Komunitas Negeri (AKN) Blitar (Blitar’s State Community College) in East Java, Indonesia. The provisional assumption is that the results show an antithesis to the assessment of human feedback with an automated system due to the conversion of scores between the rubric and the algorithm design. As the hypothesis, algorithm-based score conversion affects automated essay scoring and human rater methods, which led to antithesis feedback. The validity and reliability of the measurement maintain the scoring consistency between the two methods and the accuracy of the answers. The novelty of this article is comparing between AES system and Human Rater using statistical methods. The research shows that there is a similar result using the psychometrics approach, which indicates different metaphor expressions and language systems. Thus, the objective of this study is to provide assistance in the advancement of an information technology system that utilizes a scoring mechanism merging computer and human evaluations, employing a psychological approach known as psychometric leads.\",\"PeriodicalId\":34708,\"journal\":{\"name\":\"Studies in Learning and Teaching\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-08-31\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Studies in Learning and Teaching\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.46627/silet.v4i2.291\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Studies in Learning and Teaching","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.46627/silet.v4i2.291","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

本研究是结合统计处理方法的证明测试的一部分,通过比较自动论文评分和人工评分两种系统,收集职业教育中的评估标准。研究旨在分析印度尼西亚东爪哇的帕西坦国立社区学院(Akademi Komunitas Negeri,AKN)和布利塔尔国立社区学院(Akademi Komunitas Negeri,AKN)的论文最终评估得分。暂定假设是,由于评分标准和算法设计之间的分数转换,结果显示了自动系统对人的反馈评估的反作用。正如假设的那样,基于算法的分数转换影响了自动作文评分和人工评分方法,从而导致了反义反馈。测量的有效性和可靠性保持了两种方法的评分一致性和答案的准确性。本文的新颖之处在于使用统计方法对 AES 系统和人工评分器进行比较。研究表明,使用心理测量方法得出的结果相似,这表明隐喻表达和语言系统不同。因此,本研究的目的是利用一种被称为心理测量线索的心理学方法,帮助开发一种利用计算机和人工评价相结合的评分机制的信息技术系统。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Antithesis of Human Rater: Psychometric Responding to Shifts Competency Test Assessment Using Automation (AES System)
This research is part of proof tests to a combination of statistical processing methods, collecting assessment rubrics in vocational education by comparing two systems, automated essay scoring and human rater. It aims to analyze the final assessment score of essays in Akademi Komunitas Negeri (AKN) Pacitan (Pacitan’s State Community College) and Akademi Komunitas Negeri (AKN) Blitar (Blitar’s State Community College) in East Java, Indonesia. The provisional assumption is that the results show an antithesis to the assessment of human feedback with an automated system due to the conversion of scores between the rubric and the algorithm design. As the hypothesis, algorithm-based score conversion affects automated essay scoring and human rater methods, which led to antithesis feedback. The validity and reliability of the measurement maintain the scoring consistency between the two methods and the accuracy of the answers. The novelty of this article is comparing between AES system and Human Rater using statistical methods. The research shows that there is a similar result using the psychometrics approach, which indicates different metaphor expressions and language systems. Thus, the objective of this study is to provide assistance in the advancement of an information technology system that utilizes a scoring mechanism merging computer and human evaluations, employing a psychological approach known as psychometric leads.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
10
审稿时长
4 weeks
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信