False Positives and Deceptive Errors in SQL Assessment: A Large-scale Analysis of Online Judge Systems

IF 3.2 3区工程技术 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

ACM Transactions on Computing Education Pub Date : 2024-03-28 DOI:10.1145/3654677

Jinshui Wang, Shuguang Chen, Zhengyi Tang, Pengchen Lin, Yupeng Wang

{"title":"False Positives and Deceptive Errors in SQL Assessment: A Large-scale Analysis of Online Judge Systems","authors":"Jinshui Wang, Shuguang Chen, Zhengyi Tang, Pengchen Lin, Yupeng Wang","doi":"10.1145/3654677","DOIUrl":null,"url":null,"abstract":"<p>Online Judge Systems (OJSs) play a crucial role in evaluating SQL programming skills. However, OJSs may not accurately evaluate students’ queries due to the error-detection capabilities of test sets are insufficient, resulting in false positives that can mislead students and hinder their learning. This study analyzes a large-scale OJS’s evaluation dataset and identifies more than 110,000 (1.94%) false positive queries. It also validates existing SQL error categorization and reveals a new type of logical error called deceptive error, which occurs when students construct queries that pass specific test cases but fail to solve the actual problem. This type of error has been overlooked in previous research and can provide new insights into how to improve OJSs by enhancing test cases and feedback. This study contributes to the understanding of assessment and evaluation practices and processes in programming education, particularly the contribution that OJSs make to student learning and to course, staff and institutional development. It also suggests error prevention and detection techniques that can improve the effectiveness and fairness of OJSs in programming education and competitions.</p>","PeriodicalId":48764,"journal":{"name":"ACM Transactions on Computing Education","volume":"42 1","pages":""},"PeriodicalIF":3.2000,"publicationDate":"2024-03-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"ACM Transactions on Computing Education","FirstCategoryId":"5","ListUrlMain":"https://doi.org/10.1145/3654677","RegionNum":3,"RegionCategory":"工程技术","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Online Judge Systems (OJSs) play a crucial role in evaluating SQL programming skills. However, OJSs may not accurately evaluate students’ queries due to the error-detection capabilities of test sets are insufficient, resulting in false positives that can mislead students and hinder their learning. This study analyzes a large-scale OJS’s evaluation dataset and identifies more than 110,000 (1.94%) false positive queries. It also validates existing SQL error categorization and reveals a new type of logical error called deceptive error, which occurs when students construct queries that pass specific test cases but fail to solve the actual problem. This type of error has been overlooked in previous research and can provide new insights into how to improve OJSs by enhancing test cases and feedback. This study contributes to the understanding of assessment and evaluation practices and processes in programming education, particularly the contribution that OJSs make to student learning and to course, staff and institutional development. It also suggests error prevention and detection techniques that can improve the effectiveness and fairness of OJSs in programming education and competitions.

查看原文本刊更多论文

SQL 评估中的假阳性和欺骗性错误：在线评判系统的大规模分析

在线评判系统（OJS）在评估 SQL 编程技能方面发挥着至关重要的作用。然而，由于测试集的错误检测能力不足，OJS 可能无法准确评估学生的查询，从而导致误报，误导学生，阻碍他们的学习。本研究分析了大规模 OJS 的评估数据集，识别出超过 110,000 次（1.94%）误报查询。它还验证了现有的 SQL 错误分类，并揭示了一种新的逻辑错误类型，即欺骗性错误。这种类型的错误在以往的研究中被忽视了，它可以为如何通过增强测试用例和反馈来改进开放式在线学习系统提供新的见解。本研究有助于了解编程教育中的评估和评价实践与流程，特别是开放式在线学习系统对学生学习以及课程、员工和机构发展的贡献。本研究还提出了错误预防和检测技术，这些技术可以提高程序设计教育和竞赛中开放式联合测试的有效性和公平性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

ACM Transactions on Computing Education EDUCATION, SCIENTIFIC DISCIPLINES-

CiteScore

6.50

自引率

16.70%

发文量

期刊介绍： ACM Transactions on Computing Education (TOCE) (formerly named JERIC, Journal on Educational Resources in Computing) covers diverse aspects of computing education: traditional computer science, computer engineering, information technology, and informatics; emerging aspects of computing; and applications of computing to other disciplines. The common characteristics shared by these papers are a scholarly approach to teaching and learning, a broad appeal to educational practitioners, and a clear connection to student learning.