Duration versus accuracy—what matters for computerised adaptive testing in schools?

IF 5.1 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Computer Assisted Learning Pub Date : 2024-10-06 DOI:10.1111/jcal.13074

Nikola Ebenbeck, Morten Bastian, Andreas Mühling, Markus Gebhardt

{"title":"Duration versus accuracy—what matters for computerised adaptive testing in schools?","authors":"Nikola Ebenbeck, Morten Bastian, Andreas Mühling, Markus Gebhardt","doi":"10.1111/jcal.13074","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Computerised adaptive tests (CATs) are tests that provide personalised, efficient and accurate measurement while reducing testing time, depending on the desired level of precision. Schools have different types of assessments that can benefit from a significant reduction in testing time to varying degrees, depending on the area of application, but for which the loss of measurement accuracy has a different impact. The implementation of CAT can take several approaches, each of which can potentially affect the resulting test length and accuracy.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>We compare the methods of estimation-based CAT and binary-search-based CAT to determine the extent to which they are suitable for school assessment in terms of their length and accuracy.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>This study uses simulations based on empirical data from a cohort of pupils with and without special needs (<i>n</i> = 400) to examine the effects of probabilistic estimation-based CAT and deterministic binary-search-based CAT on the length and accuracy of an adaptive reading test for pupils with different ability levels.</p>\n </section>\n \n <section>\n \n <h3> Results and Conclusions</h3>\n \n <p>Estimation-based CAT leads to a 40% test reduction with an average accuracy of <i>r</i> = 0.96, while binary-search-based CAT leads to a test reduction of up to 88% with an average accuracy of <i>r</i> = 0.83. Both methods demonstrate the applicability of CAT in educational environments. Practical advantages and disadvantages of both methods for learning environments are discussed, as well as which method is best suited for specific assessment needs.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"40 6","pages":"3443-3453"},"PeriodicalIF":5.1000,"publicationDate":"2024-10-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jcal.13074","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.13074","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Computerised adaptive tests (CATs) are tests that provide personalised, efficient and accurate measurement while reducing testing time, depending on the desired level of precision. Schools have different types of assessments that can benefit from a significant reduction in testing time to varying degrees, depending on the area of application, but for which the loss of measurement accuracy has a different impact. The implementation of CAT can take several approaches, each of which can potentially affect the resulting test length and accuracy.

Objectives

We compare the methods of estimation-based CAT and binary-search-based CAT to determine the extent to which they are suitable for school assessment in terms of their length and accuracy.

Methods

This study uses simulations based on empirical data from a cohort of pupils with and without special needs (n = 400) to examine the effects of probabilistic estimation-based CAT and deterministic binary-search-based CAT on the length and accuracy of an adaptive reading test for pupils with different ability levels.

Results and Conclusions

Estimation-based CAT leads to a 40% test reduction with an average accuracy of r = 0.96, while binary-search-based CAT leads to a test reduction of up to 88% with an average accuracy of r = 0.83. Both methods demonstrate the applicability of CAT in educational environments. Practical advantages and disadvantages of both methods for learning environments are discussed, as well as which method is best suited for specific assessment needs.

Abstract Image

查看原文本刊更多论文

持续时间与准确性--学校电脑适应性测试的关键是什么？

背景计算机化自适应测验（CAT）是一种能提供个性化、高效和精确测量的测验，同时还能根据所需的精确程度缩短测验时间。根据应用领域的不同，学校有不同类型的评估，这些评估可以在不同程度上受益于测试时间的大幅缩短，但测量精确度的损失会对这些评估产生不同的影响。实施计算机辅助测试可以采用多种方法，每种方法都可能影响测试时间和准确性。目的我们比较了基于估算的计算机辅助测验和基于二元搜索的计算机辅助测验两种方法，以确定它们在测验长度和准确性方面适合学校评估的程度。方法本研究以一组有特殊需要和无特殊需要的学生（n = 400）的经验数据为基础进行模拟，研究基于概率估计的计算机辅助阅读和基于确定性二元搜索的计算机辅助阅读对不同能力水平学生的适应性阅读测试的长度和准确性的影响。结果和结论基于估算的计算机辅助学习可使测试时间缩短 40%，平均准确率为 r = 0.96；基于二元搜索的计算机辅助学习可使测试时间缩短 88%，平均准确率为 r = 0.83。这两种方法都证明了 CAT 在教育环境中的适用性。本文讨论了这两种方法在学习环境中的实际优缺点，以及哪种方法最适合特定的评估需求。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Computer Assisted Learning EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.70

自引率

6.00%

发文量

116

期刊介绍： The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope