Design and implementation of an AI-enabled visual report tool as formative assessment to promote learning achievement and self-regulated learning: An experimental study

IF 6.7 1区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

British Journal of Educational Technology Pub Date : 2024-01-09 DOI:10.1111/bjet.13424

Xiaofang Liao, Xuedi Zhang, Zhifeng Wang, Heng Luo

{"title":"Design and implementation of an AI-enabled visual report tool as formative assessment to promote learning achievement and self-regulated learning: An experimental study","authors":"Xiaofang Liao, Xuedi Zhang, Zhifeng Wang, Heng Luo","doi":"10.1111/bjet.13424","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <p>Formative assessment is essential for improving teaching and learning, and AI and visualization techniques provide great potential for its design and delivery. Using NLP, cognitive diagnostic and visualization techniques designed to analyse and present students' monthly exam data, we developed an AI-enabled visual report tool comprising six modules and conducted an empirical study of its effectiveness in a high school biology classroom. A total of 125 students in a ninth-grade biology course were assigned to a treatment group (<i>n</i> = 63) receiving AI-enabled visual reports as the intervention and a control group (<i>n</i> = 62) receiving overall oral feedback from the teacher. We present the main statistical results of the within-subjects design and the between-subjects design respectively, to better capture the main findings. Repeated measures ANOVA revealed a significant interaction effect of intervention and time on learning achievement, and the paired-sample Wilcoxon test indicated that the treatment group had experienced increasing learning anxiety (Cohen's <i>d</i> = 0.203, <i>p</i> = 0.046) and self-efficacy (Cohen's <i>d</i> = 1.793, <i>p</i> = 0.000) over time. Moreover, we conducted a series of non-parametric tests to compare the effects of AI-enabled visual reports and teacher feedback, but found no significant differences except for an increased self-efficacy (Cohen's <i>d</i> = 0.312, <i>p</i> = 0.046). Additionally, we had the students in the treatment group rate their favourable modules in the AI-enabled visual report and provide evaluative feedback. The study results provide important insights into the design and implementation of effective formative assessment supported by artificial AI and visualization techniques.</p>\n </section>\n \n <section>\n \n <div>\n \n <div>\n \n <h3>Practitioner notes</h3>\n <p>What is already known about this topic\n\n </p><ul>\n \n <li>Formative assessment is essential for improving teaching and learning.</li>\n \n <li>Traditional formative assessment tools lack accurate data-oriented assessment and usability.</li>\n \n <li>AI and visualization techniques have great potential for formative assessment.</li>\n </ul>\n <p>What this paper adds\n\n </p><ul>\n \n <li>This study designs and implements an AI-enabled visual report tool that generates data-driven, user-friendly reports.</li>\n \n <li>The AI-enabled visual report can not only enhance students' learning achievement and self-regulated learning over time but also increase their test anxiety.</li>\n \n <li>The AI-enabled visual report has a comparable effect with teacher feedback but leads to increased self-efficacy.</li>\n </ul>\n <p>Implications for practice and/or policy\n\n </p><ul>\n \n <li>We recommend using the AI-enabled visual report in large-size classes for its overall positive effects on both learning achievement and self-regulated learning.</li>\n \n <li>We recommend using the AI-enabled visual report over teacher feedback for its capacity to enhance students' self-efficacy.</li>\n \n <li>We recommend prioritizing the modules of Performance Ranking, Personal Mastery and Knowledge Alert when designing the AI-enabled visual report.</li>\n </ul>\n </div>\n </div>\n </section>\n </div>","PeriodicalId":48315,"journal":{"name":"British Journal of Educational Technology","volume":"55 3","pages":"1253-1276"},"PeriodicalIF":6.7000,"publicationDate":"2024-01-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"British Journal of Educational Technology","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/bjet.13424","RegionNum":1,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Formative assessment is essential for improving teaching and learning, and AI and visualization techniques provide great potential for its design and delivery. Using NLP, cognitive diagnostic and visualization techniques designed to analyse and present students' monthly exam data, we developed an AI-enabled visual report tool comprising six modules and conducted an empirical study of its effectiveness in a high school biology classroom. A total of 125 students in a ninth-grade biology course were assigned to a treatment group (n = 63) receiving AI-enabled visual reports as the intervention and a control group (n = 62) receiving overall oral feedback from the teacher. We present the main statistical results of the within-subjects design and the between-subjects design respectively, to better capture the main findings. Repeated measures ANOVA revealed a significant interaction effect of intervention and time on learning achievement, and the paired-sample Wilcoxon test indicated that the treatment group had experienced increasing learning anxiety (Cohen's d = 0.203, p = 0.046) and self-efficacy (Cohen's d = 1.793, p = 0.000) over time. Moreover, we conducted a series of non-parametric tests to compare the effects of AI-enabled visual reports and teacher feedback, but found no significant differences except for an increased self-efficacy (Cohen's d = 0.312, p = 0.046). Additionally, we had the students in the treatment group rate their favourable modules in the AI-enabled visual report and provide evaluative feedback. The study results provide important insights into the design and implementation of effective formative assessment supported by artificial AI and visualization techniques.

Practitioner notes

What is already known about this topic

Formative assessment is essential for improving teaching and learning.
Traditional formative assessment tools lack accurate data-oriented assessment and usability.
AI and visualization techniques have great potential for formative assessment.

What this paper adds

This study designs and implements an AI-enabled visual report tool that generates data-driven, user-friendly reports.
The AI-enabled visual report can not only enhance students' learning achievement and self-regulated learning over time but also increase their test anxiety.
The AI-enabled visual report has a comparable effect with teacher feedback but leads to increased self-efficacy.

Implications for practice and/or policy

We recommend using the AI-enabled visual report in large-size classes for its overall positive effects on both learning achievement and self-regulated learning.
We recommend using the AI-enabled visual report over teacher feedback for its capacity to enhance students' self-efficacy.
We recommend prioritizing the modules of Performance Ranking, Personal Mastery and Knowledge Alert when designing the AI-enabled visual report.

查看原文本刊更多论文

设计和实施人工智能可视化报告工具，作为促进学习成绩和自我调节学习的形成性评价：一项实验研究

形成性评估对于改进教学至关重要，而人工智能和可视化技术为其设计和实施提供了巨大的潜力。我们利用 NLP、认知诊断和可视化技术来分析和展示学生的月考数据，开发了一个由六个模块组成的人工智能可视化报告工具，并在高中生物课堂上对其有效性进行了实证研究。九年级生物课程的 125 名学生被分配到治疗组（n = 63）和对照组（n = 62），治疗组接受人工智能可视化报告作为干预措施，对照组则接受教师的整体口头反馈。为了更好地反映主要研究结果，我们分别列出了主体内设计和主体间设计的主要统计结果。重复测量方差分析显示，干预和时间对学习成绩有显著的交互作用，配对样本 Wilcoxon 检验表明，随着时间的推移，治疗组的学习焦虑（Cohen's d = 0.203，p = 0.046）和自我效能感（Cohen's d = 1.793，p = 0.000）都在增加。此外，我们还进行了一系列非参数检验，以比较人工智能可视化报告和教师反馈的效果，但除了自我效能感有所提高（Cohen's d = 0.312，p = 0.046）外，没有发现其他显著差异。此外，我们还让治疗组的学生在人工智能可视化报告中对其有利的模块进行评分，并提供评价反馈。传统的形成性评估工具缺乏以数据为导向的准确评估和可用性，而人工智能和可视化技术在形成性评估方面具有巨大潜力。本研究设计并实施了一种人工智能可视化报告工具，该工具可生成数据驱动、用户友好的报告。随着时间的推移，人工智能可视化报告不仅能提高学生的学习成绩和自我调节学习能力，还能增加他们的考试焦虑。我们建议在设计人工智能可视化报告时，优先考虑成绩排名、个人掌握和知识预警模块。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

British Journal of Educational Technology EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

15.60

自引率

4.50%

发文量

111

期刊介绍： BJET is a primary source for academics and professionals in the fields of digital educational and training technology throughout the world. The Journal is published by Wiley on behalf of The British Educational Research Association (BERA). It publishes theoretical perspectives, methodological developments and high quality empirical research that demonstrate whether and how applications of instructional/educational technology systems, networks, tools and resources lead to improvements in formal and non-formal education at all levels, from early years through to higher, technical and vocational education, professional development and corporate training.