Usability Evaluation of an Adaptive Courseware Approach in the Natural Language-Based Intelligent Tutoring System-Tutomat

IF 5.1 2区教育学 Q1 EDUCATION & EDUCATIONAL RESEARCH

Journal of Computer Assisted Learning Pub Date : 2025-06-09 DOI:10.1111/jcal.70071

Ani Grubišić, Ines Šarić-Grgić, Angelina Gašpar, Branko Žitko

{"title":"Usability Evaluation of an Adaptive Courseware Approach in the Natural Language-Based Intelligent Tutoring System-Tutomat","authors":"Ani Grubišić, Ines Šarić-Grgić, Angelina Gašpar, Branko Žitko","doi":"10.1111/jcal.70071","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Background</h3>\n \n <p>Adaptive educational systems have gained increasing attention due to their ability to personalise educational content based on individual learner progress. Prior research highlights that intelligent tutoring systems (ITSs) and adaptive courseware models improve learning outcomes by dynamically adjusting instructional materials. However, despite advancements in adaptive learning environments, usability remains a critical factor influencing their effectiveness and adoption. Therefore, a need exists to evaluate the usability of adaptive tutoring systems to ensure they provide optimal user experience whilst maintaining high instructional effectiveness.</p>\n </section>\n \n <section>\n \n <h3> Objectives</h3>\n \n <p>This study examines the usability and learning experience of Tutomat, an adaptive courseware system designed for automated, real-time content adaptation. Specifically, it aims to examine usability based on user interactions and feedback, assess learning effectiveness and engagement through pre-test/post-test comparisons and user feedback, identify key usability challenges and propose improvements for enhancing user experience.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A mixed-methods usability evaluation was conducted with 14 participants, applying both quantitative and qualitative research designs. The quantitative study used pre-test/post-test comparisons and System Usability Scale (SUS) scores to measure knowledge acquisition and usability perceptions. The qualitative study involved scenario-based usability testing, where participants completed eight structured tasks. Data analysis included Wilcoxon Signed-Ranks tests for learning gains, descriptive statistics for usability scores and analysis for qualitative feedback.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Findings indicate that Tutomat significantly improved student knowledge, as demonstrated by pre-test/post-test comparisons (<i>W</i> = 0, <i>p</i> < 0.05). The SUS score of 68.21 suggests acceptable usability, though strict answer validation and unclear navigation caused user frustration. Scenario-based testing identified usability challenges, particularly in feature discoverability and response flexibility, which impacted the user experience. Participants valued adaptive feedback and structured content but expressed concerns regarding interface design limitations.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>This study highlights the importance of usability in adaptive educational systems, demonstrating that real-time adaptive courseware can enhance learning engagement when designed with user-centred principles. The findings provide practical design recommendations for improving usability in ITSs, emphasising enhanced response flexibility, improved feedback mechanisms and clearer navigation features. Future research should explore longitudinal usability assessments to evaluate long-term user engagement and learning retention in adaptive environments.</p>\n </section>\n </div>","PeriodicalId":48071,"journal":{"name":"Journal of Computer Assisted Learning","volume":"41 4","pages":""},"PeriodicalIF":5.1000,"publicationDate":"2025-06-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jcal.70071","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Computer Assisted Learning","FirstCategoryId":"95","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jcal.70071","RegionNum":2,"RegionCategory":"教育学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}

引用次数: 0

Abstract

Background

Adaptive educational systems have gained increasing attention due to their ability to personalise educational content based on individual learner progress. Prior research highlights that intelligent tutoring systems (ITSs) and adaptive courseware models improve learning outcomes by dynamically adjusting instructional materials. However, despite advancements in adaptive learning environments, usability remains a critical factor influencing their effectiveness and adoption. Therefore, a need exists to evaluate the usability of adaptive tutoring systems to ensure they provide optimal user experience whilst maintaining high instructional effectiveness.

Objectives

This study examines the usability and learning experience of Tutomat, an adaptive courseware system designed for automated, real-time content adaptation. Specifically, it aims to examine usability based on user interactions and feedback, assess learning effectiveness and engagement through pre-test/post-test comparisons and user feedback, identify key usability challenges and propose improvements for enhancing user experience.

Methods

A mixed-methods usability evaluation was conducted with 14 participants, applying both quantitative and qualitative research designs. The quantitative study used pre-test/post-test comparisons and System Usability Scale (SUS) scores to measure knowledge acquisition and usability perceptions. The qualitative study involved scenario-based usability testing, where participants completed eight structured tasks. Data analysis included Wilcoxon Signed-Ranks tests for learning gains, descriptive statistics for usability scores and analysis for qualitative feedback.

Results

Findings indicate that Tutomat significantly improved student knowledge, as demonstrated by pre-test/post-test comparisons (W = 0, p < 0.05). The SUS score of 68.21 suggests acceptable usability, though strict answer validation and unclear navigation caused user frustration. Scenario-based testing identified usability challenges, particularly in feature discoverability and response flexibility, which impacted the user experience. Participants valued adaptive feedback and structured content but expressed concerns regarding interface design limitations.

Conclusion

This study highlights the importance of usability in adaptive educational systems, demonstrating that real-time adaptive courseware can enhance learning engagement when designed with user-centred principles. The findings provide practical design recommendations for improving usability in ITSs, emphasising enhanced response flexibility, improved feedback mechanisms and clearer navigation features. Future research should explore longitudinal usability assessments to evaluate long-term user engagement and learning retention in adaptive environments.

查看原文本刊更多论文

基于自然语言的智能辅导系统中自适应课件方法的可用性评价

适应性教育系统由于能够根据学习者的个人进步来个性化教育内容而受到越来越多的关注。先前的研究强调，智能辅导系统（ITSs）和自适应课件模型通过动态调整教材来提高学习效果。然而，尽管自适应学习环境取得了进步，可用性仍然是影响其有效性和采用的关键因素。因此，有必要评估适应性辅导系统的可用性，以确保它们在保持高教学效率的同时提供最佳的用户体验。本研究考察了Tutomat的可用性和学习体验，Tutomat是一个为自动化、实时内容适应而设计的自适应课件系统。具体而言，它旨在基于用户交互和反馈来检查可用性，通过测试前/测试后比较和用户反馈来评估学习有效性和参与度，确定关键的可用性挑战，并提出增强用户体验的改进建议。方法采用定量和定性研究相结合的方法，对14名被试进行可用性评价。定量研究使用测试前/测试后比较和系统可用性量表（SUS）分数来衡量知识获取和可用性感知。定性研究涉及基于场景的可用性测试，参与者完成八个结构化任务。数据分析包括学习收益的Wilcoxon Signed-Ranks测试，可用性分数的描述性统计和定性反馈的分析。结果经测试前/测试后比较，Tutomat显著提高了学生的知识水平（W = 0, p < 0.05）。SUS得分68.21表明可用性尚可，但严格的答案验证和不清晰的导航会导致用户受挫。基于场景的测试确定了可用性挑战，特别是在特性可发现性和响应灵活性方面，这影响了用户体验。参与者重视自适应反馈和结构化内容，但表达了对界面设计局限性的担忧。本研究强调了适应性教育系统中可用性的重要性，表明实时适应性课件在以用户为中心的原则下设计可以提高学习参与度。研究结果为改进信息传输系统的可用性提供了实用的设计建议，强调增强响应灵活性、改进反馈机制和更清晰的导航功能。未来的研究应该探索纵向可用性评估，以评估长期用户参与和适应性环境中的学习保留。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Computer Assisted Learning EDUCATION & EDUCATIONAL RESEARCH-

CiteScore

9.70

自引率

6.00%

发文量

116

期刊介绍： The Journal of Computer Assisted Learning is an international peer-reviewed journal which covers the whole range of uses of information and communication technology to support learning and knowledge exchange. It aims to provide a medium for communication among researchers as well as a channel linking researchers, practitioners, and policy makers. JCAL is also a rich source of material for master and PhD students in areas such as educational psychology, the learning sciences, instructional technology, instructional design, collaborative learning, intelligent learning systems, learning analytics, open, distance and networked learning, and educational evaluation and assessment. This is the case for formal (e.g., schools), non-formal (e.g., workplace learning) and informal learning (e.g., museums and libraries) situations and environments. Volumes often include one Special Issue which these provides readers with a broad and in-depth perspective on a specific topic. First published in 1985, JCAL continues to have the aim of making the outcomes of contemporary research and experience accessible. During this period there have been major technological advances offering new opportunities and approaches in the use of a wide range of technologies to support learning and knowledge transfer more generally. There is currently much emphasis on the use of network functionality and the challenges its appropriate uses pose to teachers/tutors working with students locally and at a distance. JCAL welcomes: -Empirical reports, single studies or programmatic series of studies on the use of computers and information technologies in learning and assessment -Critical and original meta-reviews of literature on the use of computers for learning -Empirical studies on the design and development of innovative technology-based systems for learning -Conceptual articles on issues relating to the Aims and Scope