Usability Evaluation of an Adaptive Courseware Approach in the Natural Language-Based Intelligent Tutoring System-Tutomat

Background

Adaptive educational systems have gained increasing attention due to their ability to personalise educational content based on individual learner progress. Prior research highlights that intelligent tutoring systems (ITSs) and adaptive courseware models improve learning outcomes by dynamically adjusting instructional materials. However, despite advancements in adaptive learning environments, usability remains a critical factor influencing their effectiveness and adoption. Therefore, a need exists to evaluate the usability of adaptive tutoring systems to ensure they provide optimal user experience whilst maintaining high instructional effectiveness.

Objectives

This study examines the usability and learning experience of Tutomat, an adaptive courseware system designed for automated, real-time content adaptation. Specifically, it aims to examine usability based on user interactions and feedback, assess learning effectiveness and engagement through pre-test/post-test comparisons and user feedback, identify key usability challenges and propose improvements for enhancing user experience.

Methods

A mixed-methods usability evaluation was conducted with 14 participants, applying both quantitative and qualitative research designs. The quantitative study used pre-test/post-test comparisons and System Usability Scale (SUS) scores to measure knowledge acquisition and usability perceptions. The qualitative study involved scenario-based usability testing, where participants completed eight structured tasks. Data analysis included Wilcoxon Signed-Ranks tests for learning gains, descriptive statistics for usability scores and analysis for qualitative feedback.

Results

Findings indicate that Tutomat significantly improved student knowledge, as demonstrated by pre-test/post-test comparisons (W = 0, p < 0.05). The SUS score of 68.21 suggests acceptable usability, though strict answer validation and unclear navigation caused user frustration. Scenario-based testing identified usability challenges, particularly in feature discoverability and response flexibility, which impacted the user experience. Participants valued adaptive feedback and structured content but expressed concerns regarding interface design limitations.

Conclusion

This study highlights the importance of usability in adaptive educational systems, demonstrating that real-time adaptive courseware can enhance learning engagement when designed with user-centred principles. The findings provide practical design recommendations for improving usability in ITSs, emphasising enhanced response flexibility, improved feedback mechanisms and clearer navigation features. Future research should explore longitudinal usability assessments to evaluate long-term user engagement and learning retention in adaptive environments.