Modeling persistence behavior in serious games: A human-centered approach using in-game and text replays

Abstract

Serious Games (SGs) have gained attention as powerful educational tools because of their potential to provide reliable assessments and evaluate hard-to-measure constructs and competencies that are difficult to capture using traditional forms of assessment. Specifically, this study presents a human-centered approach to model and detect persistence — a key component of successful learning outcomes — in the context of SGs. With this purpose in mind, we developed a comprehensive rubric to characterize persistence behaviors in SGs. To design the rubric, we identified a set of persistence profiles and characteristics from previous literature and elaborated a general rubric for identifying persistence behaviors at the level of individual attempts. These characteristics were then mapped onto measurable features within Shadowspect, the SG used for data collection. Following this rubric, two annotators manually labeled 1,374 level attempts from 64 students using two visualization methods: in-game and text replays. With a comprehensive dataset of 2,748 labeled attempts, we trained and evaluated Machine Learning (ML) models for each type of replay to classify persistence behaviors across four categories: Persistence, Non-persistence, Unproductive persistence, and No behavior. Our results indicate that while text-based replays enable efficient annotation with promising performance, in-game replays may provide finer detail for certain complex behaviors, highlighting the strengths and limitations of each visualization method. This work contributes the use of SGs for assessment, illustrating a transparent and adaptable AI-driven approach that enhances reliability and user-centered insights, highlighting the complementary role of human input in optimizing AI-based models to achieve meaningful, user-centered assessments in education.