AF-Mix: A gaze-aware learning system with attention feedback in mixed reality

Abstract

Mixed Reality (MR) has demonstrated its potential in various learning contexts. MR-based learning environments empower users to actively explore learning content visualized in multiple formats, such as 3D models, videos, and images. Nonetheless, the sophisticated visualizations in MR learning environments may result in potential visual overload, posing a challenge for users in efficiently allocating their attention. In this paper, we present AF-Mix, a learning support system that leverages eye tracking sensors in Microsoft HoloLens 2 to offer attention feedback for learners. Aiming to design AF-Mix, we conducted a participatory design study and integrated the attention feedback into our system, following users’ needs and suggestions. Furthermore, we evaluated AF-Mix in an evaluation study (n $=$ 22) following a quantitative analysis of users’ visual behavior, as well as a qualitative analysis of interview transcripts. Our findings show that providing feedback to support the learning process can be achieved effectively with eye tracking. In specific, attention feedback assists learners in retrieving previously missed information and encourages learners to reallocate their attention in the review process. Moreover, providing personalized feedback based on previous attention allocation is more effective in supporting users than a self-review approach without gaze-aware assistance in MR. Such feedback facilitates users in managing their limited attentional resources better and supports the reflection of their learning journey more effectively.