Holistic quantified-self for context-aware wearable augmented reality

Abstract

We propose Holistic Quantified-Self for wearable Augmented Reality (HQS-AR), a unified user model for users wearing AR glasses that captures multi-dimensional user states through sensors and interaction data. Despite the growing need for a personalized AR system, unified user modeling in wearable AR remains limited, often leading to suboptimal experiences. To address this gap, we identified key modeling dimensions and their corresponding factors: physical (posture, energy expenditure), cognitive-emotional (cognitive load, emotion), social (social engagement, interpersonal density), and behavioral (device type, usage log). Using motion, visual, audio, behavioral, and physiological datasets, we implemented machine learning models to infer user states and classify scenarios in office environments, achieving an average accuracy of 88% for state estimation and 99% for scenario classification in hold-out validation. Our user model is expected to serve as a state indicator for configuring wearable AR systems and a tool for modeling user routines, advancing personalized AR interfaces.