Exploring the Potential of Speech-based Virtual Assistants in Mixed Reality Applications for People with Cognitive Disabilities

Abstract

Mixed Reality (MR) has been receiving increasing interest in the rehabilitation of people with Cognitive Disabilities. The power of MR in the context of therapies is the possibility to maintain a physical and psychological relationship with the surrounding environment while experiencing customized multimedia content and tasks that are appropriate for the needs of these users. The purpose of this work is to explore the potential of MR applications integrated with interactive speech-based Virtual Assistants. We present HoloLearn, an application for Microsoft HoloLens, which aims to help people with Cognitive Disabilities to improve their autonomy and learn simple activities of their everyday life (e.g., setting the table). The paper discusses the design features of the Virtual Assistant created and the results of a controlled study. The participants involved were 15 subjects with Cognitive Disabilities who used HoloLearn in two experimental conditions - with and without a Virtual Assistant. The results, although preliminary, indicate that enriching a MR experience with the presence of a speech-based Virtual Assistant would improve the user performance in the execution of the MR activities.