Developing A novel AI enabled extended reality system for real-time automatic facial expression recognition and system performance evaluation

Abstract

Facial Expression Recognition (FER) is vital for understanding human behavior but faces challenges from varying facial features due to different poses, lighting, and angles. Addressing the growing demand for real-time FER is critical. Extended Reality (XR) offers significant potential in training, education, healthcare, user experience, and relevant data collection. This study aims to develop an AI-enabled XR system for FER by combining a novel Depthwise Separable Convolutional Neural Network (DS-CNN) approach with XR technology. The FER2013 image dataset was used to train and build the proposed FER model. The model’s performance was validated using two separate image datasets, demonstrating that the proposed CNN model outperformed existing models on both. Subsequently, the CNN model was integrated with Microsoft HoloLens 2 XR technology to create a real-time, automatic FER system. System evaluation was conducted using System Usability Scale (SUS) and NASA-TLX measures, with results indicating that the proposed smart system is high usability and lower cognitive workload compared with FER using eyes. The AI-enabled XR system offers significant practical applications and potential across various domains, providing valuable managerial insights. The integration of CNN with XR technology represents a substantial advancement in real-time FER, offering improved accuracy and usability under diverse conditions.