Evaluation of Control Modalities in Highly Automated Vehicles: A Virtual Reality Simulation-Based Study

Abstract

The integration of effective control modalities is paramount for enhancing user experience and safety in autonomous vehicles. This study investigates the performance and user experience of three control modalities i.e., voice, hand gesture, and physical button controls in high-level autonomous vehicles (Levels 4 and 5), under both distraction and non-distraction conditions. Our objective was to evaluate error rates, physiological responses, and subjective workload across these control modalities. The results revealed that distraction significantly increases error rates and perceived workload across all models. Voice control exhibited the lowest error rates without distraction but was most affected by it, whereas Hand Gesture control showed the highest error rates and workload in both scenarios. Physical Button control demonstrated moderate error rates and the least impact from distraction. Physiological data supported these findings, with significant increases in heart rate under distraction for all models, particularly in the voice control model. The NASA Task Load Index scores indicated higher workload under distraction, with hand gesture control being the most demanding. Our findings suggest that a combination of Physical Button and Voice control may offer the most effective solution, with recommendations for adaptive and multimodal interaction designs to mitigate distraction effects and enhance overall user satisfaction.