Between immersion and usability: A comparative study of 2D and mixed reality interfaces for remote music making

Abstract

Networked Music Performance (NMP) systems have traditionally focused on optimizing audio transmission for remote collaboration, often relying on basic 2D video feeds that lack the spatial awareness and visual cues essential for coordinated musical performance. While Mixed Reality (MR) technology offers promising enhancements for musical collaboration, its integration into NMP systems remains understudied—particularly regarding the balance between immersion and usability. This study addresses that gap by comparing traditional 2D video conferencing setups with MR environments featuring point cloud representations and spatial audio. Using a simulation-based approach with pre-recorded stimuli, we conducted an experimental study with professional electric guitar players (N = 16) to examine how both setups affect musical performance, movement, and sense of presence, combining quantitative motion analysis with qualitative feedback. Head movement analysis revealed significantly reduced motion in MR compared to 2D conditions (p $<$ 0.001), while detrended fluctuation analysis indicated more structured movement patterns in MR (p $<$ 0.01). Post-task evaluations showed that although MR enhanced immersion and presence (p $<$ 0.001), participants reported greater performance coherence and fewer technical interferences in the 2D condition (p $<$ 0.001). Spatial audio had minimal impact on user experience, with visual elements drawing most of the participants’ attention. These findings underscore a fundamental tension between immersion and physical expressiveness in MR-based music performance, suggesting that future systems should prioritize ergonomics alongside technological innovation. This research contributes to a deeper understanding of the potential and limitations of MR in musical collaboration within the emerging Musical Metaverse.