Exploring Multiple-Display Interaction for Live XR Performance

Abstract

Although VR concerts offer a unique experience for audience to watch a performance, people still tend to participate in a live performance physically for the co-presence and shared experience are difficult to perceive in VR. To address this issue, we propose Actualities, a live XR performance system that integrates onsite and online concerts to create a seamless experience across multiple displays. Our system utilizes various sensors to detect signals from musical instruments and onsite audiences, digitalizing onsite performance elements into a virtual world. We project the visuals onto screens and live-stream the content for audiences to watch through various devices, and we also designed several interactive elements for the audience to interact with the public display. To evaluate our system, we conducted exploratory research to help us refine our system and improve the cross-reality experience.