Improving Interaction in HMD-Based Vehicle Simulators through Real Time Object Reconstruction

Bringing real objects into the virtual world has been shown to increase usability and presence in virtual reality applications. This paper presents a system to generate a real time virtual reconstruction of real world user interface elements for use in a head mounted display based driving simulator. Our system uses sensor fusion algorithms to combine data from depth and color cameras to generate an accurate, detailed, and fast rendering of the user's hands while using the simulator. We tested our system and show in our results that the inclusion of the participants real hands, the wheel, and the shifter in the virtual environment increases the immersion, presence, and usability of the simulation. Our system can also be used to bring other real objects into the virtual world, especially when accuracy, detail, and real time updates are desired.