VRmonic: A VR Piano Playing Form Trainer

Abstract

Improper hand form during practice and performance can lead to long-term injuries in musicians of all levels. We present a demonstration of VRmonic, a novel immersive VR-based piano trainer for learning correct hand form for piano by overlaying the playing form of an expert oracle. VRmonic consists of a library of 48 scales comprising of the major, and natural, harmonic, and melodic minor scales for each of the 12 notes. We obtain correct hand form by using an automatic approach to detect hand keypoints from recordings of an expert pianist collected using an overhead RGB-D mounted camera. During training, the user can use VRmonic to load in a prior recording of themselves playing a scale and compare their hand form against the expert oracle. During playback, we display virtual oracle and user hand overlap via a color change, controlled by our form tolerance threshold slider that allows the user to determine how aggressively VRmonic should check incorrect form. VRmonic enables everyday users to practice piano playing with continuous feedback while mitigating long-term physical injury.