Sketches vs skeletons: video annotation can capture what motion capture cannot

Abstract

Good posture is vital to successful musical performance and music teachers spend a considerable amount of effort on improving their students' posture. This paper presents a user study to evaluate a skeletal motion capture system (based on the Microsoft Kinect™) for supporting teachers as they give feedback to learners about their posture and movement whilst playing an instrument. The study identified a number of problems with skeletal motion capture that are likely to make it unsuitable for this type of feedback: glitches in the capture reduce trust in the system, particularly as the motion data is removed from other contextual cues that could help judge whether it is correct or not; automated feedback can fail to account for the diversity of playing styles required by learners of different physical proportions, and most importantly, the skeleton representation leaves out many cues that are required to detect posture problems in all but the most elementary beginners. The study also included a participatory design stage which resulted in a radically redesigned prototype, which replaced skeletal motion capture with an interface that allows teachers and learners to sketch on video with the support of computer vision tracking.