Examining the role of volumetric segmentation on movement training in mixed reality

Segmentation is a technique to partition datasets to aid in comprehension. For static visualizations of the human body, segmentation of data is typically focused on exposing internal layers, such as organs or muscle. However, for dynamic visualizations of body movements, segmentation can also involve the process of deconstructing complex temporal sequences of body movement into smaller arrangements. This process is particularly relevant to learning new movement. Such movement training exercises commonly involve a co-located instructor and learner in the same place, performing, imitating, and analyzing demonstrated body movements. Mixed Reality technologies have been proposed to aid in movement learning, affording participants real-time and visual feedback. However existing approaches commonly use abstract or avatar representations, and focus on the entire body movement, rather on a specific bodily movement relevant to a particular exercise.

In this paper, we propose a segmentation technique to semantically isolate volumetric recordings of body limb movements, thereby supporting the learning of new skills in movement training. The technique captures photo-realistic personal characteristics throughout a movement, applying spatial and semantic segmentation to individual body segments. Applying the segmentation technique to dynamic movements enables learners to focus on the trajectory of specific body parts during skills training. An ideation study was conducted to investigate the use of volumetric recordings and segmentation in the domain of movement training using a remote usability virtual reality method via video conference tool, called Surrogate-Aloud. Participants ideated scenarios using volumetric segmentation and considered segmentation to be a valuable tool to better understanding complex movements and to better identify individual body limb trajectories. The value of volumetric segmentation of the human body was recognized in a variety of additional usage domains, including physiotherapy, prostheses education, and group classes for movement training.