Immersive Simulation and Training of Person-to-3D Character Dance in Real-Time

This paper presents a person-to-3D character, immersive, real-time dance training framework for leaders. Rather than immersive mimicking of pre-recorded dance animations as in previous work, the user is now trained to lead a 3D partner while dancing, initiating movements to which the 3D character responds to, in real-time. Realistic animations were applied to the rigged 3D human model of a dance partner using Inverse Kinematics (IK). The 3D dance partner is rigged based on custom skeletons using 3D geometrical bones and IK solvers. Motion capture data were applied to the rigged 3D model to create realistic dance animations. Initially, the application features a tutorial guide process. The 3D character is standing facing the user who selects dance training options using gestures such as thumb and waving hands, touches and pinch actions. The 3D character partner responds in real-time, producing body motion according to the users' dance leading actions following Salsa dance rules. The experience is evaluated as realistic dance. The software architecture could be utilized in a wide range of training systems in diverse domains, such as training in the manufacturing domain.