3D sphere virtual instrument with Kinect and MIDI

Abstract

In this paper, we present an application that dance and music using Microsoft Kinect sensor and use MIDI technology to make a visual instrument for somebody who interested in music. The program will draw a sphere in three dimensions space by OpenGL and a user can dance in the visual sphere. The user can trigger events by entering two hands or feet in a specific zone and events will generate music or chord. The sphere can be 360 degrees rotated and we can see the 3D skeleton on the screen. We also calibrated two Kinects to do calibration because when user cannot stand in front of Kinect or when user stand side face to Kinect, Kinect cannot detect user's body very well, user's body will block by a part of your arms or legs and Kinect's infrared ray can't measure your joints. The reason we need to do a calibration if we need a side view of skeleton. We calculate the radius of the sphere when the user's hand on the surface of the sphere, it will generate sound by MIDI events. We use the program to perform in concert, which combine all virtual instruments in real time and show the 3d sphere with the user's skeleton on the screen.