Dynamic Hand Gesture Tracking and Recognition for Real-Time Immersive Virtual Object Manipulation

Immersive visualisation is increasingly being used for comprehensive and rapid analysis of objects in 3D and object dynamic behaviour in 4D. Challenges are therefore presented to provide natural user interaction to enable effortless virtual object manipulation. Presented in this paper is the development and evaluation of a human-computer interaction system based on natural hand gestures. By employing a hybrid inertial and ultrasonic tracking system to provide the absolute positions and orientations of the user’s head and hands as well as a pair of high degrees-of-freedoms data gloves to provide the relative positions and orientations of finger joints and tips in both hands, the proposed system is shown to be able to automatically track and recognise a number of simple hand gestures. The effectiveness and potential of the proposed system is demonstrated through the five basic object manipulation tasks involving selection, release, translation, rotation and scaling of a 3D virtual cube.