Robust Hand Gesture Input Using Computer Vision, Inertial Measurement Unit (IMU) and Flex Sensors

Capturing the hand gesture is useful in many virtual reality applications like video games and surgery training for medical students. In this project, we have designed and built a hand tracking glove that is able to track the pose of the hand and the motion of the five fingers. We have employed sensing data from three different kinds of sensors, which includes a camera, an inertial measurement unit (IMU) and flex sensors. The ArUco marker is attached to the back of the glove to obtain the pose information of the hand from the camera. The Kalman filter is applied to stabilize the pose acquired. An IMU is adopted to increase the sampling rate up to 100Hz. Our system uses a sensor fusion scheme. Even if the ArUco marker is occluded temporarily, the pose of the glove can still be obtained. We also make use of the flex sensor to track the finger motion. In our experiment, it is shown that the motion of the hand and fingers can be obtained correctly. A virtual hand model in the computer moves simultaneously with the human hand in the real space.