Real-time 3D hand posture estimation based on 2D appearance retrieval using monocular camera

Abstract

This paper proposes a system for estimating arbitrary 3D human hand postures in real-time. It can accept not only pre-determined hand signs but also arbitrary postures and it works in a monocular camera environment. The estimation is based on a 2D image retrieval. More than 16,000 possible hand appearances are first generated from a given 3D shape model by rotating model joints and stored in an appearance database. Every appearance is tagged with its own joint angles which are used when the appearance was generated. By retrieving the appearance in the database well-matching to the input image contour, the joint angles of the input shape can be rapidly obtained. The search area is reduced by using an adjacency map in the database. To prevent tracking failures, a fixed number of the well-matching appearances are saved at every frame. After the multiple neighborhoods of the saved appearances are merged, the unified neighborhood is searched for the estimate efficiently by beam search. The posture estimates result from experimental examples are shown.