A novel PRO-CAM based interactive display surface

Vision-based human-computer interaction (HCI) is a natural and human-centered way to make interaction between human and computer. Recently, with the miniaturization of projectors and the development of embedded systems, there has been an explosion of interest in systems which combine projection technology with computer vision. Associating a projector with a camera offers a cheap means to transform any surface into an interactive display surface. However, it is very hard to segment hand and recognize hand gesture due to self-occlusion, non-rigid tissue, even when the occlusion due to projection content can be avoided. In this paper, a novel approach is proposed to recognize finger stroke under the dynamic illumination circumstances. The approach is based on one projector and two heterogeneous cameras. First, an NIR camera is used to get the finger-tip and hand model, then, the hand model is used to get the interest points of the hand in the visible camera, then we can get the disparity of interest points from the two heterogeneous images. The disparity change is related to the depth change that can be used to determine when stroke event happens. Experiment results from a prototype system show that the approach can run in real-time without using special markers or gloves.