A Hybrid Approach to Hand Detection and Type Classification in Upper-Body Videos

Detection of hands in videos and their classification into left and right types are crucial in various human-computer interaction and data mining systems. A variety of effective deep learning methods have been proposed for this task, such as region-based convolutional neural networks (R-CNNs), however the large number of their proposal windows per frame deem them computationally intensive. For this purpose we propose a hybrid approach that is based on substituting the “selective search” R-CNN module by an image processing pipeline assuming visibility of the facial region, as for example in signing and cued speech videos. Our system comprises two main phases: preprocessing and classification. In the preprocessing stage we incorporate facial information, obtained by an AdaBoost face detector, into a skin-tone based segmentation scheme that drives Kalman filtering based hand tracking, generating very few candidate windows. During classification, the extracted proposal regions are fed to a CNN for hand detection and type classification. Evaluation of the proposed hybrid approach on four well-known datasets of gestures and signing demonstrates its superior accuracy and computational efficiency over the R-CNN and its variants.