Statistical and Spatiotemporal Hand Gesture Features for Sign Language Recognition Using the Leap Motion Sensor

Abstract

Improvements in automatic sign language recognition (SLR) will lead to more enabling smart environments through digital technology to support good health and well-being. Many state-of-the-art approaches to SLR focus on the classification of static hand gestures or resource-intensive image processing. The contributions of this study are to explore optimal combinations of statistical features for an effective low-cost approach. Some signs are temporal, reflecting in many of the dynamic gestures present, which is not often considered within the state of the art. This work considers the problem of signed gesture recognition in terms of how dynamic gestures change during delivery and how different types of features affect the classification ability of a machine learning model. Eighteen common British Sign Language gestures recorded via a Leap Motion Controller sensor provide a complex classification problem, and two sets of statistical spatiotemporal attributes are extracted from 0.6-s time windows. The ANOVA F-scores and the p values show that many of the features are statistically significant. A total of 146 individual machine learning models are trained after dimensionality reduction, and the results show that the best feature selection approach is to combine statistical and spatiotemporal features, with the most significant 240 features of each type leading to a mean 86.75% K-fold accuracy. The results show that classification metrics always improve when combining the two types of features. Within a wider comparison, it was found that histogram gradient boosting could outperform the random forest with a mean accuracy of 88.089%; however, it had a much higher inference time.