On the Use of Multi-Modal Sensing in Sign Language Classification

Abstract

In literature, sign language recognition (SLR) has been proposed using multi-channel data acquisition devices with various sensing modalities. When using wearable sensors, multimodality data acquisition has been shown to be particularly useful for improving the classification accuracies as compared to single modality data acquisition. In this work, a statistical analysis is presented to quantify the performance of different combinations of wearable sensors such as surface electromyogram (sEMG), accelerometers and gyroscopes in the classification of isolated signs. Twelve signs from the Indian sign language are considered such that the signs consist of static hand postures, as well as complex motion of forearm and simple wrist motions. Following four combinations of sensor modalities are compared for classification accuracies using statistical tests: 1) accelerometer and gyroscope 2) sEMG and accelerometer, 3) sEMG and gyroscope and finally, 4) sEMG, accelerometer and gyroscope. Results obtained on actual data indicate that the combination of all three modalities, namely sEMG, accelerometer and gyroscope yield the best classification accuracy of 88.25% as compared to the remaining sensor combinations. However, the statistical analysis of the classification accuracies using analysis of variance (ANOVA) indicates that the use of sEMG sensors is particularly useful in the classification of static hand postures. Moreover, the classification of signs involving dynamic motion of hands either with simple wrist motion or motion of hand along a complex trajectory is comparatively better with any sensing modality as compared to the classification of static hand postures.