Compact convolution transformer with cross-feature aggregation for hand-gesture recognition

Abstract

Hand Gesture Recognition (HGR) plays a crucial role in intuitive human–computer interaction but continues to face challenges such as complex backgrounds, lighting variations, occlusions, and limited training data. To overcome these issues, we propose a Cross Feature Aggregation Compact Convolution Transformer (CrFe-CCT) that integrates multiscale convolutional features with a lightweight transformer architecture. In the proposed CrFe-CCT network, includes the multi-scale Cross Feature Aggregation (CrFe) and CCT modules. The CrFe module help to enhances feature robustness by fusing contextual information across scales, leading to improved recognition accuracy while maintaining low computational complexity. Also, CCT module help to preserve local spatial relationships. Unlike conventional transformers that rely on large-scale data, CrFe-CCT enables efficient learning on both small and large datasets. The experimental results demonstrate that the proposed CrFe-CCT outperforms existing state-of-the-art approaches on subject-dependent datasets, achieving accuracies of 91.95%(HGR-1), 97.70% (MUGD Set1), 95.50% (MUGD Set2), 99.06%(MUGD Set3), 99.82% (NUS-II), 99.90% (ASL-Finger Spelling (FS), and 96.80% (OUHands). On subject-independent datasets, the CrFe-CCT network achieves 40.43%(HGR-1), 85.11% (MUGD), 70.34 (NUS-II Dataset), 82.20% (ASL-Finger Spelling (FS)), respectively. Furthermore, it demonstrates superior efficiency with parameters, memory usage, FLOPs, inference time, and a throughput of images for real-world HGR applications.

The source code is available at https://github.com/satyantazi/CrFe-CCT.