Automatic sign language to text translation using MediaPipe and transformer architectures

Abstract

This study presents a transformer-based architecture for translating Sign Language to spoken language text using embeddings of body keypoints, with the mediation of glosses. To the best of our knowledge, this work is the first to successfully leverage body keypoints for Sign Language-to-text translation, achieving comparable performance to baseline models without reducing translation quality. Our approach introduces extensive augmentation techniques for body keypoints, and convolutional keypoint embeddings, and integrates Connectionist Temporal Classification Loss and position encoding for Sign2Gloss translation. For the Gloss2Text stage, we employ fine-tuning of BART, a state-of-the-art transformer model. Evaluation on the Phoenix14T dataset demonstrates that our integrated Sign2Gloss2Text model achieves competitive performance, with BLEU-4 scores that show marginal differences compared to baseline models using pixel embeddings. On the How2Sign dataset, which lacks gloss annotations, direct Sign2Text translation posed challenges, as reflected in lower BLEU-4 scores, highlighting the limitations of gloss-free approaches. This work addresses the narrow domain of the datasets and the unidirectional nature of the translation process while demonstrating the potential of body keypoints for Sign Language Translation. Future work will focus on enhancing the model’s ability to capture nuanced and complex contexts, thereby advancing accessibility and assistive technologies for bridging communication between individuals with hearing impairments and the hearing community.