From Audio to Animated Signs

Abstract

Nearly 5% of the world population is hearing-impaired and uses a sign language as their primary mode of communication. Unfortunately, sign languages are not commonly understood, and this poses a great communication barrier between the hearing-impaired and the rest. This paper builds, based on prior wok, a novel approach that breaks the barrier one-way with accessibility and accuracy as the key objectives. It achieves accessibility using mobile devices for user-facing interactions and accuracy using a transformer model. The system uses a four-stage pipeline to enable one-way communication between hearing-gifted and hearing-impaired: audio capture, audio-to-text conversion, text-to-gloss transliteration, and gloss animation. The user-facing first and last stages of the pipeline are implemented on mobile devices to ensure wide accessibility, while the computationally intensive middle stages are implemented on cloud servers. Empirical evaluations show that the approach has a high accuracy.