Jetson Nano-Based Two-Way Communication System with Filipino Sign Language Recognition Using LSTM Deep Learning Model for Able and Deaf-Mute Persons

Abstract

Communication is the foundation of what it is to be human. The majority of human communication is reliant on sounds. However, it is not the sole natural means of communication; other people employ alternative ways. One of which is the Deaf community’s language. Communication between the Deaf-mute community and hearing or able individuals is one of the various challenges the two parties encounter. In the Philippines, around 70% of the Filipino Deaf Community utilizes Filipino Sign Language (FSL) as their primary language, whereas some hearing persons may be illiterate in the Deaf community’s native language. With the given situation, this study created a twoway communication device using Jetson Nano, covering the proper translation of FSL to text and speech and the conversion of input speech into text. Ten (10) dynamic FSL gestures are considered in this study. The device used LSTM Deep Learning Model and MediaPipe to recognize the FSL gestures, then convert them into speech through Google Text-to-Speech (gTTS) API. The device also converts speech to text using Google Speech-to-Text (gSTT) API. Sixty (60) trials of a two-way conversation between a deaf-mute and a hearing person are performed. By conducting a Test of Proportion for a Two-Way Conversation, it has revealed that the prototype exceeded the standard value of 91.11%, which garnered an accuracy of 93.33%, rendering the device highly effective and reliable as a means of communication between a deaf-mute person and a fully able one.