A Robust Sign Language Recognition System with Multiple Wi-Fi Devices

Abstract

Sign language is important since it provides a way for us to the deaf culture and more opportunities to communicate with those who are deaf or hard of hearing. Since sign language chiefly uses body languages to convey meaning, Human Activity Recognition (HAR) techniques can be used to recognize them for some sign language translation applications. In this paper, we show for the first time that Wi-Fi signals can be used to recognize sign language. The key intuition is that different hand and arm motions introduce different multi-path distortions in Wi-Fi signals and generate different unique patterns in the time-series of Channel State Information (CSI). More specifically, we propose a Wi-Fi signal-based sign language recognition system called WiSign. Different from existing Wi-Fi signal-based human activity recognition systems, WiSign uses 3 Wi-Fi devices to improve the recognition performance. We implemented the WiSign using a TP-Link TL-WR1043ND Wi-Fi router and two Lenovo X100e laptops. The evaluation results show that our system can achieve a mean prediction accuracy of 93.8% and mean false positive of 1.55%.