Privacy-Preserving Continuous Sign Language Recognition

Abstract

Responsible continuous sign language recognition (CSLR) needs to consider the privacy concerns of hearing-impaired people, which are ignored by CSLR methods. And simple privacy-preserving methods, such as face blurring, replacement, and pruning low-frequency components, degrade recognition performance. In this study, we propose PrivSign, the first privacy-preserving framework for CSLR. PrivSign transforms the RGB video sequence into the frequency domain and prunes low- to partial high-frequency components. The generated high-frequency videos are invisible to the human eye and conceal biometric information effectively. To solve the dilemma between concealing biological information and maintaining recognition performance, PrivSign conducts the Semantic Rectification. Specifically, the RGB-Frequency pre-training between RGB and high-frequency video enables the CSLR model to learn comprehensive sign semantics. Furthermore, the fine-tuning process combined with learnable Gabor filters is performed on high-frequency videos to mitigate noise and emphasize the significance of sign-relevant semantics. Recognition performance and black and gloss box attack results demonstrate that PrivSign achieves a balance among concealing biological information, impeding recovery, and recognition performance.