Fine-grained cross-modality consistency mining for Continuous Sign Language Recognition

Continuous Sign Language Recognition (CSLR) involves the detection of sequential glosses from visual sign inputs. While current CSLR methods perform well with high-frequency glosses – primarily functors such as conjunctions and pronouns – they struggle to accurately recognize low-frequency content words, which are essential for conveying meaningful information. This challenge arises from limitations in existing datasets and the Connectionist Temporal Classification (CTC) training procedure, leading to poor generalization to diverse linguistic structures. As a result, CSLR systems face limited applicability in real-world scenarios. In this work, we introduce the Fine-Grained Cross-modality Consistency (FGXM) loss, a novel approach designed to align visual and linguistic models. The FGXM loss encourages consistency between visual and language representations, improving the model’s ability to integrate visual context with linguistic understanding. We also propose the unweighted word error rate (uWER), an unbiased metric for CSLR performance. Unlike the conventional word error rate (WER), uWER provides a fairer evaluation by addressing the frequency imbalance between content words and functors, offering a more accurate measure of a model’s real-world effectiveness. We extensively evaluate our approach across multiple datasets and models, demonstrating significant improvements in both accuracy and data efficiency.