Est3D2Real-estimated 3D-to-real data embeddings for real time sign language recognizer

Human pose estimation predicts 3D skeletal joints from 2D video data. These estimated 3D joints are sensitive to video data anomalies, posing a threat to applications such as real-time sign language recognition. The challenge lies in the failure of the estimation model to output pose vectors during the signing process, which significantly impacts downstream classification tasks. To address this issue, we propose the development of a lightweight estimated 3D-to-real data embedding network (Est3D2Real). This network is designed to learn the relationships between the outputs of the pose estimation framework and a 3D motion capture system. Est3D2Real is a four-layer fully connected network, consisting of one input layer, two hidden layers, and one output layer. It employs the Mean Squared Error (MSE) loss function to minimize the distance between the two modalities. The trained Est3D2Real model ensures minimal joint loss in real-time downstream classification tasks. Validation is performed on a 100-gloss 3D sign language dataset, captured using both motion capture and MediaPipe frameworks. Subsequent downstream sign classifiers built on top of the trained Est3D2Real model have shown an approximate improvement of 28%. The code with small datasets is made available at https://github.com/pvvkishore/Est3D2Real_SL_MediaPipe_2_Motion_Capture.