Handwriting Classification based on Hand Movement using ConvLSTM

Abstract

The recognition of handwritten text presents challenges due to the variability and complexity of human handwriting, making it difficult to capture subtle nuances through traditional methods. Hand gesture recognition has emerged as an alternative method for predicting handwritten text, using sensors such as Kinect, LeapMotion, gyroscopes, accelerometers, and electromyograms to extract geometric and spatial information. Continuous hand-gesture recognition using cameras is preferred due to its ease of use and low hardware costs. Researchers have proposed different methods for recognizing hand gestures, including fuzzy logic, deterministic finite automata, trajectory-based methods, and dynamic probability long short-term memory (DP-LSTM). However, the latest research has shown that using LSTM can result in spatial information being lost. Therefore, this work proposes an architecture that captures spatial information using Convolutional Neural Network (CNN) and LSTM as Conv-LSTM, achieving high recognition rates in hand gesture trajectories for letters a to e in English captured using MediaPipe. Our results show that our proposed model can achieve high accuracy in classification and attained 0.8438.