In-air Handwriting System Based on Improved YOLOv5 algorithm and Monocular Camera

Abstract

In-air handwriting is a new and more humanized human-computer interaction way. The existing in-air handwriting systems are mainly based on three-dimensional sensors, which are expensive, too large and not conducive for integration and application promotion. To solve this problem, this paper proposes a new in-air handwriting system using cheap and portable monocular camera which allows users writing freely in the air. Additionally we develop an end-to-end fingertip detection algorithm based on improved YOLOv5 algorithm to form in-air handwritten characters. Concretely, we first build a fingertip images dataset. After preprocessing and fingertip labeling, we use the dataset to train the improved YOLOv5 model, and then use the trained model to detect the coordinates of the fingertip in each video frame. After that, we connect the coordinates of the fingertip of each frame to form the character, and finally utilize the classifiers to recognize characters. The experimental results show that proposed in-air handwriting system allows user write freely in the air, and can obtain over 92 % in fingertip detection and character recognition.