Stroke Patterns Guided End-to-End In-Air Handwriting Recognition Using mmWave Radar

In the task of in-air handwriting recognition, millimeter-wave (mmWave) radar sensors offer advantages in low-power consumption, privacy protection, and robustness to environmental conditions. The traditional approach is to convert the radar echo signals into radar images by digital signal processing (DSP) algorithms and then perform trajectory tracking or recognition. In this article, we propose an innovative end-to-end recognition model that directly processes raw radar signals without designing specific DSP algorithms. In order to solve the problem of network training difficulties caused by high-dimensional raw radar signals, we introduce a multimodal feature alignment method based on variational analysis, which utilizes the common stroke pattern of handwritten trajectories to guide network training. Specifically, the method employs 2-D handwriting trajectory sequences to represent stroke patterns. Through the designed multimodal feature alignment algorithm, the raw signal features extracted by the end-to-end network gradually converge to the easily accessible 2-D handwriting trajectory features. Comparison experiments with traditional methods in complex handwriting recognition tasks demonstrate the superiority of the proposed method. Subsequent visualization analysis and ablation experiments further confirm the validity and interpretability of the model modules.