mmFinger: Talk to Smart Devices With Finger Tapping Gesture

Abstract

Contact-free finger gesture recognition unlocks plenty of applications in smart Human-Computer Interaction (HCI). However, existing solutions either require users to wear sensors on their fingers or use continuously monitored cameras, raising concerns regarding user comfort and privacy. In this paper, we propose mmFinger, an accurate and robust mmWave-based finger gesture recognition system that can extend the range of available custom commands. The core idea is that mmFinger leverages the finger tapping pattern as a basic gesture and encodes different number combinations of the basic gesture like Morse code. To enable reliable recognition across different locations and for various users, we carefully design a robust feature Dop-profile to effectively characterize finger movements. Furthermore, by leveraging the multi-views provided by multiple antennas of radar, we develop an adaptive weighted feature fusion network to enhance the system's robustness. Finally, we devise a novel sequence prediction network to enable the system to recognize new gestures without retraining. Comprehensive experiments demonstrate that mmFinger can achieve an average recognition accuracy of 92% for 36 predefined gestures and 88% for 5 new user-defined commands, and is robust against finger location and user diversity.