iPand: Accurate Gesture Input with Smart Acoustic Sensing on Hand

Abstract

Finger gesture input is emerged as an increasingly popular means of human-computer interactions. In this demo, we propose iPand, an acoustic sensing system that enables finger gesture input on the skin, which is more convenient, user-friendly and always accessible. Unlike past works, which implement gesture input with dedicated devices, our system exploits passive acoustic sensing to identify the gestures, e.g. swipe left, swipe right, pinch and spread. The intuition underlying our system is that specific gesture emits unique friction sound, which can be captured by the microphone embedded in wearable devices. We then adopt convolutional neural network to recognize the gestures. We implement and evaluate iPand using COTS smartphones and smartwatches. Results from three daily scenarios (i.e., library, lab and cafe) of 10 volunteers show that iPand can achieve the recognition accuracy of 87%, 81% and 77% respectively.