Making Vibration-based On-body Interaction Robust

Wearable devices like smartwatches and smart wristbands have gained substantial popularity in recent years. However, due to the limited size of the touch screens, smartwatches typically have a poor interactive experience for users. Recently, new technology has converted the human body into a virtual interface using finger activity induced vibrations. However, these solutions fail to meet expectations during real-world deployments, e.g., system performance significantly degrades due to human-based variations, such as hand shapes, tapping forces, and device positions. To mitigate these human-based variations, we collected a dataset of 114 users, built a deep-learning model, and designed a novel Siamese domain adversarial training algorithm. In this way, we implement a robust system which works at accuracy (97%) across different hand shapes, finger activity strengths, and smartwatch positions on the wrist. We have posted a demo video on YouTube (https://youtu.be/N5-ggvy2qfI).