Exploring Educational Applicability of Virtual Touch System on Maker Space

Abstract

Interaction with gestures is more intuitive than traditional input with a keyboard and a mouse. It has gradually become the major technology for extended reality. However, for most users, gesture control is not familiar as other interactive devices, and most user interfaces of daily-use applications are currently not designed for gesture control. Thus, we propose a mid-air hand gesture control system, named vTouch to realize a touch device trained by deep learning with a convolutional neural network for image feature extraction, attention mechanisms for extracting time-series data features, and fully connected layer for classification. Besides, there is a customized driver in kernel mode which treats the vTouch system as a virtual multi-touch device. An experimental design is also proposed to evaluate the effectiveness of the training and the usability of the system.