Recognizing hand gestures for controlling home appliances with mobile sensors

Abstract

Mobile and ambient sensors provide a scalable platform for the integration of computing devices and smart appliances for smart home. In which mobile devices, such as smart watches and smart-phone commonly embedded with actuators and sensors i.e., accelerometers and gyroscopes, have opened up chances for the user to easily control home appliances. This paper proposes an integrated method and system that utilize several deep models and mobile sensors for hand gestures applicable for smart homes. The system consists of three components of actual smart home configurations: (i) smart-watch worn on the user’s wrist for capturing gesture patterns (ii) a recognition application that runs on the smart mobile phone and sends correspond commands to the home automation platform; and (iii) home automation platform with connected smart devices instrumented with ambient sensors. In addition, we define a simple yet easy-to-learn hand-gesture vocabulary composing of 18 gestures to the user. With the F-score of over 75%, our experiment on our self-collected data-set consisting of 18 gestures from 20 subjects, demonstrates that the feasibility of the gesture recognition for controlling home appliances.