Zero-shot Learning on Gesture Movement for Interactive Dashboard Control

Abstract

Human-computer interaction (HCI), is always the mainstream in computer technology that concentrates on the communication between humans and computer. Gesture-based HCI, which sounded so modern and zippy at the time, sounds retro now. Although the idea of gesture based HCI is nothing new, this topic is still in vogue. HCI studies continually emphasize the user experience especially when it is implemented in a real-world environment. As known that every individual acts differently and more uncontrollable environmental variables might affect the performance to detect and react to the gesture performed. Even though there are many solutions and datasets proposed in the market, not each of them perfectly fitted to our needs. Hence, to propose a more tailored made gesture detection for own use, the existing zeroshot learning model will be tested on the gesture dataset introduced in this work to fine tune to own needs. The result shows that our proposed I2Hub dataset has higher accuracy compared to EgoGesture dataset (~1.01), but the elapsed time takes longer due to the higher average number of videos in each gesture action.