GACE: Gesture and Appearance Cutout Embedding for Gaming Applications

Abstract

This paper presents a lightweight game framework that provides real-time integration of human appearance and gesture-guided control within the game. It augments a new immersive experience since it allows game users to see their personal appearance interacting in real-time with other computer graphical characters in the game. With the goal to make the system easily realizable, we address the challenges in the whole pipeline of video processing, gesture recognition, and communication. To this end, we introduce the game framework, Gesture and Appearance Cutout Embedding (GACE), which runs the human appearance cutout algorithm and connects with game components by using memory mapped files. We also introduce the gesture-based support to enhance the immersion. Extensive experiments have shown that the proposed system runs reliably and comfortably in real-time with a commodity setting.