More than buttons on controllers: engaging social interactions in narrative VR games through social attitudes detection

Abstract

People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program agents or non-player characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We present a collaborative work between two game studios (Maze Theory and Dream Reality Interactive) and academia to develop an immersive machine learning (ML) pipeline for detecting social engagement. Here we introduce the motivation and the methodology of the immersive ML pipeline, then we cover the motivation for the industry-academia collaboration, how it progressed, the implications of joined work on the industry and reflective insights on the collaboration. Overall, we highlight the industry-academia collaborative work on an immersive ML pipeline for detecting social engagement. We demonstrate how creatives could use ML and VR to expand their ability to design more engaging commercial games.