From Perception to Interaction with Virtual Characters

This course will introduce students, researchers and digital artists to the recent results in perceptual research on virtual characters. It covers how technical and artistic aspects that constitute the appearance of a virtual character influence human perception, and how to create a plausibility illusion in interactive scenarios with virtual characters. We will report results of studies that addressed the influence of low-level cues like facial proportions, shading or level of detail and higher-level cues such as behavior or artistic stylization. We will place emphasis on aspects that are encountered during character development, animation, interaction design and achieving consistency between the visuals and storytelling. We will close with the relationship between verbal and non-verbal interaction and introduce some concepts which are important for creating convincing character behavior in virtual reality. The insights that we present in this course will serve as an additional toolset to anticipate the effect of certain design decisions and to create more convincing characters, especially in the case where budgets or time are limited. 1. Course Description Virtual humans are finding a growing number of applications, such as in social media apps, Spaces by Facebook, Bitmoji and Genies, as well as computer games and human-computer interfaces. Their use today has also extended from the typical on-screen display applications to immersive and collaborative environments (VR/AR/MR). At the same time, we are also witnessing significant improvements in real-time performance, increased visual fidelity of characters and novel devices. The question of how these developments will be received from the user’s point of view, or which aspects of virtual characters influence the user more, has therefore never been so important. This course will provide an overview of existing perceptual studies related to the topic of virtual characters. To make the course easier to follow, we start with a brief overview of human perception and how perceptual studies are conducted in terms of methods and experiment design. With knowledge of the methods, we continue with artistic and technical aspects which influence the design of character appearance (lighting and shading, facial feature placement, stylization, etc.). Important questions on character design will be addressed such as – if I want my character to be highly appealing, should I render with realistic or stylized shading? What facial features make my character appear more trustworthy? Do dark shadows enhance the emotion my character is portraying? We then dive deeper into the movement of the characters, exploring which information is present in the motion cues and how motion can, in combination with character appearance, guide our perception and even be a foundation of biased perception (stereotypes). Some examples of questions that we will address are – if I want my character to appear extroverted, what movement or appearance is needed to achieve this? Can character appearance influence my moral decisions in a video game? We then start to make our way into the domain of virtual reality and how it can be used to study perception of virtual characters and explore how appearance of virtual characters could affect our empathy level towards them. We also discuss possible behavioral measures for studying perception in virtual reality (VR). In the last Section, we focus on the question – How should we design interactions with virtual characters that improve task performance and are more immersive? Plausibility illusion is an important element in VR – it makes the VR experience more immersive, engaging, and ensures that skills learnt in VR can be directly applied to real life experiences. Starting with a brief review on publications evaluating plausibility illusion we will focus on the context of virtual characters, social presence or co-presence. The theory of plausibility illusion implies that the experience of interaction with a virtual character should be as close as possible to face-to-face interaction with a real person. Human face-to-face interaction is highly multimodal: the verbal content of conversation is enhanced by other, non-verbal signals that carry a lot of information, for example, tone of voice, facial expressions, gestures, gaze and spatial behaviour. Interaction with a character involves a tight loop of sensing a person and responses from the character. This course will cover sensing technologies, types of response and methods to map between the two. We will also address the relationship between verbal and non-verbal interaction including different roles that people adopt in conversation: speaking, listening, and other forms of nonc © 2020 The Author(s) Eurographics Proceedings c © 2020 The Eurographics Association. https://diglib.eg.org https://www.eg.org DOI: 10.2312/egt.20201001 E. Zell, K. Zibrek, X. Pan, M. Gillies & R. McDonnell / From Perception to Interaction with Virtual Characters verbal interaction. All of these issues will be informed by both the psychology of social interaction and current VR technology. We will use two examples to illustrate the design process of virtual character interaction in VR: one on training for doctor-patient communication, another one on our recent project with a game company on creating AI-characters for the ‘Peaky Blinders’ VR game. The course provides the overview of the relevant studies in a way that makes it easy to identify answers to practical questions in production and character development. At the same time, we avoid giving definite answers to questions of character and interaction design and encourage further investigation by listing questions left unanswered to allow for critical evaluation of the presented research. Finally, participating in a perceptual experiment is a multi-modal experience, which cannot be reproduced only by descriptive reports of the experiment design. For this reason, we will select a few representative experiments and run a highly compact version of them during the course for illustration purposes. The stimuli will be shown on the projector wall and the participants will be able to rate the stimuli within a small time-frame using their smartphones. Experiments will primarily be selected to introduce a new topic. We are fully aware that the obtained results are not representative by any means, but we believe that such live surveys will improve the understanding of the study design, increase engagement of participants, and be a welcoming break during a 180 minute talk. Previous Version of this Tutorial At SIGGRAPH 2019, we gave a shorter version (90min) of this tutorial, which was well attended (approximately 100 participants). Participants, who were less familiar with character perception research, were especially positive about the applicability of the knowledge. Given the positive feedback, we extended the tutorial on the topic of interaction. Other Related Tutorials and Courses Courses of the last 10–15 years at SIGGRAPH, SIGGRAPH Asia or Eurographics covered topics such as experiment design [CW13], visual perception of simple 3D shapes [FS09], as well as perception in graphics with applications to display technologies and virtual environments [GCL∗06a, TOY∗07, MR08]. Other courses covered a mixture of low-level stimuli perception and application in graphics, where character perception was partly addressed as well [OHM∗04, MMG11]. Finally, there are courses that focused on perception of specific aspects of virtual characters; these include: (i) the expressiveness of body motion [VGS∗06, HOP09], (ii) crowds [BKA∗14,HLLO10,DMTPT09,TOY∗07] (iii) a multidisciplinary study of emotions covering aspects of philosophy, psychology and physiology [Ges12] and (iv) the creation of believable characters for dialogues [JKF∗11]. Our course is the first to cover perception of virtual humans in a single resource, and addresses much more recent work than previous courses. We feel that it will be accessible for non-experts and a starting point for further investigation on related topics. Audience This course is suited for students, who want to get an overview of recent developments of perceptual research on virtual characters and identify open topics. Furthermore, this course is particularly designed for researchers and artists who work on virtual characters but are less familiar with the perceptual research.