Effects of interaction modalities and emotional states on user’s perceived empathy with an LLM-based embodied conversational agent

Abstract

The rise of multimodal interaction and virtual human technologies has diversified the forms of interaction between humans and conversational agents (CAs). Recent advances in Artificial Intelligence (AI) technologies, particularly Large Language Models (LLMs), have significantly enhanced the ability of CAs to empathize with humans. However, there remains a gap in understanding how the user experiences in human–agent interaction are impacted by advanced virtual humans, multimodal AI-generated techniques and user’s initial emotional states. Particularly, how the advanced techniques influence users’ perceived empathy and subsequent emotional states. Thus, we focus on investigating the effects of interaction modalities and users’ initial emotional states on users’ perceived empathy and subsequent emotional states. We created an Embodied Conversational Agent (ECA) MetaChatBot by integrating the advanced LLMs and multimodal AI-generated techniques (e.g. text, voice, gestures, and facial expressions) with MetaHuman. We conducted a within-subjects 3 × 3 experiment with 36 participants. The experimental results with subject and objective measures revealed that MetaChatBot has achieved higher levels of users’ perceived empathy and more intense emotional impacts compared to text-based interaction. Additionally, differences in initial emotional states (happy, neutral, sad) significantly influence users’ perceived affective empathy, with higher levels observed in the happy and sad states. These two initial emotional states also lead to participants maintaining or increasing the social distance between humans and ECAs. This study advances the theoretical understanding of empathy in human–agent interaction by offering empirical evidence specifically within the context of LLM-based ECA in VR. Our findings contribute valuable insights into enhancing users’ perceived empathy in the human-ECA interaction and offer practical suggestions for ECA designers.