The conversational action test: Detecting the artificial sociality of artificial intelligence

Abstract

Drawing on the “Voigt-Kampff Empathy Test”—a science fiction version of Turing’s famous thought experiment—we propose the Conversational Action Test (CAT): a new approach to evaluating conversational artificial intelligence (AI) voice agents. We compare social actions in a range of telephone service encounters where one party is an artificial conversational agent to a range of similar human-human calls. The CAT demonstrates a novel paradigm that addresses long-standing theoretical and methodological problems for ostensible “tests” of conversational AI by (a) revealing the conceptual confusion of attempting to “detect” an AI in routine service interactions and (b) focusing, instead, on the situated interactional practices through which an AI “passes” for human. We discuss the implications of the CAT for the design and evaluation of conversational AI, and for the notion of “humanness” as a goal or benchmark for such systems. Data include publicly available human/AI service calls and comparable human-human calls in British and American English.