Can a machine talk the talk though not climb the rock? A Turing Test on rock climbing

Abstract

Large Language Models demonstrate considerable fluency in human discourse. Despite their potentially transformative impact, their limits and capabilities are yet to be discovered. To mitigate potential harm and harness their potential for the benefit of society, it is important to understand their capabilities in human–machine interaction. To address this challenge, we present results from a pilot study involving rock climbers and ChatGPT-4. In our Task-Specific Turing Test, expert group members ask any question they believe will distinguish between the machine and a fellow member. The paper employs a perspective which focuses on expert discourses of social groups and their linguistic competence in expressing and demonstrating their expertise. Results show that ChatGPT is successful in passing as a rock climber in several areas of discourse but (still) falls short in one area. Experiential knowledge – in particular, embodiment – proved a revealing distinction between human and machine. We conclude by emphasizing that the language skills displayed by LLMs ultimately stems from human-AI ensembles.