Enhancing the imitation game: a trust-based model for distinguishing human and machine participants

Abstract

Since 1950, the imitation game has captured the interest of researchers investigating human‒machine differences. Designed to evaluate machine cognition through a game-based framework, its complexity demands refinement. The imitation game utilizes this game-based setup, but its inherent intricacy calls for further enhancements. The fundamental question of whether machines are capable of genuine thought has been a key issue in artificial intelligence (AI) studies. Recent developments challenge the ease of differentiation, as AI enables machines to display human-like characteristics. This paper seeks to address the shortcomings of the original Turing test and criticisms of the imitation game by introducing an integrated model. Although machines currently operate based on our instructions, they can learn from errors and produce novel responses through generative AI techniques, even though they do not experience emotions. In this study, a new trust-based model was introduced to improve the imitation game. This model integrated various factors to assess the reliability of participants’ responses, including grammatical accuracy, response time, thinking duration, response speed, creativity, and the use of human-like expressions. The goal was to calculate a trust factor that determines the likelihood of a participant being a human or machine. To evaluate the model’s performance, a comprehensive dataset was developed using a chat generative pretrained transformer (ChatGPT-3.5). Two other large language models (LLMs), the large language model meta AI (Llama 3) and the Claude LLM, were also taken into account. To simulate the experiment with human participants, human-generated text was also included. The simulation results revealed that GPT-3.5 Turbo, Llama 3, and Claude LLM performed differently in terms of grammatical accuracy, human-like phrasing, creativity, and trust factors. GPT-3.5 and Llama 3 had lower error rates but struggled with human-like phrases. Claude resulted in more grammatical errors but better creativity. The human participants consistently showed greater trust and human-like phrase usage. Probability assessments categorized machines with 71–78% accuracy, whereas humans were identified with only a 29–36% chance of being a machine.