Who listens to ChatGPT and when should they? A two-study examination of AI-assisted decision making

Abstract

As large language models (LLMs) become increasingly integrated into decision-making contexts, understanding when, why, and how individuals engage with AI-generated input is critical. Across two experimental studies, we examined psychological and contextual factors that influence AI consultation, application, and performance outcomes during a structured decision-making task. In Study 1, participants chose whether to seek input from ChatGPT and how to apply it; in Study 2, all participants received AI recommendations, but input quality was manipulated. Across both studies, applying AI suggestions—rather than merely consulting them—was the key determinant of performance. Trust in AI consistently predicted both input seeking and application, while higher self-perceived expertise reduced reliance on AI—even when participants lacked relevant knowledge. Notably, performance gains only occurred when AI output was high quality and applied effectively; in contrast, applying low-quality recommendations harmed performance. These findings underscore the need for AI systems that not only foster appropriate trust, but also support users in critically evaluating the input they receive.