How do individuals interact with an AI advisor in strategic reasoning? An experimental study in beauty contest

This paper experimentally investigates how individuals use generative AI to learn and respond in a strategic reasoning contest. An advisor based on level k theory and implemented using ChatGPT is introduced in a four-stage beauty contest experiment. The experiment is designed to explore how AI advisors influence the depth of human reasoning by shaping beliefs, learning, and sophisticated backward induction. Extended cognitive hierarchy models (Camerer et al., 2004) are applied to identify heterogeneous level distribution and more sophisticated thinking. Additionally, the interactions between participants with and without AI advisors are examined. Two key results emerge. First, individuals overestimate AI capabilities when competing against AI-guided participants, which motivates them to employ higher levels of thinking. This observed higher-level behaviour is driven by more sophisticated backward reasoning. Second, improved reasoning under AI guidance shows heterogeneous effects across Cognitive Reflection Test scores, suggesting that AI's impact depends on participants' pre-existing cognitive abilities. Overall, this early research provides insights into the interaction between generative AI and human cognition and reasoning.