A Study on the Representativeness Heuristics Problem in Large Language Models

Abstract

Large language models (LLMs) exhibit remarkable proficiency in text generation. However, their logical reasoning capabilities require enhancement. Major strides have been achieved in reasoning techniques for LLM, such as Few-shot, Zero-shot, and Chain-of-Thought (CoT). Nevertheless, these techniques have shortcomings, particularly in addressing the representativeness heuristic (RH) phenomenon. RH is a cognitive bias that occurs when a person judges the probability of an event or the likelihood that an object belongs to a particular category based on how well it matches the prototype or stereotype of that category. In this study, we investigated the pervasive issue of RH errors in LLMs. This research surpasses the constraints of previous studies by analyzing various RH scenarios that they did not cover and by directly constructing and testing the corresponding datasets. Moreover, a novel prompt called zero-shot-RH is proposed to augment the reasoning ability of LLMs, mitigate RH errors, and thus bolster logical reasoning. This approach seeks to enable LLMs to comprehend the given information better and reduce the biases stemming from RH errors. The prompt zero-shot-RH achieved an average accuracy higher than zero-shot-CoT by 0.145 and 0.277 in the tasks of correct reasoning and correct reasonings by sex, respectively, without relying on RH. The outcomes of this research endeavor are a deeper understanding of RH errors in LLMs and novel strategies to mitigate these biases, thereby advancing the domain of logical reasoning within LLMs.