High variability in LLMs’ analogical reasoning

Abstract

arising from T. Webb et al. Nature Human Behaviour https://doi.org/10.1038/s41562-023-01659-w (2025)

In a recent study, Webb, Holyoak and Lu¹ (henceforth WHL) demonstrated that a large language model (GPT-3, text-davinci-003) could match or even exceed human performance across several analogical reasoning tasks. This result led to the compelling conclusion that LLMs such as GPT-3 possess an emergent ability to reason by analogy. However, the findings were based on a single, proprietary model for which the releasing company provided limited public details and progressively restricted access to the internal probability distributions. Furthermore, text-davinci-003 was deprecated on 4 January 2024, and is no longer available through the OpenAI API. This poses a challenge to replicability in two ways. First, the lack of open access to the model and its recent deprecation make it difficult—if not impossible—for other researchers to verify or build upon the findings. Second, relying on a single model leaves open the question of whether the results can be extended to LLMs as a broader class of objects of scientific investigation. Without testing a diverse range of models, it is unclear whether the observed behaviours are specific to GPT-3 or represent a general property of comparable contemporary LLMs. Replicating experimental results based on proprietary models with public alternatives is thus crucial to ensure that the findings can be reproduced in the future², generalized to new model instances, and, more generally, to adhere to transparency principles that are of paramount importance in scientific research³.