Evaluating LLMs as proxies for humans in psycholinguistic ratings: A comparison of statistical knowledge

Abstract

Psycholinguistic research has traditionally relied on human ratings for stimulus norming, but whether large language models (LLMs) can reliably replace human ratings remains uncertain. This study compares human participants and three LLMs—one proprietary model (ChatGPT-4o) and two open-source models (LLaMA-3.3–70B and DeepSeek-V3)—with respect to their statistical knowledge of English binomials. For each binomial, we obtained ratings of frequency, dispersion, forward association strength, and backward association strength from 34 human participants and from 30 output samples per LLM. We examined rating-to-corpus consistency (consistency_{rating2corpus}), the sensitivity of statistical ratings to corpus data, and the influence of other psycholinguistic factors on ratings. All LLMs’ statistical knowledge broadly mirrored that of humans. Ratings from both groups were sensitive to corpus data but not fully consistent with it. Frequency showed the highest consistency_{rating2corpus}, whereas dispersion showed the lowest consistency_{rating2corpus} and the weakest sensitivity. LLM ratings were also influenced by word-level cues. Nonetheless, LLM ratings showed greater consistency_{rating2corpus}, heightened sensitivity, and stronger reliance on other psycholinguistic cues than human ratings. Overall, while LLMs’ performance generally aligned with that of humans, their internal statistical representations differed significantly from human cognition. The three LLMs also showed variation in their rating behavior. Thus, although multi-LLM ratings can aid pilot studies in psycholinguistics, they should not replace human ratings in formal experiments.