Toward accurate psychological simulations: Investigating LLMs’ responses to personality and cultural variables

Abstract

Large language models (LLMs) have demonstrated complex language comprehension, making them potential tools for psychological research. However, challenges remain in assessing their psychological properties, particularly regarding prompt design, comparison with human data, and the ability to simulate real-world psychological differences across cultural groups. This study investigates how LLMs respond to personality assessments and cultural variables, addressing gaps in previous research.

In three studies, we explored the impact of model parameters and prompt variations on LLM responses to personality tests. Study 1 examined how temperature, model type, and prompt templates influenced LLM responses, revealing that while temperature settings had minimal impact, prompt variations led to significant differences. Study 2 compared LLMs' self-report personality scores with human data (N = 18,192–49,159), finding that LLMs scored higher in positive traits (e.g., extroversion) and lower in negative traits (e.g., psychopathy), reflecting their training biases. Study 3 tested LLMs’ ability to simulate cultural differences by assessing virtual subjects from China and the USA. While significant differences were observed between the groups, both demonstrated East Asian self-construal patterns, indicating limitations in simulating authentic cultural psychological differences.

These findings highlight the influence of prompt design on LLM responses, the divergence between LLM and human personality profiles, and the difficulty of simulating accurate cultural psychological differences. These results underscore the need for more refined methodologies in psychological simulations using LLMs and suggest that current models struggle to represent diverse human psychological traits accurately.