How Well Do Simulated Population Samples with GPT-4 Align with Real Ones? The Case of the Eysenck Personality Questionnaire Revised-Abbreviated Personality Test.

Abstract

Background: Advances in artificial intelligence have enabled the simulation of human-like behaviors, raising the possibility of using large language models (LLMs) to generate synthetic population samples for research purposes, which may be particularly useful in health and social sciences. Methods: This paper explores the potential of LLMs to simulate population samples mirroring real ones, as well as the feasibility of using personality questionnaires to assess the personality of LLMs. To advance in that direction, 2 experiments were conducted with GPT-4o using the Eysenck Personality Questionnaire Revised-Abbreviated (EPQR-A) in 6 languages: Spanish, English, Slovak, Hebrew, Portuguese, and Turkish. Results: We find that GPT-4o exhibits distinct personality traits, which vary based on parameter settings and the language of the questionnaire. While the model shows promising trends in reflecting certain personality traits and differences across gender and academic fields, discrepancies between the synthetic populations' responses and those from real populations remain. Conclusions: These inconsistencies suggest that creating fully reliable synthetic population samples for questionnaire testing is still an open challenge. Further research is required to better align synthetic and real population behaviors.