Simulated self-assessment in large language models: A psychometric approach to AI self-efficacy

Abstract

Large language models (LLMs) are frequently perceived to have human-like comprehension capabilities. However, their proficiency in evaluating and quantifying these capacities remains uncertain. We conducted a controlled psychometric measurement study adapting the 10-item General Self-Efficacy Scale (GSES) to evaluate simulated self-assessment across 10 contemporary LLMs. Models completed the GSES under a no-task control condition and after three task-priming conditions: computational reasoning, social reasoning, and summarization. We examined between-model variation, task versus no-task differences, within-model stability, item-order robustness, internal consistency, and qualitative reasoning patterns. LLM-generated GSES responses were highly stable across repeated administrations, with nearly all model-task-item scores remaining identical across three runs. Internal consistency was high across task conditions, with Cronbach's alpha, and item-order effects were minimal, with intraclass correlation coefficients. Despite this stability, simulated self-efficacy differed significantly between models across all task and no-task conditions. Composite LLM GSES scores were lower and more variable than human norms (compared to the literature), and the latent response structure did not replicate human self-efficacy patterns. Qualitative analysis suggested that models differed in how they interpreted GSES items involving effort, coping, agency, and persistence. Some models rejected these constructs as inapplicable to artificial systems, while others translated them into task-oriented capability language. These findings indicate that LLMs can generate internally consistent psychometric self-assessments, but these outputs appear to reflect model-specific and context-sensitive communication behavior rather than human-like self-efficacy or metacognition. Psychometric prompting may help characterize how LLMs express capability, limitation, and agency under different prompting contexts, but generative responses should be interpreted cautiously to avoid anthropomorphic conclusions.