Propaganda by prompt: Tracing hidden linguistic strategies in large language models

Abstract

As large language models become increasingly integrated into news production, concerns have grown over their potential to generate polarizing propaganda. This study introduces a scalable and flexible framework for systematically tracing the rhetorical strategies LLMs use to produce propaganda-style content. We apply the framework across three versions of GPT (GPT-3.5-Turbo, GPT-4o, and GPT-4.1), generating over 340,000 articles on selected politically divisive topics in the American news landscape. Supported by highly consistent distinctions (AUROC above 98 %), our findings reveal that the persuasive strategies adopted by GPT are both coherent and evolving across model versions. All three models rely heavily on cognitive language to simulate deliberation and interpretive reasoning, combined with consistent use of moral framing. Each version layers this rhetorical core with distinct stylistic choices: GPT-3.5-Turbo emphasizes collective identity and narrative looseness; GPT-4o adopts reflective detachment through its use of impersonal pronouns and tentative language; and GPT-4.1 deploys lexical sophistication and definitive assertions to project authority. These differences reflect a rhetorical evolution driven by architectural refinements, training updates, and changes in safety guard behavior. A comparison with human-authored propaganda further shows that GPT is not simply reproducing prompt-induced rhetorical biases but appears to exhibit distinct generative tendencies beyond those present in the human-authored baselines. The framework developed here offers a practical reverse-engineering tool for researchers, policymakers, and developers to explain and audit the persuasive capabilities of LLMs. It contributes to broader efforts in AI transparency, content moderation, and the promotion of epistemic resilience in digital communication.