Agreement Helps, Disagreement Matters: Scaling Framing Function Annotation With Large Language Models

Abstract

Framing research has long relied on human annotation, but many aspects of framing – especially framing functions such as defining problems, identifying causes, making moral evaluations, or proposing solutions are difficult to label consistently and at scale. We demonstrate that large language models can reliably identify these framing functions, achieving levels of agreement comparable to those of human annotators. We evaluate three state-of-the-art models (GPT-5, Claude 3.7, and DeepSeek) alongside three trained human coders on a new dataset of social media posts about electric-vehicles, a highly topical and contested issue that holds many interpretations. This setting provides a strong test of both agreement and meaningful disagreement in framing-function annotation. Across models and humans, we find not only substantial convergence but also patterns of valid disagreement – differences that reflect legitimate alternative interpretations rather than annotation error. Recognising and preserving these cases enhances construct validity and provides a more realistic portrayal of how framing functions operate in contested debates. Our results demonstrate that large language models can scale framing function analysis while capturing the variability inherent in human interpretation, enabling forms of communication research that were previously impractical with human-only coding.