Expert-level policy style measurement via knowledge distillation with large language model collaboration

Abstract

Policy style is a crucial concept in policy science that reflects persistent patterns in the policy process across different governance settings. Despite its importance, policy style measurement faces issues of complexity, subjectivity, data sparseness, and computational cost. To overcome these obstacles, we propose KOALA, a novel KnOwledge distillation framework based on large lAnguage modeL collAboration. It transforms the weak scoring abilities of LLMs into a pairwise ranking problem, employs a small set of expert-annotated samples for non-parametric learning, and utilizes knowledge distillation to transfer insights from LLMs to a smaller, more efficient model. The framework incorporates multiple LLM-based agents (Prompter, Ranker, and Analyst) collaborating to comprehend complex measurement standards and self-explain policy style definitions. We validate KOALA on 4,572 Chinese government work reports (1954–2019) from central, provincial, and municipal levels, with a focus on the imposition dimension of policy style. Extensive experiments demonstrate KOALA’s effectiveness in measuring the intensity of policy style, highlighting its superiority over state-of-the-art methods. While GPT-4 achieves only 66% accuracy in pairwise ranking of policy styles, KOALA, despite being based on GPT-3.5, achieves a remarkable 85% accuracy, highlighting significant performance improvement. This framework offers a transferable approach for quantifying complex social science concepts in textual data, bridging computational techniques with social science research.