A novel automated policy text evaluation framework integrating PMC into large language models

Automated policy text evaluation is a critical research topic in policy informatics. Previous methods rely predominantly on manual variable assignment, making them inadequate for large-scale policy evaluation, while their context-dependent indicators extracted from specific policies lack cross-domain applicability. To address this, we propose a novel policy text automated evaluation framework by redesigning three generalized first-level evaluation indicators that are applicable to policies in any domain, integrating Policy Modeling Consistency Model (PMC) into large language models (LLM), and constructing automated PMC-scoring models based on LLaMA-3-Chinese-8B and Qwen-2.5-7B respectively. Using Chinese S&T policies as examples, we construct the first Chinese policy evaluation dataset with 22,630 labeled policy samples and train four PMC indicator automated calculation models. Compared to the baselines, the model based on Qwen-2.5-7B achieves the best performance in the evaluation of policy character, with an F1-score of 80.41%. The model based on LLaMA-3-Chinese-8B achieves best performance in the evaluation of policy normativity and policy function, with F1-scores of 75.07% and 74.11% respectively. This enables the automated calculation of PMC indices and the generation of a multi-input-output table for comprehensive policy analysis. The application in biosafety policies and data governance policies validate the cross-domain applicability of the framework. As the first framework for automated PMC evaluation, our methodology provides an innovative approach for large-scale, cross-domain policy evaluation.