A quasi-experimental analysis of capabilities and limitations of generative AI in academic content evaluation in social sciences

Abstract

The complexity of social sciences research and the limitations of traditional evaluation methods highlight the need to explore the capabilities and application potential of generative AI in academic evaluation. Previous research in fields such as biomedical and other natural sciences has demonstrated the potential of generative AI to estimate the quality of research articles. This study adopts a quasi-experimental approach, 100 volunteers produced 600 social sciences academic texts across 6 types of topics, which were evaluated by 8 mainstream generative AI models. Statistical and sentiment analysis was conducted to compare the evaluation results using zero-shot and few-shot prompting strategies. The results show that AI-generated total scores are unreliable (precision = 66.35 %), and the actual total scores differ moderately from the human benchmark (average Cohen's d = 0.425). Few-shot prompt exhibited weaker differentiation capabilities across dimensions (average correlation = 5.25), while zero-shot prompt performed better (e.g., correlation_{Clarity, Significance} = 0.13), particularly in writing quality (average standard deviation = 5.38). Significant score differences were observed across the eight models (all p < 0.001), indicating inconsistency among models. Additionally, AI-generated comments across dimensions were generally positive, with different models exhibiting strengths across various dimensions and tasks. This study provides empirical evidence for scholars, peer reviewers, and research evaluation professionals interested in integrating generative AI into social sciences’ evaluation workflows. Overall, generative AI shows potential for enhancing evaluation efficiency and reducing favoritism in the peer review of social sciences, especially in large-scale or preliminary evaluations. However, when evaluating the novelty and significance, its dependency on domain knowledge and the interpretability of the results still requires prudent consideration and refinement.