PaperEval: A universal, quantitative, and explainable paper evaluation method powered by a multi-agent system

The immediate and efficient evaluation of scientific papers is crucial for advancing scientific progress. However, traditional peer review faces numerous challenges, including reviewer bias, limited expertise, and an overwhelming volume of publications. Recent advancements in large language models (LLMs) suggest their potential as promising evaluators, capable of approximating human cognition and understanding both ordinary and scientific language. In this study, we propose a novel AI-empowered paper evaluation method, PaperEval (PE), which utilizes a multi-agent system powered by LLMs to design evaluation criteria, assess paper quality along different dimensions, and generate explainable scores. We also introduce two variants of PE, Multi-round PaperEval (MPE) and Self-correcting PaperEval (SPE), which produce comparable scores and iteratively refine the evaluation criteria, respectively. To test our methods, we conducted a comprehensive analysis of three curated datasets, encompassing about 66,000 target papers of varying quality across the fields of mathematics, physics, chemistry, and medicine. The results show that our methods can effectively discern between high- and low-quality papers based on scores derived in four dimensions: Question, Method, Result, and Conclusion. Moreover, the results highlight the evaluation’s stability over time, the impact of comparative papers, the advantages of the multi-round evaluation strategy, and the varying correlation between AI ratings and scientific impact across different disciplines. Our method can seamlessly integrate into the existing scientific evaluation system, offering valuable insights for the development of AI-driven scientific evaluation.