Identifying potentially disruptive research via a comparative power-based large model

Timely identification of potentially disruptive research is a significant research issue, since disruptive innovation in science transforms the existing paradigm and/or opens a new paradigm. This study proposes a comparative power-based large model that can promptly and accurately identify potentially disruptive research via comparative analysis of semantically-related papers. To this end, a self-constructed dataset was built by treating accumulated disruptive and consolidating citations as crowdsourced annotation data. We employed a range of machine learning models (MLs), deep learning models (DLs), and large language models (LLMs) to build classifiers. Our optimal model, Mistral-7B^+*, attains an impressive F1 score of 0.8210 and outperforms the best-performing ML and DL models by approximately 27.05 % and 14.03 %, respectively. Testing on 275 recently published biomedical papers further verifies its effectiveness. Additionally, we conduct comprehensive experiments to scrutinize the comparative power of the large model as well as the impact of the number and quality of comparative papers and distinct functional paragraphs within abstracts on identification performance. Our findings show that an appropriate number and quality of comparative papers can promote identification performance. Moreover, result-based paragraphs are the most important for identifying disruptive research, while method-based paragraphs are least important.