An integrated indicator for evaluating scientific papers: considering academic impact and novelty

IF 3.5 3区管理学 Q2 COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS

Scientometrics Pub Date : 2024-09-13 DOI:10.1007/s11192-024-05150-9

Zhaoping Yan, Kaiyu Fan

{"title":"An integrated indicator for evaluating scientific papers: considering academic impact and novelty","authors":"Zhaoping Yan, Kaiyu Fan","doi":"10.1007/s11192-024-05150-9","DOIUrl":null,"url":null,"abstract":"<p>The assessment of scientific papers has long been a challenging issue. Although numerous studies have proposed quantitative indicators for assessing scientific papers, these studies overlooked the citation characteristics and the novelty of scientific knowledge implied in the textual information of papers. Therefore, this paper constructs an integrated indicator to evaluate scientific papers from both citation and semantic perspectives. Firstly, we propose weighted citations to measure the academic impact of scientific papers, which takes time heterogeneity and citation sentiment factors into consideration. Secondly, we capture the novelty of scientific papers from a semantic perspective, utilizing FastText to represent papers as text embeddings and applying the local outlier factor to calculate it. To validate the performance of our approach, the bullwhip effect domain and the ACL Anthology corpus are used for case studies. The results demonstrate that our indicator can effectively identify outstanding papers, thus providing a more comprehensive evaluation method for evaluating academic research.</p>","PeriodicalId":21755,"journal":{"name":"Scientometrics","volume":"12 1","pages":""},"PeriodicalIF":3.5000,"publicationDate":"2024-09-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Scientometrics","FirstCategoryId":"91","ListUrlMain":"https://doi.org/10.1007/s11192-024-05150-9","RegionNum":3,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS","Score":null,"Total":0}

引用次数: 0

Abstract

The assessment of scientific papers has long been a challenging issue. Although numerous studies have proposed quantitative indicators for assessing scientific papers, these studies overlooked the citation characteristics and the novelty of scientific knowledge implied in the textual information of papers. Therefore, this paper constructs an integrated indicator to evaluate scientific papers from both citation and semantic perspectives. Firstly, we propose weighted citations to measure the academic impact of scientific papers, which takes time heterogeneity and citation sentiment factors into consideration. Secondly, we capture the novelty of scientific papers from a semantic perspective, utilizing FastText to represent papers as text embeddings and applying the local outlier factor to calculate it. To validate the performance of our approach, the bullwhip effect domain and the ACL Anthology corpus are used for case studies. The results demonstrate that our indicator can effectively identify outstanding papers, thus providing a more comprehensive evaluation method for evaluating academic research.

Abstract Image

查看原文本刊更多论文

评估科技论文的综合指标：考虑学术影响力和新颖性

长期以来，科学论文的评估一直是一个具有挑战性的问题。尽管已有大量研究提出了评估科技论文的量化指标，但这些研究忽视了论文文本信息中隐含的引文特征和科学知识的新颖性。因此，本文构建了一个综合指标，从引文和语义两个角度对科技论文进行评价。首先，我们提出了加权引文来衡量科技论文的学术影响力，其中考虑了时间异质性和引文情感因素。其次，我们从语义角度捕捉科技论文的新颖性，利用 FastText 将论文表示为文本嵌入，并应用局部离群因子进行计算。为了验证我们方法的性能，我们使用牛鞭效应域和 ACL 文选语料库进行了案例研究。结果表明，我们的指标能有效识别优秀论文，从而为学术研究评价提供了一种更全面的评价方法。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Scientometrics 管理科学-计算机：跨学科应用

CiteScore

7.20

自引率

17.90%

发文量

351

审稿时长

1.5 months

期刊介绍： Scientometrics aims at publishing original studies, short communications, preliminary reports, review papers, letters to the editor and book reviews on scientometrics. The topics covered are results of research concerned with the quantitative features and characteristics of science. Emphasis is placed on investigations in which the development and mechanism of science are studied by means of (statistical) mathematical methods. The Journal also provides the reader with important up-to-date information about international meetings and events in scientometrics and related fields. Appropriate bibliographic compilations are published as a separate section. Due to its fully interdisciplinary character, Scientometrics is indispensable to research workers and research administrators throughout the world. It provides valuable assistance to librarians and documentalists in central scientific agencies, ministries, research institutes and laboratories. Scientometrics includes the Journal of Research Communication Studies. Consequently its aims and scope cover that of the latter, namely, to bring the results of research investigations together in one place, in such a form that they will be of use not only to the investigators themselves but also to the entrepreneurs and research workers who form the object of these studies.