Subjectivity of novelty metrics based on idea decomposition

IF 16.4 1区化学 Q1 CHEMISTRY, MULTIDISCIPLINARY

Accounts of Chemical Research Pub Date : 2020-08-24 DOI:10.1080/21650349.2020.1811775

Lorenzo Fiorineschi, F. S. Frillici, F. Rotini

{"title":"Subjectivity of novelty metrics based on idea decomposition","authors":"Lorenzo Fiorineschi, F. S. Frillici, F. Rotini","doi":"10.1080/21650349.2020.1811775","DOIUrl":null,"url":null,"abstract":"ABSTRACT The novelty metric suggested by Shah and colleagues is one of the most widespread among the suggestions made by scholars, and it is based on the subjective identification of attributes and/or functions underpinning analyzed ideas. If not correctly managed, this subjectivity can lead to non-negligible ambiguity of assessments, which could potentially invalidate the research results. Several variants to this metric have been proposed in the last two decades, with some of them claiming to have improved the original metric. However, the related benefits and drawbacks are still unclear, especially in terms of subjectivity. The aim of this study is to estimate the potential misalignment between research teams that independently perform the assessment of the same set of ideas. To this purpose, the considered metrics have been applied to a set of 100 ideas by utilizing the assessment results from three independent evaluators. It was revealed that the obtained novelty scores can be extremely different owing to the plethora of different possible interpretations of the analyzed ideas. Accordingly, the results highlight that for the same set of ideas, very different novelty assessment rationales can be followed by the evaluators.","PeriodicalId":1,"journal":{"name":"Accounts of Chemical Research","volume":null,"pages":null},"PeriodicalIF":16.4000,"publicationDate":"2020-08-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1080/21650349.2020.1811775","citationCount":"9","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Accounts of Chemical Research","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1080/21650349.2020.1811775","RegionNum":1,"RegionCategory":"化学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"CHEMISTRY, MULTIDISCIPLINARY","Score":null,"Total":0}

引用次数: 9

Abstract

ABSTRACT The novelty metric suggested by Shah and colleagues is one of the most widespread among the suggestions made by scholars, and it is based on the subjective identification of attributes and/or functions underpinning analyzed ideas. If not correctly managed, this subjectivity can lead to non-negligible ambiguity of assessments, which could potentially invalidate the research results. Several variants to this metric have been proposed in the last two decades, with some of them claiming to have improved the original metric. However, the related benefits and drawbacks are still unclear, especially in terms of subjectivity. The aim of this study is to estimate the potential misalignment between research teams that independently perform the assessment of the same set of ideas. To this purpose, the considered metrics have been applied to a set of 100 ideas by utilizing the assessment results from three independent evaluators. It was revealed that the obtained novelty scores can be extremely different owing to the plethora of different possible interpretations of the analyzed ideas. Accordingly, the results highlight that for the same set of ideas, very different novelty assessment rationales can be followed by the evaluators.

查看原文本刊更多论文

基于概念分解的新颖性度量的主观性

摘要Shah及其同事提出的新颖性度量是学者们提出的建议中最普遍的一个，它基于对支撑分析思想的属性和/或函数的主观识别。如果管理不当，这种主观性可能会导致评估的不可忽视的模糊性，这可能会使研究结果无效。在过去的二十年里，已经提出了该指标的几种变体，其中一些变体声称改进了原始指标。然而，相关的好处和缺点仍然不清楚，尤其是在主观性方面。本研究的目的是估计独立评估同一套想法的研究团队之间的潜在偏差。为此，通过利用三位独立评估者的评估结果，将所考虑的指标应用于一组100个想法。研究表明，由于对所分析的想法有过多不同的可能解释，所获得的新颖性得分可能会极为不同。因此，研究结果强调，对于同一组想法，评估者可以遵循非常不同的新颖性评估理由。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Accounts of Chemical Research 化学-化学综合

CiteScore

31.40

自引率

1.10%

发文量

312

审稿时长

2 months

期刊介绍： Accounts of Chemical Research presents short, concise and critical articles offering easy-to-read overviews of basic research and applications in all areas of chemistry and biochemistry. These short reviews focus on research from the author’s own laboratory and are designed to teach the reader about a research project. In addition, Accounts of Chemical Research publishes commentaries that give an informed opinion on a current research problem. Special Issues online are devoted to a single topic of unusual activity and significance. Accounts of Chemical Research replaces the traditional article abstract with an article "Conspectus." These entries synopsize the research affording the reader a closer look at the content and significance of an article. Through this provision of a more detailed description of the article contents, the Conspectus enhances the article's discoverability by search engines and the exposure for the research.