Research Evaluation最新文献_第9页

The production of scientific and societal value in research evaluation: a review of societal impact assessment methods 研究评估中科学与社会价值的产生:社会影响评估方法综述

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-04-11 DOI: 10.1093/RESEVAL/RVAB002

J. Smit, L. Hessels

{"title":"The production of scientific and societal value in research evaluation: a review of societal impact assessment methods","authors":"J. Smit, L. Hessels","doi":"10.1093/RESEVAL/RVAB002","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB002","url":null,"abstract":"\u0000 Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48860281","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 24

An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations 同行评审公开期刊审稿人评论探究：评审语言的特点及评审审查与引用的关系

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-03-13 DOI: 10.1093/RESEVAL/RVAB005

D. Wolfram, Peiling Wang, Fuad Abuzahra

{"title":"An exploration of referees’ comments published in open peer review journals: The characteristics of review language and the association between review scrutiny and citations","authors":"D. Wolfram, Peiling Wang, Fuad Abuzahra","doi":"10.1093/RESEVAL/RVAB005","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB005","url":null,"abstract":"\u0000 Journals that adopt open peer review (OPR), where review reports of published articles are publicly available, provide an opportunity to study both review content characteristics and quantitative aspects of the overall review process. This study investigates two areas relevant to the quality assessment of manuscript reviews. First, do journal policies for reviewers to identify themselves influence how reviewers evaluate the merits of a manuscript based on the relative frequency of hedging terms and research-related terms appearing in their reviews? Second, is there an association between the number of reviews/reviewers and the manuscript’s research impact once published as measured by citations? We selected reviews for articles published in 17 OPR journals from 2017 to 2018 to examine the incidence of reviewers’ uses of hedging terms and research-related terms. The results suggest that there was little difference in the relative use of hedging term usage regardless of whether reviewers were required to identify themselves or if this was optional, indicating that the use of hedging in review contents was not influenced by journal requirements for reviewers to identify themselves. There was a larger difference observed for research-related terminology. We compared the total number of reviews for a manuscript, rounds of revisions, and the number of reviewers with the number of Web of Science citations the article received since publication. The findings reveal that scrutiny by more reviewers or conducting more reviews or rounds of review do not result in more impactful papers for most of the journals studied. Implications for peer review practice are discussed.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-03-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"49265908","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Transforming science and society? Methodological lessons from and for transformation research 改变科学和社会？转型研究的方法论经验教训

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-03-12 DOI: 10.1093/RESEVAL/RVAA034

K. Hölscher, Julia M. Wittmayer, M. Hirschnitz-Garbers, A. Olfert, Jörg Walther, G. Schiller, Benjamin Brunnow

{"title":"Transforming science and society? Methodological lessons from and for transformation research","authors":"K. Hölscher, Julia M. Wittmayer, M. Hirschnitz-Garbers, A. Olfert, Jörg Walther, G. Schiller, Benjamin Brunnow","doi":"10.1093/RESEVAL/RVAA034","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAA034","url":null,"abstract":"\u0000 Transformation research has in the past years emerged as a shared lens to study and support radical societal change towards sustainability. Given the nascent and exploratory—yet highly normative and ambitious—character of transformation research, we aim to enhance the understanding of transformation research: when do research designs qualify as transformation research, what is needed for putting transformation research into practice, and what are results? To this end, we develop a framework that identifies criteria for designing and reflecting on research results, design and processes as transformation research. We employ this framework to reflect on our work in a research project that was designed in the spirit of transformation research: The TRAFIS (Transformations towards resource-conserving and climate-resilient coupled infrastructures) project sought to understand and support the development of innovative coupled infrastructures to mobilize their critical role in achieving sustainability transformations. Our results yield lessons and recommendations about what transformation research looks like in practice and how it can be strengthened, focussing on 1, redefining and re-valuing research for societal impact; 2, redesigning research to integrate perspectives on radical societal change; and 3, re-equipping researchers and research partners for social learning. We conclude that while transformation research already contributes to framing and generating knowledge about real-world sustainability challenges, its transformative impact is still limited. Practicing transformation research requires far-reaching changes in the science system, but also continuous reflection about legitimacy, power relations, and impacts.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-03-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/RESEVAL/RVAA034","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45815610","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 25

Estimating the effects of public subsidies on the performance of supported enterprises across firm sizes 估计公共补贴对不同企业规模的支持企业绩效的影响

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-03-03 DOI: 10.1093/RESEVAL/RVAB004

O. Dvouletý, I. Blažková, Oto Potluka

{"title":"Estimating the effects of public subsidies on the performance of supported enterprises across firm sizes","authors":"O. Dvouletý, I. Blažková, Oto Potluka","doi":"10.1093/RESEVAL/RVAB004","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB004","url":null,"abstract":"\u0000 Only a few studies consider heterogeneity in the effects of investment grants (subsidies) across firm size. Therefore, we investigate the effects of particular public policy on firm-level performance with a focus on firm size heterogeneity as an important determinant. We aim to investigate whether the larger-sized firms benefit from the direct financial assistance to a lesser extent when compared with micro- and small-sized firms. Specifically, we study the microeconomic effects of the Czech Operational Programme Enterprise and Innovation (OPEI) that was implemented during the period of 2007–13. Compared to previous studies, we work with a large firm-level dataset, consisting of 13,924 firms, of which 3,572 are supported firms (57% of beneficiaries of the OPEI programme). We implement the propensity score matching in combination with a difference in differences approach. We measure the overall financial performance of firms by using tangible fixed assets and their depreciation, sales, return on assets, and total factor productivity. While there are improvements in all evaluated indicators for microenterprises 2 years after the end of the programme, there are only minor positive effects of subsidies for larger enterprises.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/RESEVAL/RVAB004","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42230592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 5

SSH researchers make an impact differently. Looking at public research from the perspective of users SSH研究人员产生了不同的影响。从用户的角度看公共研究

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-03-02 DOI: 10.1093/RESEVAL/RVAB008

A. Bonaccorsi, F. Chiarello, G. Fantoni

{"title":"SSH researchers make an impact differently. Looking at public research from the perspective of users","authors":"A. Bonaccorsi, F. Chiarello, G. Fantoni","doi":"10.1093/RESEVAL/RVAB008","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB008","url":null,"abstract":"\u0000 With the rise of the impact assessment revolution, governments and public opinion have started to ask researchers to give evidence of their impact outside the traditional audiences, i.e. students and researchers. There is a mismatch between the request to demonstrate the impact and the current methodologies for impact assessment. This mismatch is particularly worrisome for the research in Social Sciences and Humanities. This paper gives a contribution by examining systematically a key element of impact, i.e. the social groups that are directly or indirectly affected by the results of research. We use a Text mining approach applied to the Research Excellence Framework (REF) collection of 6,637 impact case studies in order to identify social groups mentioned by researchers. Differently from previous studies, we employ a lexicon of user groups that includes 76,857 entries, which saturates the semantic field, permits the identification of all users and opens the way to normalization. We then develop three new metrics measuring Frequency, Diversity and Specificity of user expressions. We find that Social Sciences and Humanities exhibit a distinctive structure with respect to frequency and specificity of users.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-03-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/RESEVAL/RVAB008","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46982973","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 6

On the relationship between interdisciplinarity and impact: Distinct effects on academic and broader impact 论跨学科与影响的关系:对学术和更广泛影响的显著影响

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-03-01 DOI: 10.1093/RESEVAL/RVAB007

Lin Zhang, Beibei Sun, Lidan Jiang, Ying Huang

{"title":"On the relationship between interdisciplinarity and impact: Distinct effects on academic and broader impact","authors":"Lin Zhang, Beibei Sun, Lidan Jiang, Ying Huang","doi":"10.1093/RESEVAL/RVAB007","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB007","url":null,"abstract":"\u0000 Addressing many of the world’s contemporary challenges requires a multifaceted and integrated approach and, in this respect, interdisciplinary research (IDR) is increasingly recognized as central to both academic interests and national science policies. In spite of the growing attention given to IDR, the impact of IDR remains under-investigated. In this study, we analyzed the influence of interdisciplinarity on citation impact (particularly, WoS citation) and broader impact (particularly, PloS usage) at the article level. We measured IDR in terms of three different elements of diversity—variety, balance, and disparity—as well as the integrated diversity overall. The results of negative binomial regression analysis with field fixed effects and robust standard errors show the positive effects of interdisciplinarity on both academic and broader impact. From an analysis of trends over time, the results show that higher interdisciplinary publications tend to attract more citations and have higher PLoS usage. Compared to citations, which need a more extended period to accumulate, the advantage of measuring impact with PLoS usage is its immediacy. Also, there are signs that PLoS usage and citations can mutually reinforce each other.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-03-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/RESEVAL/RVAB007","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48577544","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 21

Does government support of a few leading universities have a broader impact on the higher education system? Evaluation of the Russian University Excellence Initiative 政府对几所顶尖大学的支持是否对高等教育体系产生了更广泛的影响?俄罗斯大学卓越计划评价

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-02-28 DOI: 10.1093/RESEVAL/RVAB006

Andrey Lovakov, Anna Panova, Ivan Sterligov, M. Yudkevich

{"title":"Does government support of a few leading universities have a broader impact on the higher education system? Evaluation of the Russian University Excellence Initiative","authors":"Andrey Lovakov, Anna Panova, Ivan Sterligov, M. Yudkevich","doi":"10.1093/RESEVAL/RVAB006","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB006","url":null,"abstract":"\u0000 Many governments attempt to improve national higher education through the competitive support of universities. These policy approaches raise questions about the impact on the entire system—both in research and educational—of targeted support for a small number of universities. Addressing challenges in the measurement of university excellence initiatives are among the most vital topics in research evaluation due to the central roles they often play in national research and university policy efforts. Using data from the Russian University Excellence Initiative (RUEI), we measure the spillover effects of such focused support and demonstrate that a broader impact does exist. In particular, we examine the performance of higher education institutions that were not part of RUEI and were not directly supported by it. We compare the university performance in regions both with and without RUEI universities. In doing so, we measure the indirect impact of RUEI on the higher education sector at the regional level. We show a positive effect on the level of publication activity that has recently become apparent. However, there has been no effect on the share of young faculty, international collaboration in publications, or the quality of enrollment. Judging from the broader research policyresearch evaluation perspective, our study sheds light on the systemic effects of excellence initiatives, which are often neglected. Besides, excellence initiatives could trigger a change in the approach to evaluating research. So government should develop measure properly, taking into account various consequences, some of which are considered in our article.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46125338","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

The effect of facilitating interdisciplinary cooperation on the research productivity of university research teams: The moderating role of government assistance 促进跨学科合作对大学科研团队研究生产力的影响:政府资助的调节作用

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-01-27 DOI: 10.1093/RESEVAL/RVAB001

Aihua Chen, Xiao-tang Wang

{"title":"The effect of facilitating interdisciplinary cooperation on the research productivity of university research teams: The moderating role of government assistance","authors":"Aihua Chen, Xiao-tang Wang","doi":"10.1093/RESEVAL/RVAB001","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAB001","url":null,"abstract":"\u0000 Interdisciplinary research is vital to research productivity. However, the existing research has not given a reasonable explanation on how and under what condition facilitating interdisciplinary research cooperation promotes research productivity. This article aims to contribute to the existing literature by examining the link between facilitating interdisciplinary cooperation and research productivity as well as the role of government assistance in this link. Using a sample of 314 members of Chinese university research teams, we show that management, operations, evaluation, team building, and collaborating with industry to facilitate interdisciplinary cooperation all have significantly positive effects on the research productivity of university research teams. Among them, evaluation has the most significant impact on research productivity. Moreover, government assistance weakens the relationship between evaluation and research productivity but strengthens the relationship between management and research productivity. However, the moderating effects of government assistance on the links among team building, operations, collaborating with industry, and team research productivity are not statistically significant. Overall, our study has important implications for governments and universities seeking to facilitate interdisciplinary cooperation.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-01-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1093/RESEVAL/RVAB001","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41778714","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Funding acknowledgements in scientific publications: A literature review 科学出版物中的资助确认：文献综述

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-01-19 DOI: 10.1093/RESEVAL/RVAA038

B. Álvarez-Bornstein, M. Montesi

{"title":"Funding acknowledgements in scientific publications: A literature review","authors":"B. Álvarez-Bornstein, M. Montesi","doi":"10.1093/RESEVAL/RVAA038","DOIUrl":"https://doi.org/10.1093/RESEVAL/RVAA038","url":null,"abstract":"\u0000 The topic of acknowledgements has produced abundant research since the 1970s, though, as previous studies point out, the value of acknowledgements has not yet been demonstrated and further research is limited by lack of conceptualization. This study focuses on funding acknowledgements (FAs), considering that funding represents an important input in the scientific process. In this context, 183 scientific publications retrieved from Scopus from the 1970s until June 2020 were analyzed, with the aim of systematizing conceptually this body of research and contributing to a theory of acknowledgements. Results are summarized into the following main themes: the meaning of FAs; data sources for acknowledgements; the process of funding; association of funding with productivity, impact, and collaboration; and other aspects affected by funding. The literature reviewed shows that a theory of acknowledgements based on the reward triangle, as in previous studies, is unable to capture the extreme complexity of the scientific activity affecting and being affected by FAs. Funding bodies appear as clear and influential actors in the scientific communication system, making important decisions on the research that is supported, and influencing the type of knowledge produced. Funding agencies hold a responsibility regarding the data that they may collect on their programs, as well as the normalization policies they need to develop so that funded authors can reference with less ambiguity the financial source of their projects. Finally, the need to assess the impact of research funding beyond the scientific community that is, the societal impact, is also addressed.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2021-01-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42653862","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 18

Mirror, mirror on the wall: is economics the fairest of them all? An investigation into the social sciences and humanities in Vietnam 镜子，墙上的镜子：经济学是所有经济学中最公平的吗？越南社会科学与人文学科调查

IF 3.3 4区管理学

Research Evaluation Pub Date : 2021-01-17 DOI: 10.1093/RESEVAL/RVAA036

Q. Vuong, A. Bui, Viet-Phuong La, Minh-Hoang Nguyen, Hung-Hiep Pham, Thanh-Hang Pham, Thi-Hanh Vu, Thu-Trang Vuong, Manh-Toan Ho

引用次数: 9