Research Evaluation最新文献

Generative AI can and should accelerate research evaluation reform to better recognize 'distinctly human contributions'. 生成式人工智能可以而且应该加速研究评估改革，以更好地认识“明显的人类贡献”。

IF 2.5 4区管理学

Research Evaluation Pub Date : 2026-04-11 eCollection Date: 2026-01-01 DOI: 10.1093/reseval/rvag020

Mohammad Hosseini, Brian D Earp, Sebastian Porsdam Mann, Kristi Holmes

{"title":"Generative AI can and should accelerate research evaluation reform to better recognize 'distinctly human contributions'.","authors":"Mohammad Hosseini, Brian D Earp, Sebastian Porsdam Mann, Kristi Holmes","doi":"10.1093/reseval/rvag020","DOIUrl":"https://doi.org/10.1093/reseval/rvag020","url":null,"abstract":"<p><p>As generative artificial intelligence (GenAI) revolutionizes how research is conducted, it also challenges traditional methods of scholarly evaluation. Productivity metrics such as publication and citation counts are widely understood to be poor proxies for gauging meaningful impact. These metrics are becoming even less reliable as GenAI accelerates text-based and computational work while leaving other forms of research labor (e.g. community engagement, in-person mentorship and team development) largely unaffected. This uneven effect risks exacerbating existing evaluative biases. We argue that evaluation reforms should be organized around two categories of 'distinctly human contributions' that are indispensable to research, but which are inadequately captured by metrics: (1) the epistemic-ethical category, encompassing situated judgment under accountability (e.g. deciding what to trust, justifying that decision, and standing behind it); and (2) the socio-relational category, encompassing sustained forms of valuable human engagement (e.g. mentoring, teaching, community partnership and trust-building). We suggest practical mechanisms for supporting evaluation reform including modified CRediT (Contributor Role Taxonomy) statements, recognition of a broader array of outputs, and strengthened narrative CVs and third-person testimonies. However, we acknowledge that these suggestions, particularly those relying on narrative self-presentation, are themselves vulnerable to GenAI manipulation and are insufficient on their own. If distinctly human contributions to research require judgment and relationships that resist automation, then evaluation cannot be reduced to instruments designed to minimize human evaluative effort. GenAI, therefore, does not require entirely new systems of evaluation. Rather, it increases the cost of avoiding what good and ethically sound performance evaluation has always required.</p>","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"35 ","pages":"rvag020"},"PeriodicalIF":2.5,"publicationDate":"2026-04-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13094469/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147785769","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Correction to: Methods for measuring social and conceptual dimensions of convergence science 更正为聚合科学的社会和概念测量方法

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-12-07 DOI: 10.1093/reseval/rvad036

引用次数: 0

Correction to: Stated preference methods and STI policy studies: a foreground approach 更正：陈述偏好方法与科技创新政策研究：前景方法

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-12-07 DOI: 10.1093/reseval/rvad037

引用次数: 0

A tribute to our dearly departed colleague and friend: An introduction to the Special Issue in memory of Prof. Paul Benneworth 向我们逝去的同事和朋友致敬：纪念保罗-本纽斯教授特刊导言

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-11-29 DOI: 10.1093/reseval/rvad010

G. E. Derrick, J. Olmos-Peñuela, S. D. de Jong

引用次数: 0

The legal foundation of responsible research assessment: An overview on European Union and Italy 负责任研究评估的法律基础：欧盟和意大利概况

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-11-28 DOI: 10.1093/reseval/rvad035

G. Peruginelli, Janne Pölönen

{"title":"The legal foundation of responsible research assessment: An overview on European Union and Italy","authors":"G. Peruginelli, Janne Pölönen","doi":"10.1093/reseval/rvad035","DOIUrl":"https://doi.org/10.1093/reseval/rvad035","url":null,"abstract":"During the past decade, responsible research assessment (RRA) has become a major science policy goal to advance responsible research and innovation and open science. Starting with the DORA declaration in 2012, common understanding of the needs and demands of RRA has been shaped by a growing number of initiatives, culminating in a European Commission supported Agreement on reforming research assessment published in July 2022. In this paper, we outline and investigate the normative framework for RRA, ranging from binding legal norms (hard law), rules of conduct which, in principle, have no legally binding force but which nevertheless may have practical effects’ (soft law) to self-regulation in supranational (the European Union) and one national (Italy) context. As shown by our analysis of hard and soft legal sources for RRA, apart from constitutional and legal provisions regarding freedom of research, autonomy of universities and non-discrimination and gender equality, the legal foundation for RRA rests mainly on soft law and policy norms. In the discussion we relate the normative framework for RRA to research integrity guidelines and consider the challenges of ensuring coherent RRA implementation across countries and research funding and performing institutions. We conclude that self-regulation and soft law are good starting points and may pave the way for hard law down the line: this can be seen as the optimum mix of regulatory mechanisms to mainstream RRAs.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"204 1","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-11-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139220180","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

The conflict of impact for early career researchers planning for a future in the academy 早期职业研究人员在学术界规划未来的影响力冲突

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-11-21 DOI: 10.1093/reseval/rvad024

Marta Natalia Wróblewska, Corina Balaban, Gemma Derrick, Paul Benneworth

{"title":"The conflict of impact for early career researchers planning for a future in the academy","authors":"Marta Natalia Wróblewska, Corina Balaban, Gemma Derrick, Paul Benneworth","doi":"10.1093/reseval/rvad024","DOIUrl":"https://doi.org/10.1093/reseval/rvad024","url":null,"abstract":"It has been argued that due to the growing importance attributed to research impact and forms of its evaluation, an academic ‘culture of impact’ is emerging. It would include certain concepts, values, and skills related to the area of generating and documenting impact. We use thematic and discourse analysis to analyse open answers from 100 questionnaires on research impact submitted by ECRs working in the social sciences and humanities (SSH) in Europe. We explore ECR’s early-career stage positions relative to societal impact and the trade-offs necessary to assure an academic career. The results show how, as the first generation of scholars to be socialized towards value of academic research beyond academia, ECRs are confronted with policy signals that encourage a drive for impact, which are at the same time often in line with respondents’ personal values around impact beyond academia. However, ECRs face a number of competing signals about research value within the evaluation spaces necessary to navigate an academic career. Current evaluative structures often dismiss the achievement of societal impact favouring instead narrower definitions of research excellence. Career structures and organizational realities are often unfavourable to impact-related activity, which has implications for an ECRs’ ability to develop coherent professional positionings.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"35 26","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-11-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139252593","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Individual and organizational effects of international mobility policy: Effects of Mexico’s Scholarship Program in nanotechnology 国际流动政策对个人和组织的影响:墨西哥纳米技术奖学金计划的影响

4区管理学

Research Evaluation Pub Date : 2023-11-10 DOI: 10.1093/reseval/rvad031

Mayra M Tirado

{"title":"Individual and organizational effects of international mobility policy: Effects of Mexico’s Scholarship Program in nanotechnology","authors":"Mayra M Tirado","doi":"10.1093/reseval/rvad031","DOIUrl":"https://doi.org/10.1093/reseval/rvad031","url":null,"abstract":"Abstract This paper addresses policy effects of international mobility in developing countries. It proposes a multilevel approach to study research training policies, specifically focusing on international mobility schemes as capacity-building instruments, where effects need to be identified at both individual and organizational levels. The paper asserts that current categorizations of country efforts to build a solid scientific base should consider transference of effects in domestic research contexts. Using a survey and interview-based study of a mobility policy in Mexico, and nanotechnology-related disciplines and sectors, this paper’s findings suggest that mobility policies are crucial in creating high-quality individual research skills, but that their impact on building domestic research capacity may be modest. This paper contributes to the evaluation literature on national research training programs and their influence on research capacity in two ways. First, it mobilizes a multilevel analytical approach to examine the effects of policies on individuals and organizations, offering a broader conceptualization of impact. Second, by introducing these individual and organizational effects, the paper provides insights into how effects are enacted and transferred.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" 520","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135186623","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices 目标明确，可操作和公平:审稿人报告作为反馈及其对ECR职业选择的影响

4区管理学

Research Evaluation Pub Date : 2023-11-02 DOI: 10.1093/reseval/rvad034

Gemma Elizabeth Derrick, Alessandra Zimmermann, Helen Greaves, Jonathan Best, Richard Klavans

{"title":"Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices","authors":"Gemma Elizabeth Derrick, Alessandra Zimmermann, Helen Greaves, Jonathan Best, Richard Klavans","doi":"10.1093/reseval/rvad034","DOIUrl":"https://doi.org/10.1093/reseval/rvad034","url":null,"abstract":"Abstract Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the ‘best’ decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful grant application, is associated with motivating applicants career decisions to persist (reapply for funding at T1), or to switch (not to reapply, or else leave academia). This study combined data from interviews with unsuccessful ECR applicants (n = 19) to The Wellcome Trust 2009–19, and manual coding of reviewer comments received by applicants (n = 81). All applicants received feedback on their application at T0 with a large proportion of unsuccessful applicants reapplying for funding at T1. Here, peer-review-comments-as-feedback sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. Feedback associated by unsuccessful applicants as motivating their decision to resubmit had three characteristics: actionable; targeted; and fair. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The provision of quality reviewer-reports-as-feedback to applicants, ensures that peer review acts as a participatory research governance tool focused on supporting the development of individuals and their future research plans.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"13 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135976131","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Effects of transdisciplinary research on scientific knowledge and reflexivity 跨学科研究对科学知识和反身性的影响

4区管理学

Research Evaluation Pub Date : 2023-11-01 DOI: 10.1093/reseval/rvad033

Oskar Marg, Lena Theiler

引用次数: 0

Evaluating transformative innovation policy in a formative way: Insights from Vinnova’s food mission experiment 以形成的方式评估变革性创新政策:来自Vinnova食品使命实验的见解

4区管理学

Research Evaluation Pub Date : 2023-10-28 DOI: 10.1093/reseval/rvad029

Alejandra Boni, Diana Velasco, Jordi Molas-Gallart, Johan Schot

{"title":"Evaluating transformative innovation policy in a formative way: Insights from Vinnova’s food mission experiment","authors":"Alejandra Boni, Diana Velasco, Jordi Molas-Gallart, Johan Schot","doi":"10.1093/reseval/rvad029","DOIUrl":"https://doi.org/10.1093/reseval/rvad029","url":null,"abstract":"Abstract This article presents the insights from an evaluation of a transformative policy experiment, in the food domain, conducted at the Swedish Agency of Innovation (Vinnova). To be consistent with the principles and objectives of these policies, it was necessary to implement a formative evaluation approach developed in the Transformative Innovation Policy Consortium (TIPC), that is supportive of experimental policies designed for highly complex and uncertain environments. The article presents the main characteristics of this approach and its differences and similarities with other proposals to assess transformative innovation policies. Next, the article describes how the evaluation was conducted in Vinnova during a 2-year engagement. Subsequently, the main insights gleaned from the process are highlighted in terms of (1) understanding the expectations of actors; (2) what has been learned by doing, from a technical and organizational point of view; and (3) the constraints that emerged during the engagement. It is then argued that the TIPC- formative evaluation approach challenges the role of evaluators, stretching them, and requires different attitudes from those that are common in evaluation practice.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"1 3","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-10-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136160057","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0