Research Evaluation最新文献_第3页

Boundary-work and social closure in academic recruitment: Insights from the transdisciplinary subject area Swedish as a Second Language 学术招聘中的边界工作和社会封闭：瑞典语作为第二语言跨学科领域的见解

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-05-15 DOI: 10.1093/reseval/rvad015

Natalia Ganuza, L. Salö

{"title":"Boundary-work and social closure in academic recruitment: Insights from the transdisciplinary subject area Swedish as a Second Language","authors":"Natalia Ganuza, L. Salö","doi":"10.1093/reseval/rvad015","DOIUrl":"https://doi.org/10.1093/reseval/rvad015","url":null,"abstract":"\u0000 This article explores practices of evaluation in academic recruitment in Swedish as a Second Language (SSL), an expanding and transdisciplinary subject area. As is common elsewhere, Swedish academia relies on a tradition of external expert review intended to ensure a meritocratic process. Here, we present an analysis of 109 written expert reports concerning recruitment to 57 positions in SSL during 2000–20. Because SSL lacks institutional autonomy, and is spread across several sub-disciplines in the humanities and social sciences, the material encompasses experts with diverse academic backgrounds. The SSL reports are broadly characterized by qualitative assessment. In contrast to other fields, the SSL experts seldom use quantitative proxy measures. Instead, they mainly rely on received conceptions of the boundaries of SSL as a means of justifying their inclusion and exclusion of candidates. This dominant regularity consists of attempts to define and delimit SSL and its core research areas, to locate the candidates in a core-to-periphery scheme with respect to these boundaries, and to rank them accordingly. This mechanism of social closure serves to restrict access to SSL to candidates with qualifications that conform to the experts’ own conceptions of SSL. As we show, the experts’ internally ambiguous conceptions of SSL tend to be constructed in relation to their own scientific habitus and investments. Beyond evaluating applicants’ possession of scientific capital, their distinctive style of reasoning around research qualifications and skills thus involves power-laden boundary-work, which leaves ample room for individual, yet habitus-specific arbitrariness.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42729543","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

The missing links of research impact 研究影响的缺失环节

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-05-12 DOI: 10.1093/reseval/rvad011

David Budtz Pedersen, Rolf Hvidtfeldt

{"title":"The missing links of research impact","authors":"David Budtz Pedersen, Rolf Hvidtfeldt","doi":"10.1093/reseval/rvad011","DOIUrl":"https://doi.org/10.1093/reseval/rvad011","url":null,"abstract":"\u0000 In this article, we present a conceptual framework for studying research impact focusing on the foundations that need to be in place to accelerate an observable change of policy, practice or behaviour. The article investigates the relationship between micro-impacts and societal change, and how smaller impacts scale into larger cascades of end effects and value creation. We define micro-impacts as interactions and connections where information is exchanged between a researcher or research group and external audiences, stakeholders or co-producers. Micro-impacts are elements in highly complex causal relations between research activities and larger societal macroshifts. We argue that even though these causal relations are complex, micro-impacts are tangible and observable and should be integrated in research evaluations as constitutive elements of causal impact relations leading to larger macroshifts. We suggest a working model for studying micro-impacts and for reflecting on the causality of impacts by drawing on contributions from philosophy of causation. A proper understanding of causation is a prerequisite for eventually understanding and capturing research impact, which itself is a prerequisite for responsible research assessment and planning.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-05-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48033661","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Promoting narrative CVs to improve research evaluation? A review of opinion pieces and experiments 推动叙事简历提高研究评价?评论文章和实验

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-04-19 DOI: 10.1093/reseval/rvad013

Frédérique Bordignon, Lauranne Chaignon, D. Egret

{"title":"Promoting narrative CVs to improve research evaluation? A review of opinion pieces and experiments","authors":"Frédérique Bordignon, Lauranne Chaignon, D. Egret","doi":"10.1093/reseval/rvad013","DOIUrl":"https://doi.org/10.1093/reseval/rvad013","url":null,"abstract":"\u0000 As the academic community has become increasingly concerned about the drifts of research evaluation, mostly researchers’ evaluation, because of the overreliance on metrics, many expert groups have made recommendations to improve the way researchers should be evaluated. In this study, we focus on the recommendation to use narrative curriculum vitae (CVs). We review 28 opinion pieces and 7 experiments to better understand what a narrative CV can refer to, and to explore whether the narrative function that is specific to this kind of CV is proving effective in response to the concerns raised by evaluation practices. A close reading of these documents reveals the conceptual basis of the narrative CV and the problems it is intended to solve; we propose five commonly reported features of the narrative CV: avoid lists, contextualize achievements, fight metrics, enlarge the spectrum of contributions taken into consideration and foster diversity and inclusion. But the promoters of the narrative CV pay little to investigate how the narrative feature itself can lead to any benefits. However, the feedback collected from both applicants and evaluators is quite positive. Regardless of whether it is justified or not, the enthusiasm aroused by the implementation of this new type of CV undeniably has the advantage of opening up the debate, raising awareness and calling to question the bad practices and biases that exist in the researchers’ assessment processes. The narrative nature of the CV is, in the end, just a pretext for raising interest and working towards the adoption of good practices.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-04-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46169854","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

The price of quality: Scholarly publishing business is the primary predictor of citation-based indicators of journal performance in ecology and evolutionary biology 质量的代价:学术出版业务是基于引用的生态学和进化生物学期刊绩效指标的主要预测指标

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-04-19 DOI: 10.1093/reseval/rvad012

A. Ruggiero

{"title":"The price of quality: Scholarly publishing business is the primary predictor of citation-based indicators of journal performance in ecology and evolutionary biology","authors":"A. Ruggiero","doi":"10.1093/reseval/rvad012","DOIUrl":"https://doi.org/10.1093/reseval/rvad012","url":null,"abstract":"\u0000 Citation-based indicators of journals’ performance are often assumed to offer an objective, albeit indirect, way of measuring research quality. However, recent concerns about their applicability for research evaluation suggested these indicators could depend on historical and socioeconomic factors associated with scholarly publishing tradition and business, respectively. The present study addressed this issue quantitatively, using data on h-index and Scimago Journal Rank (SJR) for 566 journals within the fields of ecology and evolutionary biology, and applying Partial Least Squared Structural Equation Modelling. The Tradition Model accounted for <50% of the variation in h-index and SJR, showing that journals’ performance increased with an increase in articles’ international collaboration, and decreased for journals published by non-profit organizations. The Business Model accounted for >60% of the variation in h-index and SJR, showing that journals’ performance increased in association with the global50 ranking of publishers and high article processing charges. Countries recognized as world science centres, the use of English, the journal’s, and publisher’s year of origin, and the increase in science investment and scientific production promoted by the richest economies worldwide had no impact on journal performance. Results suggest that the h-index for journals and the SJR reflect multi-dimensional aspects of scholarly publishing, potentially affected by marketing strategies boosted by the biggest commercial publishers. Given the limitations of poor scientific communities in terms of publication costs, uncritical application of these indexes for research evaluation worldwide may reinforce the idea that high quality research is produced only by rich scientific societies.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-04-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41598351","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Correction to: Journal citation reports and the definition of a predatory journal: The case of the Multidisciplinary Digital Publishing Institute (MDPI) 期刊引文报告与掠夺性期刊的定义:以多学科数字出版研究所(MDPI)为例

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-04-18 DOI: 10.1093/reseval/rvad014

引用次数: 9

Methods for measuring social and conceptual dimensions of convergence science 测量趋同科学的社会和概念维度的方法

4区管理学

Research Evaluation Pub Date : 2023-04-01 DOI: 10.1093/reseval/rvad020

Alexander Michael Petersen, Felber Arroyave, Ioannis Pavlidis

{"title":"Methods for measuring social and conceptual dimensions of convergence science","authors":"Alexander Michael Petersen, Felber Arroyave, Ioannis Pavlidis","doi":"10.1093/reseval/rvad020","DOIUrl":"https://doi.org/10.1093/reseval/rvad020","url":null,"abstract":"Abstract Convergence science is an intrepid form of interdisciplinarity defined by the US National Research Council as ‘the coming together of insights and approaches from originally distinct fields’ to strategically address grand challenges. Despite its increasing relevance to science policy and institutional design, there is still no practical framework for measuring convergence. We address this gap by developing a measure of disciplinary distance based upon disciplinary boundaries delineated by hierarchical ontologies. We apply this approach using two widely used ontologies—the Classification of Instructional Programs and the Medical Subject Headings—each comprised of thousands of entities that facilitate classifying two distinct research dimensions, respectively. The social dimension codifies the disciplinary pedigree of individual scholars, connoting core expertise associated with traditional modes of mono-disciplinary graduate education. The conceptual dimension codifies the knowledge, methods, and equipment fundamental to a given target problem, which together may exceed the researchers’ core expertise. Considered in tandem, this decomposition facilitates measuring social-conceptual alignment and optimizing team assembly around domain-spanning problems—a key aspect that eludes other approaches. We demonstrate the utility of this framework in a case study of the human brain science (HBS) ecosystem, a relevant convergence nexus that highlights several practical considerations for designing, evaluating, institutionalizing, and accelerating convergence. Econometric analysis of 655,386 publications derived from 9,121 distinct HBS scholars reveals a 11.4% article-level citation premium attributable to research featuring full topical convergence, and an additional 2.7% citation premium if the social (disciplinary) configuration of scholars is maximally aligned with the conceptual (topical) configuration of the research.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"131 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135722650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Evaluating co-creation in social innovation projects: Towards a process orientated framework for EU projects and beyond 评估社会创新项目中的共同创造:面向欧盟项目及其他项目的过程导向框架

4区管理学

Research Evaluation Pub Date : 2023-04-01 DOI: 10.1093/reseval/rvad017

Peter Meister Broekema, Elisabeth A M Bulder, Lummina G Horlings

{"title":"Evaluating co-creation in social innovation projects: Towards a process orientated framework for EU projects and beyond","authors":"Peter Meister Broekema, Elisabeth A M Bulder, Lummina G Horlings","doi":"10.1093/reseval/rvad017","DOIUrl":"https://doi.org/10.1093/reseval/rvad017","url":null,"abstract":"Abstract In the last two decades, co-creation and social innovation have become important concepts in academic research and public policy. The two concepts are conceptually linked, but this relationship has hardly been problematized in academic literature. In addition, social innovation and especially co-creation are not defined in EU policies, but merely included because they support policy aims. The lack of problematization and definition not only hampers progress in the academic field, but is also constringing co-creation into an exercise of merely including stakeholders therefore neglecting the full potential of co-creation. The key question addressed in this article is therefore: how can we evaluate the application of co-creation in EU-funded social innovation projects? A literature review revealed that co-creation and social innovation have become connected only very recently in academic literature. In this publication, we analyse the meta narratives of this emerging body of literature and conclude that we can distinguish three distinct segments with their own characteristics. We used these insights to develop an adaptive evaluation framework. This framework can be used to assess the application of co-creation within social innovation in, for example, EU-funded projects. This could push the emerging academic field forward and open up new research themes and designs. We also suggest that the framework could specifically support policymakers in their efforts to evaluate processes of co-creation instead of focusing on the dominant impact evaluations.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135672470","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Stated preference methods and STI policy studies: a foreground approach 陈述偏好方法和科技创新政策研究:前景方法

4区管理学

Research Evaluation Pub Date : 2023-04-01 DOI: 10.1093/reseval/rvad022

Víctor Gómez-Valenzuela

{"title":"Stated preference methods and STI policy studies: a foreground approach","authors":"Víctor Gómez-Valenzuela","doi":"10.1093/reseval/rvad022","DOIUrl":"https://doi.org/10.1093/reseval/rvad022","url":null,"abstract":"Abstract This discussion article explores the ontological and epistemic basis for analysing social preferences in the broader interdisciplinary field of Science, Technology, and Innovation (STI) policy studies and its evaluation using stated preference (SP) methods. STI policy studies base their approximations of policy problems on a revealed preference (RP) approach, which analyses economic agents' actual market behaviours based on standardized data sources. SP methods arose as an alternative to address the analysis of public goods for which the market fails to assign prices efficiently and can only be evaluated in hypothetical or contingent situations. In an analytical context of complexity defined by grand societal challenges related to the provision of public goods to be addressed by STI transformative policies, analysing social preferences by SP methods could support a more robust and holistic approach to STI policy analysis and its evaluation, improving the policy-making process and promoting more informed policy mixes and evaluation policy mixes. A kind of Kantian categorical imperative favouring SP methods is discussed based on the new STI policy research agenda on transformative change and supported by axiology around social choice, welfare, and a more participative STI policy governance.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135771336","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Improving universities’ activities in academic startup support through public interventions: The effectiveness of the German programme ‘EXIST—leverage of potentials’ 通过公共干预改善大学在学术创业支持方面的活动:德国“存在-潜力杠杆”项目的有效性

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-03-28 DOI: 10.1093/reseval/rvad009

C. Mueller

{"title":"Improving universities’ activities in academic startup support through public interventions: The effectiveness of the German programme ‘EXIST—leverage of potentials’","authors":"C. Mueller","doi":"10.1093/reseval/rvad009","DOIUrl":"https://doi.org/10.1093/reseval/rvad009","url":null,"abstract":"\u0000 Academic startups have a considerable economic impact, which is why public support programmes for them are considered an important component of innovation and technology policy. In this context, university support programmes can be an important part of the policy toolkit by aiming to improve universities’ startup environment and thus promote startup activity at those institutions. Assessing the effectiveness of these programmes is a key evaluation task inasmuch as it provides an evidence base for decision-makers and broadens the discourse on promoting startup culture at universities. This study reports on the background, methodology, and results of the evaluation of the effectiveness of a large university support programme in the academic startup sector in Germany, ‘EXIST—Leverage of potentials’. This programme supports universities which have little experience in building a startup culture and startup-supportive structures. Reliable data are available for two indicators that can be employed to assess intervention effects by means of a difference-in-differences design, namely for the number of applications universities submitted and the number of grants they received in what is Germany’s largest funding programme for prospective startups. The findings indicate that funding by ‘EXIST—Leverage of potentials’ positively affects universities’ activities in the area of startup support.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-03-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42915796","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Changing conceptualization of innovation in the European Union and its impact on universities: Critical junctures and evolving institutional demands 欧盟不断变化的创新概念及其对大学的影响：关键时刻和不断变化的制度需求

IF 3.3 4区管理学

Research Evaluation Pub Date : 2023-03-17 DOI: 10.1093/reseval/rvad006

Ridvan Cinar, P. Benneworth, Lars Coenen

{"title":"Changing conceptualization of innovation in the European Union and its impact on universities: Critical junctures and evolving institutional demands","authors":"Ridvan Cinar, P. Benneworth, Lars Coenen","doi":"10.1093/reseval/rvad006","DOIUrl":"https://doi.org/10.1093/reseval/rvad006","url":null,"abstract":"\u0000 This article explores underlying mechanisms triggering a change in conceptualization of innovation in the European Union (EU), the impact of this change on institutional demands upon European universities and implications for evaluation procedures. We mobilize the theoretical concept of critical junctures to explore significant periods that have affected understanding of innovation in the EU as well as institutional expectations from universities. Through an analysis of European policy corpus relating to innovation, we identify three distinct periods, 1983–6, 1995–2000, and 2008–12, entailing fairly fundamental shifts that have considerably broadened the understanding of innovation and then demonstrate the way this broadening conceptualization has affected institutional demands upon universities. Following this, we discuss the implications of the critical junctures on evaluation approaches. We conclude by arguing that different approaches towards innovation have created complex institutional environment for universities to navigate and suggest that implementing more nuanced and customized evaluation schemes aligned to the institutional demands of each critical juncture could help addressing this complexity.","PeriodicalId":47668,"journal":{"name":"Research Evaluation","volume":" ","pages":""},"PeriodicalIF":3.3,"publicationDate":"2023-03-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46389285","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1