{"title":"Between Bibliometrics and Peer Review: The Evolution and Challenges of Brazil's Qualis System","authors":"A. Brasil","doi":"10.55835/6442f8d5389baf2145b264d9","DOIUrl":"https://doi.org/10.55835/6442f8d5389baf2145b264d9","url":null,"abstract":"This study presents a critical analysis of Brazil's journal classification system, Qualis, examining its evolution and effectiveness in assessing journal quality across diverse academic disciplines. The analysis highlights recent Qualis reforms, including unique classifications, and the adoption of \"mother areas\", plus the application of bibliometric indicators. Preliminary findings from the mid-term evaluation of graduate programs reveal discrepancies between how chosen indicators, including CiteScore and Journal Impact Factor, fail to align with expert committees’ assessment of the same journals. As expected, while these indicators correlate with peer review in a high percentage of STEM journals, they are far from effective in SSH. The study emphasizes the importance of further analysing Qualis' improvements and the role of evaluation committees as the final Qualis classification is released in 2023, thus determining if the system has evolved into a more representative evaluation framework or if its critics are justified in calling for its extinction.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134647385","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Rushforth, Marta Sienkiewicsz, Haley Hazlett, R. Schmidt, S. de Rijcke, Stephen Curry
{"title":"Introducing RRA-Tracker: An online tool mapping the global research assessment reform landscape","authors":"A. Rushforth, Marta Sienkiewicsz, Haley Hazlett, R. Schmidt, S. de Rijcke, Stephen Curry","doi":"10.55835/6439616d4e3bac8a15a26df3","DOIUrl":"https://doi.org/10.55835/6439616d4e3bac8a15a26df3","url":null,"abstract":"Over the past decade, international efforts to reform how research is recognized and rewarded in academia have gathered substantial momentum. Animated by concerns that overly-narrow criteria and indicators of research quality are creating unsustainable and unequal career systems and narrowing knowledge production, various reform movements focusing on responsible metrics, research integrity, open research, Diversity, Equity and Inclusion have coalesced under the label of Responsible Research Assessment (RRA).\u0000\u0000Reform of research assessment remains a challenging process because it impacts established processes, cultures and norms. Despite the growing recognition of the need for change, organisations embarking on reform programmes may not always be sure where is the best place to start. The RRA-Tracker aims to help them overcome that hesitancy. The RRA-Tracker is an online tool for exploring how academic research institutions around the world have introduced and implemented new responsible research assessment practices. The tool enables more experienced institutions to share what they are doing with the wider world, while helping other institutions gain an overview of global developments in assessment reform. It’s populated with policies, roadmaps and other documents from hundreds of institutions all over the world covering innovations in academic hiring, promotion and tenure assessments, together with expertly curated insights to help users discover and discuss what’s possible.\u0000\u0000This poster presentation will introduce the RRA-Tracker, describing its origins, the co-creation process behind it, current coverage and scope, and potential for further development. It will provide screenshots of the tool, as well as QR codes linking the STI audience directly to the tool.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"75 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133182656","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
D. De Filippo, María Luisa Lascurain, Fernando CASANI FERNANDEZ NAVARRETE, Flor Sánchez
{"title":"Diagnosis on the implementation of open science in Spanish institutions. Analysis of projects, publications and perception of stakeholders","authors":"D. De Filippo, María Luisa Lascurain, Fernando CASANI FERNANDEZ NAVARRETE, Flor Sánchez","doi":"10.55835/644237beaa1201f45e9b1ad3","DOIUrl":"https://doi.org/10.55835/644237beaa1201f45e9b1ad3","url":null,"abstract":"This paper presents a methodological and conceptual framework to define the main variables and dimensions to study the academic activities on open science with Spanish participation. We propose a battery of indicators to operationalize the analysis. Using a scientometric methodology, national and international projects obtained in competitive calls related to open science, and scientific publications have been analyzed. Surveys of stakeholders are also included. An intense participation in European projects, one of the main sources of funding for Spanish institutions, is evident. Universities are the predominant sector in national projects, while in European projects institutional diversity is greater. Universities are the ones that publish the most publications on the subject, with a high rate of leadership. Topics related to open data, open innovation, open access and citizen science stand out. The results of the surveys show that academic staff and citizens are poorly informed about open science activities and that open science is still in the process of being implemented in universities.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"187 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133031276","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Is corporate science growing or declining?","authors":"R. Camerani, N. Grassano, D. Rotolo","doi":"10.55835/6442bcd51194fc7a73b429f5","DOIUrl":"https://doi.org/10.55835/6442bcd51194fc7a73b429f5","url":null,"abstract":"The topic of corporate involvement in science has been extensively researched in the innovation literature. However, there is no consensus on whether the trend is on the rise or decline, as various studies have reported contradictory findings. This research aims to bridge this gap by analysing the scientific publication activity over time of a large multi-country and multi-sector sample of firms. The findings of the study indicate that corporate science has grown in the period from 2011 to 2015. However, this growth has only been in absolute terms, as the overall growth of publications in science has been greater. The study also found significant variations in corporate science activity across different sectors. It is important to note, however, that the results may be highly sensitive to the methodological choices, which makes it difficult to compare findings from different studies or make definitive conclusions about the phenomenon.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"147 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132930824","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Comparing Finnish universities’ publication profiles using multidimensional field-normalized indicators","authors":"Otto Auranen, Janne Pölönen","doi":"10.55835/64416fdfb12efcf39479c136","DOIUrl":"https://doi.org/10.55835/64416fdfb12efcf39479c136","url":null,"abstract":"In this study we use eight field-normalized indicators to analyze Finnish universities’ publication profiles across major fields of arts and sciences, and possible changes in publication profiles in 2016-2021. Our data consists of 241,575 publications (publication years 2016–2021) from the national VIRTA publication information service. Results indicate that the Finnish universities differ considerably in their publication profiles in Science communication, Bibliodiversity, Multilingualism, Domestic publishing, Domestic collaboration, International collaboration, Research performance and Open Access. Our indicators show the variety of organizations’ research and publishing profiles as well as the variety of overall organizational landscape. Use of comprehensive data and multidimensional indicators of publication profiles could inform and support strategic planning and monitoring of research performing organisations.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130413222","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Cibele Aguiar, S. Salles-Filho, Ana Carolina Spatti, Karen Esteves Fernandes Pinto, Sérgio Parreiras Pereira
{"title":"How can altmetrics improve the Public Communication of Science and Technology? An analysis on universities and altmetrics","authors":"Cibele Aguiar, S. Salles-Filho, Ana Carolina Spatti, Karen Esteves Fernandes Pinto, Sérgio Parreiras Pereira","doi":"10.55835/644315ea164c35f6489ab9e7","DOIUrl":"https://doi.org/10.55835/644315ea164c35f6489ab9e7","url":null,"abstract":"In current research evaluation models, monitoring and impact evaluation are extended beyond peer-reviewed articles to include Public Communication of Science and Technology activities. Through an online survey, we analyzed the perceptions of relevance and degree of application of the altmetric indicators for the PCST of 51 sampled Brazilian federal universities. Perceptions of relevance and application of altmetrics proved to be an outlier in 26 indicators. 66.7% of respondents said they did not know the relevance of altmetrics for the PCST or considered it not applicable to the field. Regarding the perception of relevance, the indicator “Mentions tracked by altmetrics” received high relevance scores (7 and 9) from 21.5% of respondents. The indicator was also the least applied, with only one university (1.9%) using it. In addition, 45% of respondents reported having no intention of applying it, 41.1% intend to apply it in the long term, and 11.7% in the short term.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127234644","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Broadening the conception of ‘what counts’ – example of a narrative CV in a university alliance","authors":"Maria Pietilä, J. Kekäle, K. Rintamäki","doi":"10.55835/644192caf38f9678c0feaff0","DOIUrl":"https://doi.org/10.55835/644192caf38f9678c0feaff0","url":null,"abstract":"Research(er) assessment is currently under reform. For example, the CoARA initiative aims at building critical mass to change assessment systems to lean more on qualitative assessment instead of metrics. There are also articulated needs to recognise the diversification of academic careers and contributions in academic work. For example, despite the aims towards Open Science, assessment systems rarely take researchers’ contributions in Open Science into account. This paper is about a practical approach, which aims at helping universities to broaden the ways researchers are assessed, for example for the purposes of academic recruitment. It presents a narrative CV template covering diverse aspects of academic work. The paper is based on the work conducted in YUFERING, a joint project by the European university alliance YUFE.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124418616","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
T. Thune, Magnus Gulbrandsen, Ingvild Reymert, S. Tellmann
{"title":"Exploring ministries’ use of research in policymaking - knowledge stock, circulation, or routines?","authors":"T. Thune, Magnus Gulbrandsen, Ingvild Reymert, S. Tellmann","doi":"10.55835/6442e1ea4e3daf2fc8306960","DOIUrl":"https://doi.org/10.55835/6442e1ea4e3daf2fc8306960","url":null,"abstract":"This paper addresses the use of scientific knowledge in policy work as an organization-level phenomenon and draws on the absorptive capacity concept. Through an empirical study of 14 Norwegian ministries, we seek to understand how organizational characteristics and processes that may play a role for whether and how policy organisations use scientific knowledge in policy making. We make use of the distinction between potential and realized absorptive capacity, and we investigate the following issues: First, we explore whether the knowledge stocks of a government organization influence the propensity to acquire research. Second, we are interested in understanding if knowledge circulation activities influence the propensity to use research-based knowledge in government organizations. Finally, we will look at whether having specific organizational routines in place influences both knowledge acquisition and use in government organizations. Methodologically, we make a qualitative comparative analysis (Ragin, 1987; Rihoux and Ragin, 2012) of 14 ministries, based on both qualitative and quantitative data, including survey data from 1325 ministerial civil servants.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"172 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133086270","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Apex Index: A Hit-based Metric for Measuring the Impact of Papers and Scientists","authors":"A. Yang, Qi Zhang, Hongya Wang, Sanhong Deng","doi":"10.55835/642af773d167ce5ddffa5e08","DOIUrl":"https://doi.org/10.55835/642af773d167ce5ddffa5e08","url":null,"abstract":"In this study, we introduce a novel metric called the Apex index, which is a hit-based measure for assessing the impact of scientific publications and scientists. The basic principle of the Apex index involves quantifying the number of hit papers that cite the focal paper. Specifically, we identify the top 1% most highly cited papers in a given field and year as the hit papers. We then calculate the Apex index for all publications in the MAG database, which contains approximately 200 million documents. Our study reveals that Nobel Prize-winning papers display a higher Apex index compared to other papers, and Nobel laureates exhibit a higher Apex index than their peers. Moreover, we demonstrate that the Apex index has a higher convergent validity in evaluating scientists and identifying laureates. Overall, the Apex index presents a valuable and effective tool for assessing the impact of scientific publications and researchers.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125972015","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Karen Esteves Fernandes Pinto, Sérgio Robles Reis de Queiroz, Bruno Brandão Fischer, Yohanna Juk, Vanessa Avanci
{"title":"An Equity, Diversity, and Inclusion Approach in Entrepreneurial and Innovation Ecosystems","authors":"Karen Esteves Fernandes Pinto, Sérgio Robles Reis de Queiroz, Bruno Brandão Fischer, Yohanna Juk, Vanessa Avanci","doi":"10.55835/6442ffbec93d17c257de1fff","DOIUrl":"https://doi.org/10.55835/6442ffbec93d17c257de1fff","url":null,"abstract":"In this paper, we intend to verify if more inclusive spin-offs tend to obtain better performance or access to funding sources. Thus, we have analyzed the spin-off companies of the State University of Campinas (Unicamp) in Brazil. The Inova Agency, i.e., the technology transfer office of Unicamp, provided the data concerning all university spin-offs. The outcomes indicate an unbalanced sample in terms of gender, and the same is expected regarding race and socioeconomic profile. We hope this study may serve as a basis for policymakers to better direct efforts to create and base policies that strengthen inclusion in scientific activities, especially those related to university-related entrepreneurial activities.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117085158","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}