{"title":"Journal metrics as predictors of Research Excellence Framework 2021 results: Comparison of impact factor quartiles and Finnish expert-ratings","authors":"Janne Pölönen, Raf Guns, Tim C. E. Engels","doi":"10.55835/643e529c0b149e8673ee2d95","DOIUrl":"https://doi.org/10.55835/643e529c0b149e8673ee2d95","url":null,"abstract":"This study compares citation-based and expert-based journal metrics as predictors of peer-assessed research quality based on 154,826 journal articles submitted to UK’s Research Excellence Framework (REF) 2021. The Finnish expert-based Julkaisufoorumi (JUFO) level ratings of journals determined by expert-panels per field produce scores that correlate more strongly with REF scores than those based on citation-based Journal Impact Factor (JIF) or Journal Citation Indicator (JCI) Quartiles. This holds true at aggregate levels of 34 Subject areas, 157 Higher Education Institutions (HEI), and 1,888 Units of Assessment (UoA). Especially non-field-normalised JIF-based scores correlate poorly with REF scores. All types of journal metrics are more aligned with expert-based REF scores at the highest aggregate level of HEIs and agree less at the lower aggregate level of UoAs and Subject areas.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129713620","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Guideline Impact Factor – A new indicator to assess journals cited in medical guidelines","authors":"V. Aman, Nikita Sorgatz","doi":"10.55835/644169a0565e92f0541abf8d","DOIUrl":"https://doi.org/10.55835/644169a0565e92f0541abf8d","url":null,"abstract":"Despite of its many limitations, the Journal Impact Factor (JIF) is widely used to evaluate research institutions and individual researchers. Using references from 41 German medical guidelines we show that clinical relevance as assessed by guideline authors is uncorrelated to the JIF suggesting that a journal’s clinical relevance is independent of its JIF. As a consequence, evaluations solely relying on the JIF end up under-valuing clinically important research. We therefore propose a Guideline Impact Factor (GLIF) quantifying the relevance of journals for medical guideline development as an independent quality criterion for journals.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"43 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131331144","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"How networked are Medical Schools? Evidence from Portugal","authors":"Lígia Ernesto, B. Damásio, S. Mendonça","doi":"10.55835/644311248fad6804d7e6b0d6","DOIUrl":"https://doi.org/10.55835/644311248fad6804d7e6b0d6","url":null,"abstract":"Institutional collaboration between universities and other actors is crucial to generate new knowledge and for advancing innovation. But, how important is this for the healthcare sector? This work analyses 441 institutional collaborations between Portuguese Medical Schools and other entities (pharmaceutical industry, funding organisations, hospitals, other universities, non-profit organisations, other private for-profit organisations, public bodies and public research organisations). We identify, validate, disambiguate, classify and analyse evidence available from a variety of sources. Our original database reveals that most of the partnerships of Portuguese Medical Schools are with academic institutions. A sectoral failure regarding partnerships with other type of actors (e.g. industry, other research organisations) is suggested. As for future policy objectives, we argue that a systems building view could be considered.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125357433","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Assessment of academic work in AI using Bibliometrics: The Case of Lund University","authors":"Vinicius Muraro, Devrim Göktepe-Hultén","doi":"10.55835/6442a2612e5bbccb9a78fe1e","DOIUrl":"https://doi.org/10.55835/6442a2612e5bbccb9a78fe1e","url":null,"abstract":"Artificial Intelligence (AI) is considered as “General Purpose Technology” that could have large impact effects in several sectors and cause disruptive changes in societies. Universities are often seen as the seedbeds of new knowledge and scientific research, as a point of departure, by using bibliometric approach we aim to assess the scientific output for AI, to understand the characteristics and potential of AI research at Lund University. The preliminary findings showed the diverse partnerships of LU scientists with other universities, companies, research institutes, and hospitals, funding agencies, and international partnerships. Our study aims to establish the groundwork for future assessments of AI research in a sustainable and transparent manner. We plan to identify both traditional scientific inputs and outputs, to unpack the unknown dimensions of academic work at universities, with a more holistic approach to assess academic work and pave the way for future studies.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"237 ","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114096284","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Field Effects in Predicting Exceptional Growth in Research Communities","authors":"R. Klavans, K. Boyack, Caleb Smith","doi":"10.55835/643f1aa90f649f6042841876","DOIUrl":"https://doi.org/10.55835/643f1aa90f649f6042841876","url":null,"abstract":"Using a model of the literature indexed in Scopus, we have increased the accuracy of our ability to predict which of 20,747 research communities would achieve exceptional growth from 32.2 to 39.6 using double exponential smoothing of inertial indicators and by doing predictions in each of 26 fields rather than across the entire model. Each field nominated two (out of a possible 123) indicators as ‘best predictors’ following the procedure described in our previous studies. Significant diversity was found in which indicators performed best in each field, suggesting that field effects should be accounted for in predictive analytics. Nevertheless, there were groupings of contiguous fields with a surprising level of homogeneity in predictive indicators. Possible reasons for the similarities and differences are discussed.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114164512","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Internationalisation of Russian medical research – its main drivers and future prospects through the lens of publications","authors":"E. Dyachenko, Iurii Agafonov, K. Guba","doi":"10.55835/6442ffb78264b1bf681c4891","DOIUrl":"https://doi.org/10.55835/6442ffb78264b1bf681c4891","url":null,"abstract":"Decades after the fall of the Iron Curtain, medical research in Russia is poorly integrated into global science. In this study we analyze how the presence of Russian medical research in international journals have changed in recent years (2010-2020). We collected the data from different sources – Web of Science, Scopus, Medline, Russian Science Citation Index. Although the articles in international journals still make up a smaller part of all Russian medical publications, it has grown significantly in recent years. Among these publications, articles in high impact journals account for about a third of the total flow. International cooperation is one of the main drivers of top-level Russian medical publications. Considering how many ties and collaborations with foreign scientists have been cut or suspended recently, we can assume a significant reduction in Russia's presence in core medical journals in the nearest future.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130094256","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
S. Ismail, Alain Mermoud, L. Maréchal, Samuel Orso, Dimitri Percia David
{"title":"Capturing Trends Using OpenAlex and Wikipedia Page Views as Science Indicators: The Case of Data Protection and Encryption Technologies","authors":"S. Ismail, Alain Mermoud, L. Maréchal, Samuel Orso, Dimitri Percia David","doi":"10.55835/6436bfc7353eb8e707e4d5df","DOIUrl":"https://doi.org/10.55835/6436bfc7353eb8e707e4d5df","url":null,"abstract":"This paper presents a novel science indicator to identify, analyze, and capture technology trends based on Wikipedia page views and OpenAlex presented at STI2022. Our webometric methodology is grounded in open science practices and applied to crowd-sourced, open, and free data. We explore the relationships between 36 data protection and encryption technologies, by measuring and classifying their time-varying attention. These highly research-intensive technologies are particularly suitable to illustrate our approach. We first find that Blockchain, Hash Function, and Asymmetric Encryption are the technologies that generate significant public interest. Conversely, niche or longstanding technologies such as Disk Encryption and Email Encryption are considered low-interest technologies with no growth. Our findings suggest that monitoring public attention on Wikipedia can serve as a scientific indicator to provide valuable information on technology trends and inform decision-making related to investment, assessment, and technology road-mapping.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"232 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134449595","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Heterogenous treatment effects of incorporating public innovation intermediaries: Evidence from extension, research, and inventive activities","authors":"N. Fukugawa","doi":"10.55835/643f7d8920c713bf1901cb5e","DOIUrl":"https://doi.org/10.55835/643f7d8920c713bf1901cb5e","url":null,"abstract":"The incorporation of public organizations is implemented by the central and local governments to improve their efficiency, which should enhance their contributions to national and local economies. In Japan, a series of administrative reforms were implemented at the national and local governments since the late 1990s, some of which took the form of the incorporation. Taking an example of the incorporation of technology extension service (TES) providers established by local governments, this study evaluates the average treatment effects on the treated (ATT) by applying the difference-in-differences (DID) model to panel data of TES providers for small and medium-sized enterprises (SMEs), Kohsetsushi. Unlike the uniform and simultaneous incorporation of national universities, the incorporation of Kohsetsushi is at the discretion of local governments and the timings of incorporation vary. Applying the conventional two-way fixed effects (TWFE) model to panel data with staggered treatments may yield biased ATTs. Following Callaway and Sant’Anna (2021), this study adopted their DID model (CS-DID) to avoid forbidden comparisons between late and early treated units. The ATTs in terms of extension activities are heterogenous and significantly positive for TWFE-DID but statistically insignificant for CS-DID. The sources of heterogenous treatment effects seem to be decline in innovation agglomerations that made localized spillover insignificant and unobservable organizational capabilities of local governments to exploit the incentive system reform for geographically broader spillover.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125041631","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Investigating the influence of AI research topics in the academic, public, and industry spheres","authors":"G. Berman, Kate Williams, Sandra Michalska","doi":"10.55835/6442070e78340aab60459654","DOIUrl":"https://doi.org/10.55835/6442070e78340aab60459654","url":null,"abstract":"The Artificial Intelligence research field sits at the intersection of several overlapping spheres (academia, industry, media), each with their own logics and commitments. The influence of research within these worlds is studied through a number of bibliometric methods, including citation metrics for measuring influence within academia, and counts of patents and news-media mentions for influence in industry and the media. Using a large-scale, publicly-available dataset of research outputs, we compare the topical content of outputs that are highly influential in each of these worlds. We identify significant differences between the content of influential research in these worlds, indicating that the academic, industry and media worlds value different aspects of the Artificial Intelligence field. These differences provide new insights on the evaluation of research produced within the Artificial Intelligence field.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"9 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128439947","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Exploratory analysis of policy document sources in Altmetric.com and Overton","authors":"Biegzat Murat, E. Noyons, R. Costas","doi":"10.55835/6442b915bdab695b3f03d666","DOIUrl":"https://doi.org/10.55835/6442b915bdab695b3f03d666","url":null,"abstract":"Policy documents are one of the altmetric sources most crucial for comprehending the interaction between science and policy. In policy documents, the source is usually the institution or organization that published the given policy document. In this study we compare the policy document sources indexed by Altmetric.com and Overton. Altmetric.com is an altmetric aggregator that has been around for over a decade. Overton is a newer database aiming at being the most complete collection of international policy documents. Our findings reveal that Overton covered more policy organizations than Altmetric.com, although the overlap in sources is quite small. This low overlap may suggest that both data aggregators may be using slightly different operationalizations of their notion of “policy documents”, which calls for more transparency on what notion of policy documents and policy organizations are considered by them. Regarding policy organizations indexed by both data aggregators, most policy documents are published by government and non-profit organizations. Future research should delve more into the different policy organizations that are being covered in Overton and Altmetric.com, as well as their geographical distribution.","PeriodicalId":334841,"journal":{"name":"27th International Conference on Science, Technology and Innovation Indicators (STI 2023)","volume":"130 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126902748","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}