How similar are field-normalized scores from different free or commercial databases calculated for large German universities?

Abstract

We compared field-normalized citation scores from the freely available bibliographic database OpenAlex with those from three commercial databases (Web of Science, Scopus, and Dimensions). The scores have been calculated for nearly 335,000 publications published by 48 German universities in four OECD subject areas between 2013 and 2017. We found varying but - overall - strong agreement according to Lin's concordance coefficient. We also calculated (aggregated) mean normalized citation scores for the 48 universities and found that the agreements across different databases are low. On the one hand, the results suggest that comparisons of universities using field-normalized citation scores across different databases should be avoided. On the other hand, the difference of the concordance coefficients on the paper and university level is a good example for the problem of ecological fallacy in bibliometrics: The mean impact is not representative for the single papers’ impact in the set.