Finding “similar” universities using ChatGPT for institutional benchmarking: A large-scale comparison of European universities

Abstract

The study objective was to evaluate the efficacy of ChatGPT in identifying “similar” institutions for benchmarking the research performance of a university. Benchmarking is deemed a promising approach to compare “similar with similar” as a better alternative to rankings (comparing “different” universities). Current approaches either focus on a limited number of “quantitative” dimensions or are too complex for most users. We conducted large-scale testing by tasking ChatGPT with identifying the most similar European universities in terms of research performance, utilizing the European Tertiary Education Register data. We tested whether the peers suggested by ChatGPT were similar to the focal university on size, research intensity, and subject composition. Additionally, we evaluated whether providing more specific instructions improved the results. The findings offer a nuanced perspective on the potential and risks of using ChatGPT to identify peer institutions for benchmarking. On one hand, solely using ChatGPT would replicate the visibility biases associated with university rankings, thereby undermining the rationale for benchmarking. On the other hand, relying on semantic associations might capture dimensions of university similarity that are relevant and difficult to capture through quantitative methods. We finally reflected on the broader implications for scholars in higher education and science studies research.