Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001

Abstract

The question of how best to assess research performance is clearly of great concern. In December 2007, HEFCE launched a national consultation on the future of Research Assessment, proposing that a Research Excellence Framework replaces the current Research Assessment Exercise. Fundamentally the Research Excellence Framework involves a shift to metrics. Views on the effectiveness of metrics for assessing research are mixed, so in this report we seek to explore empirically the question of whether metrics based on citation counts are strongly correlated with peer review assessments of research quality. We use data from RAE 2001, covering all departments and all universities in the UK. At a more disaggregated level the data used in this study includes: [i] the individual submissions made to RAE 2001 - a database of 203,743 research output records, one for each output submitted in RAE 2001; [ii] the citation counts for each of the submitted items when they are journal articles (141,789 of the original 203,743 items). We have interrogated the ISA Web of Science and produced citation counts for every article included in RAE 2001 where data is available. This gives us citation counts for 112,201 publications (55.1% of the original 203,743 submitted items); [iii] the actual RAE 2001 scores produced and published by HEFCE. We contrast these scores - which are based on peer review - with those that would have been produced had bibliometrics, based on citation counts, been used in RAE 2001. The findings of this analysis are: [i] citation counts are a reasonable proxy for peer assessment in some subjects, such as Biological Sciences, Clinical Sciences, Chemistry and Psychology; [ii] however, citation counts are a weak proxy for a large number of disciplines, including fields within Biomedical related subjects and Engineering-related subjects. This is despite the fact that these subjects have good coverage in the Web of Science, in terms of the outputs submitted to RAE 2001; [iii] as others have reported, the Web of Science offers only partial coverage in some subjects, and hence the use of bibliometrics becomes increasingly less valuable as we move from Biomedical, Physical Sciences and Engineering, to Socials Sciences, Literature and Arts and Humanities.