Defining, Measuring, and Rewarding Scholarly Impact: Mind the Level of Analysis

Abstract

We address the grossly incorrect inferences that result from using journal impact factor (JIF) as a proxy to assess individual researcher and article scholarly impact. This invalid practice occurs because of confusion about the definition and measurement of impact at different levels of analysis. Specifically, JIF is a journal-level measure of impact, computed by aggregating citations of individual articles (i.e., upward effect), and is therefore inappropriatewhenmeasuring impact at lower levels of analysis, such as that of individual researchers, or of individual articles published in a particular journal (i.e., downward effect). We illustrate the severity of the errors that occurwhen using JIF to evaluate individual scholarly impact, and advocate for an immediate moratorium on the exclusive use of JIF and other journal-level (i.e., higher level of analysis) measures when assessing the impact of individual researchers and individual articles (i.e., lower level of analysis). Given the importance and interest in assessing the scholarly impact of researchers and articles, we delineate level-appropriate and readily availablemeasures.