Rating accuracy, leniency, and rater perceptions when using the RPM and BARS

Abstract

Researchers have argued that social-comparative rating formats hold important psychometric advantages over traditional absolute ratings. We asked 152 participants to observe and assess the videotaped performance of individuals completing a task using a social-comparative (Relative Percentile Method; RPM) and absolute rating (Behaviorally Anchored Rating Scale; BARS) formats. After collecting expert ratings on the same set of videos, we calculated accuracy indices and leniency. Additionally, we collected rater perceptions of accuracy and fairness for both formats. Our study revealed that the BARS was perceived as more accurate and fairer than the RPM. However, the RPM was found to be better at combating rater leniency. We discuss the implications of these findings.