Uncertainty about Rater Variance and Small Dimension Effects Impact Reliability in Supervisor Ratings

IF 2.9 4区 心理学 Q2 PSYCHOLOGY, APPLIED
D. Jackson, G. Michaelides, Christopher Dewberry, Amanda Jones, S. Toms, Benjamin Schwencke, Wei-Ning Yang
{"title":"Uncertainty about Rater Variance and Small Dimension Effects Impact Reliability in Supervisor Ratings","authors":"D. Jackson, G. Michaelides, Christopher Dewberry, Amanda Jones, S. Toms, Benjamin Schwencke, Wei-Ning Yang","doi":"10.1080/08959285.2022.2111433","DOIUrl":null,"url":null,"abstract":"ABSTRACT We modeled the effects commonly described as defining the measurement structure of supervisor performance ratings. In doing so, we contribute to different theoretical perspectives, including components of the multifactor and mediated models of performance ratings. Across two reanalyzed samples (Sample 1, N ratees = 392, N raters = 244; Sample 2, N ratees = 342, N raters = 397), we found a structure primarily reflective of general (>27% of variance explained) and rater-related (>49%) effects, with relatively small performance dimension effects (between 1% and 11%). We drew on findings from the assessment center literature to approximate the proportion of rater variance that might theoretically contribute to reliability in performance ratings. We found that even moderate contributions of rater-related variance to reliability resulted in a sizable impact on reliability estimates, drawing them closer to accepted criteria.","PeriodicalId":47825,"journal":{"name":"Human Performance","volume":"35 1","pages":"278 - 301"},"PeriodicalIF":2.9000,"publicationDate":"2022-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Human Performance","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1080/08959285.2022.2111433","RegionNum":4,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"PSYCHOLOGY, APPLIED","Score":null,"Total":0}
引用次数: 0

Abstract

ABSTRACT We modeled the effects commonly described as defining the measurement structure of supervisor performance ratings. In doing so, we contribute to different theoretical perspectives, including components of the multifactor and mediated models of performance ratings. Across two reanalyzed samples (Sample 1, N ratees = 392, N raters = 244; Sample 2, N ratees = 342, N raters = 397), we found a structure primarily reflective of general (>27% of variance explained) and rater-related (>49%) effects, with relatively small performance dimension effects (between 1% and 11%). We drew on findings from the assessment center literature to approximate the proportion of rater variance that might theoretically contribute to reliability in performance ratings. We found that even moderate contributions of rater-related variance to reliability resulted in a sizable impact on reliability estimates, drawing them closer to accepted criteria.
评级方差的不确定性和小尺寸效应对监理评级可靠性的影响
摘要:我们对通常被描述为定义主管绩效评级的衡量结构的影响进行了建模。在这样做的过程中,我们对不同的理论观点做出了贡献,包括绩效评级的多因素和中介模型的组成部分。在两个重新分析的样本中(样本1,N个评分者=392,N个评价者=244;样本2,N个评级者=342,N个评等者=397),我们发现一个结构主要反映了一般(解释的方差的27%以上)和评分者相关(>49%)的影响,表现维度的影响相对较小(在1%到11%之间)。我们利用评估中心文献中的发现来近似评分者方差的比例,这在理论上可能有助于绩效评分的可靠性。我们发现,即使评分者相关方差对可靠性的适度贡献也会对可靠性估计产生相当大的影响,使其更接近公认的标准。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Human Performance
Human Performance PSYCHOLOGY, APPLIED-
CiteScore
4.30
自引率
0.00%
发文量
16
期刊介绍: Human Performance publishes research investigating the nature and role of performance in the workplace and in organizational settings and offers a rich variety of information going beyond the study of traditional job behavior. Dedicated to presenting original research, theory, and measurement methods, the journal investigates individual, team, and firm level performance factors that influence work and organizational effectiveness. Human Performance is a respected forum for behavioral scientists interested in variables that motivate and promote high-level human performance, particularly in organizational and occupational settings. The journal seeks to identify and stimulate relevant research, communication, and theory concerning human capabilities and effectiveness. It serves as a valuable intellectual link between such disciplines as industrial-organizational psychology, individual differences, work physiology, organizational behavior, human resource management, and human factors.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信