Organizational Research Methods最新文献_第8页

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing 从叙述性评论中评分维度水平的工作表现:使用自然语言处理时的有效性和概括性

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-07-01 DOI: 10.1177/1094428120930815

Andrew B. Speer

{"title":"Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing","authors":"Andrew B. Speer","doi":"10.1177/1094428120930815","DOIUrl":"https://doi.org/10.1177/1094428120930815","url":null,"abstract":"Performance appraisal narratives are qualitative descriptions of employee job performance. This data source has seen increased research attention due to the ability to efficiently derive insights using natural language processing (NLP). The current study details the development of NLP scoring for performance dimensions from narrative text and then investigates validity and generalizability evidence for those scores. Specifically, narrative valence scores were created to measure a priori performance dimensions. These scores were derived using bag of words and word embedding features and then modeled using modern prediction algorithms. Construct validity evidence was investigated across three samples, revealing that the scores converged with independent human ratings of the text, aligned numerical performance ratings made during the appraisal, and demonstrated some degree of discriminant validity. However, construct validity evidence differed based on which NLP algorithm was used to derive scores. In addition, valence scores generalized to both downward and upward rating contexts. Finally, the performance valence algorithms generalized better in contexts where the same qualitative survey design was used compared with contexts where different instructions were given to elicit narrative text.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"572 - 594"},"PeriodicalIF":9.5,"publicationDate":"2021-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428120930815","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42014110","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 11

Corrigendum to From Nuisance to Novel Research Questions: Using Multilevel Models to Predict Heterogeneous Variances 从麻烦到新颖的研究问题:使用多水平模型预测异质方差的勘误表

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-07-01 DOI: 10.1177/10944281211011778

引用次数: 0

Planned Missingness: How to and How Much? 计划导弹：如何发射，发射多少？

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-05-28 DOI: 10.1177/10944281211016534

Charlene Zhang, Martin C. Yu

引用次数: 5

A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique 基于策略捕获技术的测试-复验可靠性综合评判元分析

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-05-12 DOI: 10.1177/10944281211011529

Ze Zhu, Alan J. Tomassetti, R. Dalal, Shannon W. Schrader, Kevin Loo, Isaac E. Sabat, Balca Alaybek, You Zhou, Chelsea Jones, Shea Fyffe

{"title":"A Test-Retest Reliability Generalization Meta-Analysis of Judgments Via the Policy-Capturing Technique","authors":"Ze Zhu, Alan J. Tomassetti, R. Dalal, Shannon W. Schrader, Kevin Loo, Isaac E. Sabat, Balca Alaybek, You Zhou, Chelsea Jones, Shea Fyffe","doi":"10.1177/10944281211011529","DOIUrl":"https://doi.org/10.1177/10944281211011529","url":null,"abstract":"Policy capturing is a widely used technique, but the temporal stability of policy-capturing judgments has long been a cause for concern. This article emphasizes the importance of reporting reliability, and in particular test-retest reliability, estimates in policy-capturing studies. We found that only 164 of 955 policy-capturing studies (i.e., 17.17%) reported a test-retest reliability estimate. We then conducted a reliability generalization meta-analysis on policy-capturing studies that did report test-retest reliability estimates—and we obtained an average reliability estimate of .78. We additionally examined 16 potential methodological and substantive antecedents to test-retest reliability (equivalent to moderators in validity generalization studies). We found that test-retest reliability was robust to variation in 14 of the 16 factors examined but that reliability was higher in paper-and-pencil studies than in web-based studies and was higher for behavioral intention judgments than for other (e.g., attitudinal and perceptual) judgments. We provide an agenda for future research. Finally, we provide several best-practice recommendations for researchers (and journal reviewers) with regard to (a) reporting test-retest reliability, (b) designing policy-capturing studies for appropriate reportage, and (c) properly interpreting test-retest reliability in policy-capturing studies.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"541 - 574"},"PeriodicalIF":9.5,"publicationDate":"2021-05-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/10944281211011529","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44952843","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 2

Systematicity in Organizational Research Literature Reviews: A Framework and Assessment 组织研究文献综述的系统性：框架与评价

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-05-04 DOI: 10.1177/10944281211008652

Zeki Simsek, B. Fox, Ciaran Heavey

引用次数: 30

New Network Models for the Analysis of Social Contagion in Organizations: An Introduction to Autologistic Actor Attribute Models 组织中社会传染分析的新网络模型:自行为者属性模型导论

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-04-20 DOI: 10.1177/10944281211005167

Andrew Parker, F. Pallotti, A. Lomi

{"title":"New Network Models for the Analysis of Social Contagion in Organizations: An Introduction to Autologistic Actor Attribute Models","authors":"Andrew Parker, F. Pallotti, A. Lomi","doi":"10.1177/10944281211005167","DOIUrl":"https://doi.org/10.1177/10944281211005167","url":null,"abstract":"Autologistic actor attribute models (ALAAMs) provide new analytical opportunities to advance research on how individual attitudes, cognitions, behaviors, and outcomes diffuse through networks of social relations in which individuals in organizations are embedded. ALAAMs add to available statistical models of social contagion the possibility of formulating and testing competing hypotheses about the specific mechanisms that shape patterns of adoption/diffusion. The main objective of this article is to provide an introduction and a guide to the specification, estimation, interpretation and evaluation of ALAAMs. Using original data, we demonstrate the value of ALAAMs in an analysis of academic performance and social networks in a class of graduate management students. We find evidence that both high and low performance are contagious, that is, diffuse through social contact. However, the contagion mechanisms that contribute to the diffusion of high performance and low performance differ subtly and systematically. Our results help us identify new questions that ALAAMs allow us to ask, new answers they may be able to provide, and the constraints that need to be relaxed to facilitate their more general adoption in organizational research.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"513 - 540"},"PeriodicalIF":9.5,"publicationDate":"2021-04-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/10944281211005167","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41724651","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 9

Temporal Brokering: A Measure of Brokerage as a Behavioral Process 时间中介：作为一种行为过程的中介度量

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-04-15 DOI: 10.1177/10944281211002911

E. Quintane, M. Wood, John Dunn, L. Falzon

{"title":"Temporal Brokering: A Measure of Brokerage as a Behavioral Process","authors":"E. Quintane, M. Wood, John Dunn, L. Falzon","doi":"10.1177/10944281211002911","DOIUrl":"https://doi.org/10.1177/10944281211002911","url":null,"abstract":"Extant research in organizational networks has provided critical insights into understanding the benefits of occupying a brokerage position. More recently, researchers have moved beyond the brokerage position to consider the brokering processes (arbitration and collaboration) brokers engage in and their implications for performance. However, brokering processes are typically measured using scales that reflect individuals’ orientation toward engaging in a behavior, rather than the behavior itself. In this article, we propose a measure that captures the behavioral process of brokering. The measure indicates the extent to which actors engage in arbitration versus collaboration based on sequences of time stamped relational events, such as emails, message boards, and recordings of meetings. We demonstrate the validity of our measure as well as its predictive ability. By leveraging the temporal information inherent in sequences of relational events, our behavioral measure of brokering creates opportunities for researchers to explore the dynamics of brokerage and their impact on individuals, and also paves the way for a systematic examination of the temporal dynamics of networks.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"459 - 489"},"PeriodicalIF":9.5,"publicationDate":"2021-04-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/10944281211002911","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47591297","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Faking Detection Improved: Adopting a Likert Item Response Process Tree Model 改进伪造检测:采用李克特项目反应过程树模型

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-04-15 DOI: 10.1177/10944281211002904

Tianjun Sun, Bo Zhang, Mengyang Cao, F. Drasgow

{"title":"Faking Detection Improved: Adopting a Likert Item Response Process Tree Model","authors":"Tianjun Sun, Bo Zhang, Mengyang Cao, F. Drasgow","doi":"10.1177/10944281211002904","DOIUrl":"https://doi.org/10.1177/10944281211002904","url":null,"abstract":"With the increasing popularity of noncognitive inventories in personnel selection, organizations typically wish to be able to tell when a job applicant purposefully manufactures a favorable impression. Past faking research has primarily focused on how to reduce faking via instrument design, warnings, and statistical corrections for faking. This article took a new approach by examining the effects of faking (experimentally manipulated and contextually driven) on response processes. We modified a recently introduced item response theory tree modeling procedure, the three-process model, to identify faking in two studies. Study 1 examined self-reported vocational interest assessment responses using an induced faking experimental design. Study 2 examined self-reported personality assessment responses when some people were in a high-stakes situation (i.e., selection). Across the two studies, individuals instructed or expected to fake were found to engage in more extreme responding. By identifying the underlying differences between fakers and honest respondents, the new approach improves our understanding of faking. Percentage cutoffs based on extreme responding produced a faker classification precision of 85% on average.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"490 - 512"},"PeriodicalIF":9.5,"publicationDate":"2021-04-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/10944281211002904","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45621939","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 13

Corrigendum to On Ignoring the Random Effects Assumption in Multilevel Models: Review, Critique, and Recommendations 关于忽略多层次模型中随机效应假设的更正：综述、批评和建议

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-04-01 DOI: 10.1177/10944281211002293

引用次数: 1

Meta-Analyses as a Multi-Level Model 元分析作为一个多层次模型

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2021-04-01 DOI: 10.1177/1094428119857471

Janaki Gooty, G. Banks, Andrew C. Loignon, Scott Tonidandel, Courtney E. Williams

{"title":"Meta-Analyses as a Multi-Level Model","authors":"Janaki Gooty, G. Banks, Andrew C. Loignon, Scott Tonidandel, Courtney E. Williams","doi":"10.1177/1094428119857471","DOIUrl":"https://doi.org/10.1177/1094428119857471","url":null,"abstract":"Meta-analyses are well known and widely implemented in almost every domain of research in management as well as the social, medical, and behavioral sciences. While this technique is useful for determining validity coefficients (i.e., effect sizes), meta-analyses are predicated on the assumption of independence of primary effect sizes, which might be routinely violated in the organizational sciences. Here, we discuss the implications of violating the independence assumption and demonstrate how meta-analysis could be cast as a multilevel, variance known (Vknown) model to account for such dependency in primary studies’ effect sizes. We illustrate such techniques for meta-analytic data via the HLM 7.0 software as it remains the most widely used multilevel analyses software in management. In so doing, we draw on examples in educational psychology (where such techniques were first developed), organizational sciences, and a Monte Carlo simulation (Appendix). We conclude with a discussion of implications, caveats, and future extensions. Our Appendix details features of a newly developed application that is free (based on R), user-friendly, and provides an alternative to the HLM program.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"389 - 411"},"PeriodicalIF":9.5,"publicationDate":"2021-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428119857471","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47635585","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 6