Organizational Research Methods最新文献

Can Generative Large Language Models Serve as Raters for Test Development? A Systematic Evaluation Across Tasks, Models, and Inference Configurations 生成式大型语言模型可以作为测试开发的评分器吗？跨任务、模型和推理配置的系统评估

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-09-02 DOI: 10.1177/10944281261475596

Mina Son, Philseok Lee

{"title":"Can Generative Large Language Models Serve as Raters for Test Development? A Systematic Evaluation Across Tasks, Models, and Inference Configurations","authors":"Mina Son, Philseok Lee","doi":"10.1177/10944281261475596","DOIUrl":"https://doi.org/10.1177/10944281261475596","url":null,"abstract":"The present study investigates the effectiveness of generative large language models (LLMs) as raters across three common rating tasks: (a) social desirability ratings, (b) content validity ratings, and (c) trait importance ratings. Specifically, we examine reliability and validity of LLM-generated ratings across varying occupational contexts, rating methods, LLM families (i.e., GPT-4, GPT-5, and Sonnet 4.5), and inference configurations (i.e., prompt design and temperature settings). Results indicate that LLM ratings exhibit strong reliability and convergent validity in social desirability ratings across occupational contexts, as well as acceptable convergence with human ratings in Likert-type content validity evaluations. In contrast, reliability and convergent validity for trait importance ratings were inconsistent across occupational contexts. Variations in prompt design and temperature settings generally produced small to negligible effects on reliability and validity. Overall, the findings suggest that LLMs can function as effective supplementary raters in test development and validation processes, although greater caution is warranted for certain rating tasks. Practical implications and directions for future research are discussed.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"16 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-09-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148877302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Interaction Effects May Indeed Be Artifacts of Common Method Variance 相互作用的影响可能确实是共同方法差异的产物

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-07-23 DOI: 10.1177/10944281261465310

Arturs Kalnins

{"title":"Interaction Effects May Indeed Be Artifacts of Common Method Variance","authors":"Arturs Kalnins","doi":"10.1177/10944281261465310","DOIUrl":"https://doi.org/10.1177/10944281261465310","url":null,"abstract":"A common belief in the organizational sciences is that estimated interaction effects in ordinary least squares regression cannot be artifacts of common method variance (CMV). This belief rests on the claim that CMV universally attenuates estimated interaction effects. As a result, researchers frequently dismiss CMV concerns when testing moderated relationships. We present an analytic closed-form regression model demonstrating that this universal attenuation claim is false in commonplace scenarios. In particular, when a quadratic term legitimately affects the dependent variable (DV) in the presence of CMV, an estimated interaction effect may be purely artifactual. The common belief holds only in two special cases: (1) when quadratic terms have zero effect on the DV, or (2) when one primary term that enters the interaction is unambiguously unaffected by CMV. We conclude that researchers should apply all standard CMV precautions when testing moderated hypotheses; the fact that one is assessing interactions should not be viewed as a “get out of CMV jail free” card. In addition, we recommend estimating and reporting model specifications both with and without quadratic terms to assess robustness.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"33 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-07-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148552785","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

A Multimodal Item Response Modeling for Personality Assessment in Organizational Research 组织研究中人格评估的多模态项目反应模型

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-07-01 DOI: 10.1177/10944281261457337

Dongbo Tu, Fumei Zhang, Siwei Peng, Daxun Wang, Yan Cai

{"title":"A Multimodal Item Response Modeling for Personality Assessment in Organizational Research","authors":"Dongbo Tu, Fumei Zhang, Siwei Peng, Daxun Wang, Yan Cai","doi":"10.1177/10944281261457337","DOIUrl":"https://doi.org/10.1177/10944281261457337","url":null,"abstract":"Recent advances in process data collection have made it possible to efficiently collect multimodal behavioral indicators, such as response times and eye-tracking measures. These multimodal data have been widely applied in cognitive and achievement assessments, where they have improved the accuracy of latent construct estimation. However, the use of informative multimodal process data in noncognitive assessments, such as personality measures widely used in organizational research, has received considerably less attention. To address this gap, we integrate response time and eye-tracking data into a conventional item response model to capture respondents’ response processes, thereby improving differentiation across trait levels and enhancing noncognitive assessment. Simulation studies were conducted to evaluate the performance of the proposed model and compare it with a conventional IRT model. Results indicate that model parameters can be accurately recovered and that incorporating multimodal data significantly improves the accuracy of person latent trait estimates. Finally, an empirical analysis was conducted to demonstrate the applicability and advantages of the proposed model in personality assessment.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"269 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148355552","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Argument Mining for Organizational Research: A Computer-Aided Analysis of Organizational Talk 组织研究的论据挖掘：组织谈话的计算机辅助分析

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-07-01 DOI: 10.1177/10944281261453948

Cornelia Fedtke, Gregor Wiedemann, Cristina Besio

{"title":"Argument Mining for Organizational Research: A Computer-Aided Analysis of Organizational Talk","authors":"Cornelia Fedtke, Gregor Wiedemann, Cristina Besio","doi":"10.1177/10944281261453948","DOIUrl":"https://doi.org/10.1177/10944281261453948","url":null,"abstract":"Argument mining—the automatic identification, classification, and linking of argumentative text—has been studied in natural language processing (NLP) for more than a decade. Despite its claimed potential for applications in legal, political, and social contexts, it remained largely unexplored in organizational research. This article introduces aspect-based argument mining (ABAM) as a methodical innovation for studying how organizations justify decisions, construct legitimacy, and relate to their environments through communicative acts. By scaling up the analysis of argumentative structures beyond the limits of small-scale, qualitative studies, ABAM enables the recognition and systematic analysis of argumentation patterns in large text corpora that were hardly detectable with previous (computational) approaches. The potential is demonstrated by a longitudinal case study of Twitter debates on nuclear energy in Germany, revealing how shifting societal values—particularly the reframing of nuclear energy from a safety to a climate issue—produced growing misalignments between organizational talk of a political party organization and its social media environment.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"13 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148355605","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Rhythms of Organizational Life: Rhythmanalysis as an Organizational Research Method 组织生活的节奏：节奏分析作为一种组织研究方法

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-06-15 DOI: 10.1177/10944281261443052

Albane Grandazzi, Gazi Islam

引用次数: 0

Assessing Partial Measurement Invariance in Cross-Group, Longitudinal, Congruence, and Multilevel Organizational Studies: Introducing the MEI Package in R 评估跨群体、纵向、同余和多层次组织研究的部分测量不变性：引入R中的MEI包

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-05-20 DOI: 10.1177/10944281261449198

Gordon W. Cheung, Changya Hu, Elena Zubielevitch

{"title":"Assessing Partial Measurement Invariance in Cross-Group, Longitudinal, Congruence, and Multilevel Organizational Studies: Introducing the MEI Package in R","authors":"Gordon W. Cheung, Changya Hu, Elena Zubielevitch","doi":"10.1177/10944281261449198","DOIUrl":"https://doi.org/10.1177/10944281261449198","url":null,"abstract":"Measurement equivalence/invariance (ME/I) is a prerequisite for cross-group comparisons when using survey data. Although popular structural equation modeling software programs, including Mplus and lavaan, enable tests of ME/I using simple commands, identifying noninvariant items when full ME/I is rejected is more challenging. This paper reviews current procedures for identifying noninvariant items, particularly when there are more than two groups. We recommend systematically rotating the reference items and conducting pairwise comparisons on the factor loadings estimated in the configural invariance model and the intercepts estimated in the metric invariance model. The results are then summarized with the list-and-delete method to identify sets of invariant items and clusters of invariant groups. A custom R package, MEI, is developed to implement our recommended procedures. With simple commands, MEI automatically conducts ME/I tests, identifies noninvariant items, and compares latent means with partial measurement invariance. This allows researchers to interpret cross-group comparison results more precisely. Finally, our procedures for testing ME/I from cross-group comparisons and the MEI package are extended to longitudinal studies with panel data, congruence studies with dyadic data, and multilevel studies with nested data.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"26 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-05-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148286585","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

The Case for Reporting Control Variable Coefficients 报告控制变量系数的案例

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-05-14 DOI: 10.1177/10944281261449203

Arturs Kalnins, J. Myles Shaver

引用次数: 0

Discrete Choice Experiments in Management Research 管理研究中的离散选择实验

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-05-07 DOI: 10.1177/10944281261428690

Angelyn Otteson Fairchild, Travis Howell

引用次数: 0

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-04-27 DOI: 10.1177/10944281261432629

Siyi Liu, Louis Hickman, Linus Dahlander, Henning Piezunka

{"title":"Textual Similarity in Organizational Research: Review of Applications, Consistency of Methods, and Best Practice Recommendations","authors":"Siyi Liu, Louis Hickman, Linus Dahlander, Henning Piezunka","doi":"10.1177/10944281261432629","DOIUrl":"https://doi.org/10.1177/10944281261432629","url":null,"abstract":"Organizational research increasingly uses natural language processing (NLP) to measure textual similarity. Despite common usage, the meaning and consistency of similarity measures (e.g., cosine similarity and Euclidean distance) across common NLP methods (e.g., n -grams and document embeddings) is unclear. This risks misalignment between theoretical constructs and textual measures, undermining the comparability of findings across studies. To address this gap, we review studies using textual similarity in organizational and psychological research, finding a jingle-jangle fallacy: identical labels are used for similarity estimates from different NLP methods, and different labels are used for the same method. Additionally, we examine the consistency of similarity measures across and within NLP methods. Different transformer-based embeddings’ similarity results are interchangeable. However, n -grams yield distinct, inconsistent results and are less appropriate for estimating similarity with distance measures. When applied to multi-word inputs, dictionaries and word embeddings return similar results reflecting linguistic style. We provide best practice recommendations and example code for operationalizing textual similarity, including clarifying which NLP methods correspond to content similarity, linguistic style similarity, and semantic similarity at the word, sentence, and document-levels of analysis.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"151 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-04-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147751554","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

“How Many Interviews Do I Need?” An Examination of Interview Numbers and Sampling Moves in Qualitative Research “我需要多少次面试？”定性研究中访谈数和抽样动作的检验

IF 9.5 2区管理学

Organizational Research Methods Pub Date : 2026-04-09 DOI: 10.1177/10944281261424516

Kasper Trolle Elmholdt, Michael Gill, Jeppe Agger Nielsen

{"title":"“How Many Interviews Do I Need?” An Examination of Interview Numbers and Sampling Moves in Qualitative Research","authors":"Kasper Trolle Elmholdt, Michael Gill, Jeppe Agger Nielsen","doi":"10.1177/10944281261424516","DOIUrl":"https://doi.org/10.1177/10944281261424516","url":null,"abstract":"Qualitative researchers face an enduring question: How many interviews do I need? While a variety of guidelines exist, there is limited consensus over which specific factors should determine the number of interviews required. We examined the determination of interview sample sizes in 562 qualitative studies across six high-impact management and organizational journals over a decade. Our findings reveal considerable variance in interview numbers, yet limited information is often provided on the criteria used to determine them. To promote clearer alignment between sample sizes and methodology, we examined studies with detailed descriptions of their interview sampling. We identified specific “sampling moves” used to determine the number of interviews, categorized into three types—opening, focusing, and closing sampling moves—that researchers use to establish confidence in the sample and support theoretical insights. By implication, our study refutes the notion of a “magic” interview number. Instead, sampling moves are heuristic tools that qualitative researchers can thoughtfully adapt to their analytical aims when determining appropriate sample sizes.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"28 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2026-04-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147635728","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0