{"title":"Reliability Estimates for IRT-Based Forced-Choice Assessment Scores","authors":"Yin Lin","doi":"10.1177/1094428121999086","DOIUrl":"https://doi.org/10.1177/1094428121999086","url":null,"abstract":"Forced-choice (FC) assessments of noncognitive psychological constructs (e.g., personality, behavioral tendencies) are popular in high-stakes organizational testing scenarios (e.g., informing hiring decisions) due to their enhanced resistance against response distortions (e.g., faking good, impression management). The measurement precisions of FC assessment scores used to inform personnel decisions are of paramount importance in practice. Different types of reliability estimates are reported for FC assessment scores in current publications, while consensus on best practices appears to be lacking. In order to provide understanding and structure around the reporting of FC reliability, this study systematically examined different types of reliability estimation methods for Thurstonian IRT-based FC assessment scores: their theoretical differences were discussed, and their numerical differences were illustrated through a series of simulations and empirical studies. In doing so, this study provides a practical guide for appraising different reliability estimation methods for IRT-based FC assessment scores.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"575 - 590"},"PeriodicalIF":9.5,"publicationDate":"2021-03-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428121999086","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47986107","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
David Antons, Christoph F. Breidbach, Amol M. Joshi, T. Salge
{"title":"Computational Literature Reviews: Method, Algorithms, and Roadmap","authors":"David Antons, Christoph F. Breidbach, Amol M. Joshi, T. Salge","doi":"10.1177/1094428121991230","DOIUrl":"https://doi.org/10.1177/1094428121991230","url":null,"abstract":"The substantial volume, continued growth, and resulting complexity of the scientific literature not only increases the need for systematic, replicable, and rigorous literature reviews, but also highlights the natural limits of human researchers’ information processing capabilities. In search of a solution to this dilemma, computational techniques are beginning to support human researchers in synthesizing large bodies of literature. However, actionable methodological guidance on how to design, conduct, and document such computationally augmented literature reviews is lacking to date. We respond by introducing and defining computational literature reviews (CLRs) as a new review method and put forward a six-step roadmap, covering the CLR process from identifying the review objectives to selecting algorithms and reporting findings. We make the CLR method accessible to novice and expert users alike by identifying critical design decisions and typical challenges for each step and provide practical guidelines for tailoring the CLR method to four conceptual review goals. As such, we present CLRs as a literature review method where the choice, design, and implementation of a CLR are guided by specific review objectives, methodological capabilities, and resource constraints of the human researcher.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"26 1","pages":"107 - 138"},"PeriodicalIF":9.5,"publicationDate":"2021-03-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428121991230","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"48441588","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Marshall, F. Yammarino, S. Parameswaran, Minyoung Cheong
{"title":"Using CATA and Machine Learning to Operationalize Old Constructs in New Ways: An Illustration Using U.S. Governors’ COVID-19 Press Briefings","authors":"J. Marshall, F. Yammarino, S. Parameswaran, Minyoung Cheong","doi":"10.1177/10944281221098607","DOIUrl":"https://doi.org/10.1177/10944281221098607","url":null,"abstract":"Increased computing power and greater access to online data have led to rapid growth in the use of computer-aided text analysis (CATA) and machine learning methods. Using “big data”, researchers have not only advanced new streams of research, but also new research methodologies. Noting this trend and simultaneously recognizing the value of traditional research methods, we lay out a methodology that bridges the gap between old and new approaches to operationalize old constructs in new ways. With a combination of web scraping, CATA, and supervised machine learning, using labeled ground truth data (i.e., data with known inputs and outputs), we train a model to predict CIP (Charismatic-Ideological-Pragmatic) leadership styles from running text. To illustrate this method, we apply the model to classify U.S. state governors’ COVID-19 press briefings according to their CIP leadership style. In addition, we demonstrate content and convergent validity of the method.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"1 1","pages":""},"PeriodicalIF":9.5,"publicationDate":"2021-02-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43328744","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"On the Use of Balanced Item Parceling to Counter Acquiescence Bias in Structural Equation Models","authors":"Bert Weijters, H. Baumgartner","doi":"10.1177/1094428121991909","DOIUrl":"https://doi.org/10.1177/1094428121991909","url":null,"abstract":"We propose the use of balanced item parcels to account for method effects caused by acquiescent responding. The use of balanced parcels avoids the need to model method effects explicitly and results in a parsimonious specification of measurement and full structural equation models in the presence of unwanted method effects, particularly when a scale consists of a relatively large number of items. Balanced item parcels are sums or averages of individual items consisting of an equal number of regular and reversed items measuring the same construct. When regular and reversed items are combined into parcels, method effects cancel out (assuming that the method effects affecting the regular and reversed items in a parcel are equal in magnitude), and model fit and parameter estimates will no longer be negatively affected by acquiescent responding. We discuss why balanced item parceling works and when it is likely to prove beneficial, and we present a step-by-step procedure explaining how to use balanced item parceling in practice. We also report a brief hypothetical example to illustrate the proposed approach.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"170 - 180"},"PeriodicalIF":9.5,"publicationDate":"2021-02-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428121991909","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46996236","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Sample Selection in Systematic Literature Reviews of Management Research","authors":"Martin R. W. Hiebl","doi":"10.1177/1094428120986851","DOIUrl":"https://doi.org/10.1177/1094428120986851","url":null,"abstract":"Systematic review techniques are about to become the “new normal” in reviews of management research. However, there is not yet much advice on how to organize the sample selection process as part of such reviews. This article addresses this void and analyzes this vital part of systematic reviews in more detail. In particular, it offers a critical review of systematic literature reviews published in the Academy of Management Annals and the International Journal of Management Reviews between 2004 and 2018. Based on this methodological literature review, the article presents issues to consider in the most critical choices during the sample selection process. Furthermore, this review identifies several descriptive features such as the mean number of research items included in systematic reviews, the mean number of databases used, and the mean coverage period of such reviews. These numbers may be used as benchmark figures in future reviews.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"26 1","pages":"229 - 261"},"PeriodicalIF":9.5,"publicationDate":"2021-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428120986851","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43614493","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
John R. Busenbark, Scott D. Graffin, R. Campbell, Eric Y. Lee
{"title":"A Marginal Effects Approach to Interpreting Main Effects and Moderation","authors":"John R. Busenbark, Scott D. Graffin, R. Campbell, Eric Y. Lee","doi":"10.1177/1094428120976838","DOIUrl":"https://doi.org/10.1177/1094428120976838","url":null,"abstract":"This Short Methodological Report builds on research about moderation practices by focusing on a marginal effects approach to interpreting how a main effect is informed by the presence of a moderating variable. Following a content analysis of published studies and a survey of management researchers, our findings suggest there is a great deal of confusion about the ways in which to interpret how a main effect may fluctuate owing to a moderating variable. We therefore provide explicit instructions on how to implement and interpret a marginal effects approach that depicts the nature of a main effect in the presence of a moderator. We use different scenarios and examples to illustrate how researchers can employ the marginal effects technique, which provides an indication of the relationship between the independent and dependent variables over different values of the moderator. We argue and demonstrate that the marginal effects approach helps resolve conflicting findings that may arise from using other prevailing techniques to interpret both main effects and moderation.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"25 1","pages":"147 - 169"},"PeriodicalIF":9.5,"publicationDate":"2021-01-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428120976838","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44971423","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
J. Cortina, Hannah M. Markell-Goldstein, Jennifer P. Green, Yingyi Chang
{"title":"How Are We Testing Interactions in Latent Variable Models? Surging Forward or Fighting Shy?:","authors":"J. Cortina, Hannah M. Markell-Goldstein, Jennifer P. Green, Yingyi Chang","doi":"10.25384/SAGE.C.4665197.V1","DOIUrl":"https://doi.org/10.25384/SAGE.C.4665197.V1","url":null,"abstract":"Latent variable models and interaction effects have both been common in the organizational sciences for some time. Methods for incorporating interactions into latent variable models have existed si...","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"26-54"},"PeriodicalIF":9.5,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"69207895","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Building Transparency and Trustworthiness in Inductive Research Through Computer-Aided Qualitative Data Analysis Software","authors":"P. O’Kane, Anne D. Smith, Michael P. Lerman","doi":"10.1177/1094428119865016","DOIUrl":"https://doi.org/10.1177/1094428119865016","url":null,"abstract":"Many scholars have called for qualitative research to demonstrate transparency and trustworthiness in the data analysis process. Yet these processes, particularly within inductive research, often remain shrouded in mystery. We suggest that computer-aided/assisted qualitative data analysis software (CAQDAS) can support qualitative researchers in their efforts to present their analysis and findings in a transparent way, thus enhancing trustworthiness. To this end, we propose, describe, and illustrate working examples of six CAQDAS building blocks, three combined CAQDAS techniques, and two coder consistency checks. We argue that these techniques give researchers the language to write about their methods and findings in a transparent manner and that their appropriate use enhances a research project’s trustworthiness. Specific CAQDAS techniques are rarely discussed across an array of inductive research processes. Thus, we see this article as the beginning of a conversation about the utility of CAQDAS to support inductive qualitative research.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"104 - 139"},"PeriodicalIF":9.5,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428119865016","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45888085","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Croon’s Bias-Corrected Factor Score Path Analysis for Small- to Moderate-Sample Multilevel Structural Equation Models","authors":"Ben Kelcey, Kyle Cox, N. Dong","doi":"10.1177/1094428119879758","DOIUrl":"https://doi.org/10.1177/1094428119879758","url":null,"abstract":"Maximum likelihood estimation of multilevel structural equation model (MLSEM) parameters is a preferred approach to probe theories involving latent variables in multilevel settings. Although maximum likelihood has many desirable properties, a major limitation is that it often fails to converge and can incur significant bias when implemented in studies with a small to moderate multilevel sample (e.g., fewer than 100 organizations with 10 or less individuals/organization). To address similar limitations in single-level SEM, literature has developed Croon’s bias-corrected factor score path analysis estimator that converges more regularly than maximum likelihood and delivers less biased parameter estimates with small to moderate sample sizes. We derive extensions to this framework for MLSEMs and probe the degree to which the estimator retains these advantages with small to moderate multilevel samples. The estimator emerges as a useful alternative or complement to maximum likelihood because it often outperforms maximum likelihood in small to moderate multilevel samples in terms of convergence, bias, error variance, and power. The proposed estimator is implemented as a function in R using lavaan and is illustrated using a multilevel mediation example.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"55 - 77"},"PeriodicalIF":9.5,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428119879758","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45851821","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Using Outliers for Theory Building","authors":"M. Gibbert, L. B. Nair, M. Weiss, M. Hoegl","doi":"10.1177/1094428119898877","DOIUrl":"https://doi.org/10.1177/1094428119898877","url":null,"abstract":"Outliers are promising candidates for theory building because they defy expected cause-and-effect relationships. Nonetheless, researchers often treat them as a nuisance and exclude them from further study. In fact, our analysis founds only two article using outliers for theory development in all quantitative articles published from 1993 to 2012 in six major management journals, and less than 5% cared to even mention them (relaying reasons for deleting them, mostly). To rectify this, we provide a roadmap for empirical researchers interested in theory building.","PeriodicalId":19689,"journal":{"name":"Organizational Research Methods","volume":"24 1","pages":"172 - 181"},"PeriodicalIF":9.5,"publicationDate":"2021-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/1094428119898877","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43107351","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"管理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}