{"title":"Against the Mainstream: On the Limitations of Non-Invariance Diagnostics: Response to Fischer et al. and Meuleman et al.","authors":"C. Welzel, S. Kruse, Lennart Brunkert","doi":"10.1177/00491241221091754","DOIUrl":"https://doi.org/10.1177/00491241221091754","url":null,"abstract":"Our original 2021 SMR article “Non-Invariance? An Overstated Problem with Misconceived Causes” disputes the conclusiveness of non-invariance diagnostics in diverse cross-cultural settings. Our critique targets the increasingly fashionable use of Multi-Group Confirmatory Factor Analysis (MGCFA), especially in its mainstream version. We document—both by mathematical proof and an empirical illustration—that non-invariance is an arithmetic artifact of group mean disparity on closed-ended scales. Precisely this arti-factualness renders standard non-invariance markers inconclusive of measurement inequivalence under group-mean diversity. Using the Emancipative Values Index (EVI), OA-Section 3 of our original article demonstrates that such artifactual non-invariance is inconsequential for multi-item constructs’ cross-cultural performance in nomological terms, that is, explanatory power and predictive quality. Given these limitations of standard non-invariance diagnostics, we challenge the unquestioned authority of invariance tests as a tool of measurement validation. Our critique provoked two teams of authors to launch counter-critiques. We are grateful to the two comments because they give us a welcome opportunity to restate our position in greater clarity. Before addressing the comments one by one, we reformulate our key propositions more succinctly.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":"52 1","pages":"1438 - 1455"},"PeriodicalIF":6.3,"publicationDate":"2022-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41829124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
L. Vila‐Henninger, C. Dupuy, Virginie Van Ingelgom, M. Caprioli, Ferdinand Teuber, Damien Pennetreau, Margherita Bussi, Cal Le Gall
{"title":"Abductive Coding: Theory Building and Qualitative (Re)Analysis","authors":"L. Vila‐Henninger, C. Dupuy, Virginie Van Ingelgom, M. Caprioli, Ferdinand Teuber, Damien Pennetreau, Margherita Bussi, Cal Le Gall","doi":"10.1177/00491241211067508","DOIUrl":"https://doi.org/10.1177/00491241211067508","url":null,"abstract":"Qualitative secondary analysis has generated heated debate regarding the epistemology of qualitative research. We argue that shifting to an abductive approach provides a fruitful avenue for qualitative secondary analysts who are oriented towards theory-building. However, the concrete implementation of abduction remains underdeveloped—especially for coding. We address this key gap by outlining a set of tactics for abductive analysis that can be applied for qualitative analysis. Our approach applies Timmermans and Tavory's ( Timmermans and Tavory 2012 ; Tavory and Timmermans 2014 ) three stages of abduction in three steps for qualitative (secondary) analysis: Generating an Abductive Codebook, Abductive Data Reduction through Code Equations, and In-Depth Abductive Qualitative Analysis. A key contribution of our article is the development of “code equations”—defined as the combination of codes to operationalize phenomena that span individual codes. Code equations are an important resource for abduction and other qualitative approaches that leverage qualitative data to build theory.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-14","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46997816","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"In Search of a Comparable Measure of Generalized Individual Religiosity in the World Values Survey","authors":"A. Remizova, M. Rudnev, E. Davidov","doi":"10.1177/00491241221077239","DOIUrl":"https://doi.org/10.1177/00491241221077239","url":null,"abstract":"Individual religiosity measures are used by researchers to describe and compare individuals and societies. However, the cross-cultural comparability of the measures has often been questioned but rarely empirically tested. In the current study, we examined the cross-national measurement invariance properties of generalized individual religiosity in the sixth wave of the World Values Survey. For the analysis, we used multiple group confirmatory factor analysis and alignment. Our results demonstrated that a theoretically driven measurement model was not invariant across all countries. We suggested four unidimensional measurement models and four overlapping groups of countries in which these measurement models demonstrated approximate invariance. The indicators that covered praying practices, importance of religion, and confidence in its institutions were more cross-nationally invariant than other indicators.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"46774458","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Visual Design and Cognition in List-Style Open-Ended Questions in Web Probing","authors":"Katharina Meitinger, Tanja Kunz","doi":"10.1177/00491241221077241","DOIUrl":"https://doi.org/10.1177/00491241221077241","url":null,"abstract":"<p>Previous research reveals that the visual design of open-ended questions should match the response task so that respondents can infer the expected response format. Based on a web survey including specific probes in a list-style open-ended question format, we experimentally tested the effects of varying numbers of answer boxes on several indicators of response quality. Our results showed that using multiple small answer boxes instead of one large box had a positive impact on the number and variety of themes mentioned, as well as on the conciseness of responses to specific probes. We found no effect on the relevance of themes and the risk of item non-response. Based on our findings, we recommend using multiple small answer boxes instead of one large box to convey the expected response format and improve response quality in specific probes. This study makes a valuable contribution to the field of web probing, extends the concept of response quality in list-style open-ended questions, and provides a deeper understanding of how visual design features affect cognitive response processes in web surveys.</p>","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":"44 38","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138506061","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Comparing Single- and Multiple-Question Designs of Measuring Family Income in China Family Panel Studies","authors":"Qiong Wu, Li-na Gu","doi":"10.1177/00491241221077238","DOIUrl":"https://doi.org/10.1177/00491241221077238","url":null,"abstract":"Family income questions in general purpose surveys are usually collected with either a single-question summary design or a multiple-question disaggregation design. It is unclear how estimates from the two approaches agree with each other. The current paper takes advantage of a large-scale survey that has collected family income with both methods. With data from 14,222 urban and rural families in the 2018 wave of the nationally representative China Family Panel Studies, we compare the two estimates, and further evaluate factors that might contribute to the discrepancy. We find that the two estimates are loosely matched in only a third of all families, and most of the matched families have a simple income structure. Although the mean of the multiple-question estimate is larger than that of the single-question estimate, the pattern is not monotonic. At lower percentiles up till the median, the single-question estimate is larger, whereas the multiple-question estimate is larger at higher percentiles. Larger family sizes and more income sources contribute to higher likelihood of inconsistent estimates from the two designs. Families with wage income as the main income source have the highest likelihood of giving consistent estimates compared with all other families. In contrast, families with agricultural income or property income as the main source tend to have very high probability of larger single-question estimates. Omission of certain income components and rounding can explain over half of the inconsistencies with higher multiple-question estimates and a quarter of the inconsistencies with higher single-question estimates.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47384693","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Do Different Devices Perform Equally Well with Different Numbers of Scale Points and Response Formats? A test of measurement invariance and reliability","authors":"Natalja Menold, V. Toepoel","doi":"10.1177/00491241221077237","DOIUrl":"https://doi.org/10.1177/00491241221077237","url":null,"abstract":"Research on mixed devices in web surveys is in its infancy. Using a randomized experiment, we investigated device effects (desktop PC, tablet and mobile phone) for six response formats and four different numbers of scale points. N = 5,077 members of an online access panel participated in the experiment. An exact test of measurement invariance and Composite Reliability were investigated. The results provided full data comparability for devices and formats, with the exception of continuous Visual Analog Scale (VAS), but limited comparability for different numbers of scale points. There were device effects on reliability when looking at the interactions with formats and number of scale points. VAS, use of mobile phones and five point scales consistently gained lower reliability. We suggest technically less demanding implementations as well as a unified design for mixed-device surveys.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"47694064","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"What's to Like? Facebook as a Tool for Survey Data Collection.","authors":"Daniel Schneider, Kristen Harknett","doi":"10.1177/0049124119882477","DOIUrl":"https://doi.org/10.1177/0049124119882477","url":null,"abstract":"<p><p>In this paper, we explore the use of Facebook targeted advertisements for the collection of survey data. We illustrate the potential of survey sampling and recruitment on Facebook through the example of building a large employee-employer linked dataset as part of The Shift Project. We describe the workflow process of targeting, creating, and purchasing survey recruitment advertisements on Facebook. We address concerns about sample selectivity, and apply post-stratification weighting techniques to adjust for differences between our sample and that of \"gold-standard\" data sources. We then compare univariate and multi-variate relationships in the Shift data against the Current Population Survey and the National Longitudinal Survey of Youth-1997. Finally, we provide an example of the utility of the firm-level nature of the data by showing how firm-level gender composition is related to wages. We conclude by discussing some important remaining limitations of the Facebook approach, as well as highlighting some unique strengths of the Facebook targeting advertisement approach, including the ability for rapid data collection in response to research opportunities, rich and flexible sample targeting capabilities, and low cost, and we suggest broader applications of this technique.</p>","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":"51 1","pages":"108-140"},"PeriodicalIF":6.3,"publicationDate":"2022-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1177/0049124119882477","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"9407478","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Moving Beyond Linear Regression: Implementing and Interpreting Quantile Regression Models With Fixed Effects","authors":"Fernando Rios-Avila, Michelle Lee Maroto","doi":"10.1177/00491241211036165","DOIUrl":"https://doi.org/10.1177/00491241211036165","url":null,"abstract":"<p>Quantile regression (QR) provides an alternative to linear regression (LR) that allows for the estimation of relationships across the distribution of an outcome. However, as highlighted in recent research on the motherhood penalty across the wage distribution, different procedures for conditional and unconditional quantile regression (CQR, UQR) often result in divergent findings that are not always well understood. In light of such discrepancies, this paper reviews how to implement and interpret a range of LR, CQR, and UQR models with fixed effects. It also discusses the use of Quantile Treatment Effect (QTE) models as an alternative to overcome some of the limitations of CQR and UQR models. We then review how to interpret results in the presence of fixed effects based on a replication of Budig and Hodges’s work on the motherhood penalty using NLSY79 data.</p>","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":"38 ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-02-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138506042","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Updating a Time-Series of Survey Questions: The Case of Abortion Attitudes in the General Social Survey","authors":"Sarah K. Cowan, M. Hout, Stuart Perrett","doi":"10.1177/00491241211043140","DOIUrl":"https://doi.org/10.1177/00491241211043140","url":null,"abstract":"Long-running surveys need a systematic way to reflect social change and to keep items relevant to respondents, especially when they ask about controversial subjects, or they threaten the items’ validity. We propose a protocol for updating measures that preserves content and construct validity. First, substantive experts articulate the current and anticipated future terms of debate. Then survey experts use this substantive input and their knowledge of existing measures to develop and pilot a large battery of new items. Third, researchers analyze the pilot data to select items for the survey of record. Finally, the items appear on the survey-of-record, available to the whole user community. Surveys-of-record have procedures for changing content that determine if the new items appear just once or become part of the core. We provide the example of developing new abortion attitude measures in the General Social Survey. Current questions ask whether abortion should be legal under varying circumstances. The new abortion items ask about morality, access, state policy, and interpersonal dynamics. They improve content and construct validity and add new insights into Americans’ abortion attitudes.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-01-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41519369","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Relevant, Irrelevant, or Ambiguous? Toward a New Interpretation of QCA’s Solution Types","authors":"Tim Haesebrouck","doi":"10.1177/00491241211036153","DOIUrl":"https://doi.org/10.1177/00491241211036153","url":null,"abstract":"The field of qualitative comparative analysis (QCA) is witnessing a heated debate on which one of the QCA’s main solution types should be at the center of substantive interpretation. This article argues that the different QCA solutions have complementary strengths. Therefore, researchers should interpret the three solution types in an integrated way, in order to get as much information as possible on the causal structure behind the phenomenon under investigation. The parsimonious solution is capable of identifying causally relevant conditions, the conservative solution of identifying contextually irrelevant conditions. In addition to conditions for which the data provide evidence that they are causally relevant or contextually irrelevant, there will be conditions for which the data neither suggest that they are relevant nor contextually irrelevant. In line with the procedure for crafting the intermediate solution, it is possible to make clear for which of these ambiguous conditions it is not plausible that they are relevant in the context of the research.","PeriodicalId":21849,"journal":{"name":"Sociological Methods & Research","volume":" ","pages":""},"PeriodicalIF":6.3,"publicationDate":"2022-01-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45282212","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"社会学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}