Estimating Item Wording Effects in Self-Report Measures with Generalizability Theory-Based SEMs: Illustrations Using the Self-Description Questionnaire-III.

Abstract

Handling item wording effects within Likert-style, self-report questionnaires has long been a challenge when measuring psychological traits. When testing models for such traits, wording effects are commonly addressed by correlating uniquenesses for negatively and positively phrased items or including separate uncorrelated method factors for each effect. However, the magnitude of wording effects is rarely considered in such analyses or distinguished from effects of multiple sources of measurement error. In this article, we demonstrate how generalizability theory-based structural equation model designs are well suited for such purposes using results from all subscales within the Self-Description Questionnaire-III taken by a large sample of college students (n = 1,796) on two occasions. Results emphasized the importance of separating construct, item wording, and measurement error (specific-factor, transient, and random-response) effects for each individual subscale and the effectiveness of generalizability theory-based techniques in doing so. Within the most complete designs, average proportions of explained observed score variance were highest for targeted constructs, followed respectively by random-response error, transient error, specific-factor error, and item wording. We provide code in R for analyzing both generalizability theory and parallel conventional congeneric structural equation models to estimate construct, wording, and measurement error effects using both single- and multiple-occasion designs.