Much Ado About Survey Tables: A Comparison of Chi-Square Tests and Software to Analyze Categorical Survey Data

Abstract

Chi-square tests are often employed to examine the association of categorical variables, the homogeneity of proportions between two or more samples, and the goodness-of-fit for a specified distribution. To account for the complex design of survey data, variants of chi-square tests as well as software packages that implement these tests have been developed. Nevertheless, from a survey practitioner’s perspective, there is a lack of applied literature that reviews and compares alternative options of survey chi-square tests and their associated programming and output. This paper aims to fill such a gap.Many modern statistical software packages for survey analysis are capable of computing either the Wald chi-square test or the Rao-Scott chi-square test, along with other types of chi-square tests, including the Rao-Scott likelihood ratio chi-square test and the Wald log-linear chi-square test. This paper focuses on these four types of chi-square tests, and examines four statistical packages that compute them in SAS®, R, Python and SUDAAN^®. While the same type of tests using different packages yield similar results, different types of chi-square tests may yield variations in p-values when conducting the same comparison. Sample programming code is included in Appendix for readers’ reference.