More accurate estimation for nonrandom sampling surveys: A post hoc correction method

Abstract

Nonprobability sample surveys are commonly used in applied linguistics research, but their lack of representativeness makes it difficult to generalize their results to the wider population. To address this issue, this study introduces a post hoc statistical correction method that uses an existing probability sample survey as reference data. The paper begins by outlining the importance of such corrections and provides details of a practical correction procedure that employs propensity scores as a balancing factor. The method is then applied to real-world data from the 2022 Survey of Workers’ English Use, a nonprobability web panel survey investigating English use among Japanese people as an international language. The corrected estimates are subsequently compared to those from a nationally representative survey. The results showed that the proposed correction method effectively adjusted the proportions and mean values of self-reported English proficiency and replicated the population. However, the method was less effective in adjusting the mean values for attitudes toward globalization as well as correlation coefficients, although it did not worsen these estimates. Based on these findings, the paper advocates for the adoption of this correction method in applied linguistic studies where feasible, with some critical considerations and caveats highlighted.