Towards more appropriate modelling of linguistic complexity measures: Beyond traditional regression models

Abstract

Despite a recent emphasis on appropriate quantitative data analysis as part of the methodological reform, applied linguists often overlook scrutinising the adequacy of their analytical methods, risking model misspecification. This article critiques the use of regression models assuming normal error distributions for modelling linguistic complexity measures. It examines two alternative approaches: weighted linear regression with a log-transformed outcome variable and negative binomial regression, demonstrating how they mitigate associated limitations. Normal error regression models are inadequate for modelling count-based ratio variables due to two main issues: (i) count variables are theoretically lower-bounded, a constraint not addressed by normal error regression models, and (ii) variations in count quantities lead to differences in sampling variability, violating the homoscedasticity assumption and potentially inflating the Type I error rate. Analysis of 14 syntactic complexity measures and an artificial-data simulation show that the lower bound of the prediction intervals for normal error regression models often falls below the theoretical minimum in realistic scenarios. Moreover, the denominator count of syntactic complexity measures negatively correlates with variability, causing heteroscedasticity, a higher false-positive rate, and reduced true value coverage by 80% confidence intervals. Both alternative approaches outperform normal error regression models in these criteria. These findings challenge the suitability of normal error regressions for modelling syntactic complexity measures and caution against their use for other count-based measures in second language research and corpus linguistics. This necessitates reevaluating the widespread use of normal error regression in applied linguistics, urging methodologists to develop and validate more structurally faithful modelling approaches.