Beyond linear regression: Statistically modeling aptitude-treatment interactions and the differential effectiveness of educational interventions

Abstract

Research on aptitude-treatment interactions and the differential effectiveness of educational interventions faces statistical challenges that may contribute to sparse findings and unclear replicability. These challenges include the presence of nonlinear-, floor-, or ceiling effects, underpowered samples, and the multivariate nature of learner aptitudes. Linear regression, which prevails as the typical statistical approach in this research area, lacks the flexibility to meet these challenges. As alternatives, we present three statistical approaches: (1) Additive regression models to capture and control nonlinear or floor/ceiling effects, (2) Bayesian multilevel modeling, which can improve statistical power and allows for more complex models, and (3) clustering multivariate constellations of learner aptitudes via latent profile analysis. We demonstrate these three approaches on a motivating dataset from a scientific reasoning training, discussing their relative (dis-)advantages and how these and further models may aid research into differential effectiveness across different research topics and designs.

Educational relevance statement

In educational interventions, researchers and practitioners are often interested in knowing for whom an intervention works best or worst. We present three statistical models that can help examine this question and overcome issues that have long bugged this field. We discuss how these approaches can help research across multiple areas, for example to examine the effects of educational technologies (augmented & virtual reality).