Flynn effects are biased by differential item functioning over time: A test using overlapping items in Wechsler scales

Abstract

The items of intelligence tests can demonstrate differential item functioning across different groups: cross-sample differences in item difficulty or discrimination, independently of any difference of ability. This is also true of comparisons over time: as the cultural context changes, items may increase or decrease in difficulty. This phenomenon is well-known, but its impact on estimates of the Flynn effect has not been systematically investigated. In the current study, we tested differential item functioning in a subset of 111 items common to consecutive versions of the French WAIS-R (1989), WAIS-III (1999) and/or WAIS-IV (2009), using the three normative samples (total N = 2979). Over half the items had significant differential functioning over time, generally becoming more difficult from one version to the next for the same level of ability. The magnitude of differential item functioning tended to be small for each item separately, but the cumulative effect over all items led to underestimating the Flynn effect by about 3 IQ points per decade, a bias close to the expected size of the effect itself. In this case, this bias substantially affected the conclusions, even creating an ersatz negative Flynn effect for the 1999–2009 period, when in fact ability increased (1989–1999) or stagnated (1999–2009) when accounting for differential item functioning. We recommend that studies of the Flynn effect systematically investigate the possibility of differential item functioning to obtain unbiased ability estimates.