Using automatic item generation to construct scheduling problems measuring planning ability

Abstract

Planning is a core component of executive functioning that has been hypothesized to be central to many activities in daily life and occupational settings. Despite its practical and theoretical relevance, there is a lack of psychometric tests, whose item parameters can be predicted by item design features, that have been shown to be linked to cognitive processes involved in planning (=radicals). In the present article the automatic min-max approach was used to construct k = 140 (study I: N = 1573) and k = 17 (study II: N = N = 548 Austrian and N = 572 Italian adolescents) scheduling problems measuring planning. The psychometric quality of the items was evaluated in three studies. The results indicated, that the 1PL Rasch model and the Linear Logistic Test model fitted the data reasonably well, and that the item- and basic parameter estimates can be assumed to be invariant across relevant socio-demographic (study I and II). The radicals jointly explained 89.30% of the variance in the item parameter estimates, and all of them contributed significantly to the prediction of the item parameters. Furthermore, planning – as measured by the scheduling problems and the Tower of London (TOL-F) – was moderately correlated with G_c, G_q, and G_v, and highly correlated with G_f (study III: N = 249). By contrast, G_f was highly correlated with planning ability and the other three second stratum factors. Thus, G_f and planning ability differ in their structural relation to other second stratum factors, which provides evidence that planning ability cannot be regarded to be synonymous to G_f. The article discusses the theoretical and practical implications of these findings.