PISA Country Rankings for Italy and Spain: Revised Results

Abstract

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology, a two-parameter logistic model is applied to PISA mathematics item data from Italy and Spain for the year 2009. In the estimation procedure, item difficulty and dispersion parameters were allowed to differ across the two countries and samples were restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters were also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability or proficiency scores obtained from the methods employed here are significantly higher for Spain, in sharp contrast to PISA results, which gave both countries virtually the same rank in mathematics (489 for Italy and 488 for Spain). These results raise serious questions about PISA methodology and the role that PISA results play in the formulation educational policy in member countries.