Reactivity, validity, and measurement in the social sciences

Abstract

Reactivity takes place when being measured or classified affects a subject's attitudes and behaviour to such an extent that this affects results in (subsequent) measurement or classification. How one should recognize and evaluate such changes over time is a prominent topic in the psychometric literature, where it falls under the broader concept of response shift. A key question in that literature is if reactive changes are ever natural, that is, whether there are cases in which the measurement results before and after the shift are both accurate to the underlying phenomenon. This is thought to depend on what causal regularities obtain in the area of study, but also on whether the phenomenon is in part constituted by normative and changeable evaluations by the respondent. While the naturalness of reactivity is typically discussed exclusively at the level of measures of individual respondents, much philosophical attention has also been spent in recent years on the reactivity in measurement of macro-social entities, like countries and institutions. This article offers conceptual clarification for that new field by mapping concepts from psychometrics to the measurement of macro-social phenomena. In particular, this article distinguishes two existing arguments about the naturalness of reactivity in psychometrics (‘the argument from regularity’ and ‘the argument from strong evaluation’) and shows what is needed to extrapolate them fruitfully, by focusing on two examples of public measures, viz., democracy indicators and university rankings.