Developing a Framework for the Examination of Anchoring Vignette Assumptions Using Cognitive Interviews: A Demonstration in the ICT Skills Domain

Self-assessment measures are commonly used in questionnaire surveys. However, one of the problems with self-reports is that they may be prone to differences in scale usage among respondents. The anchoring vignette method addresses this issue. It relies on two assumptions: response consistency and vignette equivalence. Here we aim to develop a framework for the examination of these assumptions using cognitive interviews and demonstrate it in the Information and Communications Technology (ICT) skills domain. First, we develop new anchoring vignettes describing various ICT skill domains. Second, we examine both assumptions using cognitive interviews with students who are studying different fields. Our analysis shows that the assumptions are indeed not adhered to, especially the assumption of vignette equivalence. Third, we develop a general framework for the examination of the method’s assumptions.