The illusion of validity: how effort inflates the perceived validity of interview questions

ABSTRACT Interviewers are often confident in the validity of their interview questions. What drives this confidence and is it justified? In three studies, we found that question creators judged their own interview questions as more valid than when the same questions are judged by an evaluator. We also found that effort expenditure inflated the perceived validity of interview questions but not question quality. Question creators’ perceptions of validity were primarily driven by their self-confidence, and not the question quality. As an intervention, we nudged participants into holding more favourable attitudes towards better questions (i.e., structured questions) by allowing them to choose a subset of them from a pre-written list. Together, we found that while effort expenditure was responsible for the illusion of validity when evaluating unstructured (i.e., low-quality) questions, the same mechanism could also be used to improve interviewers’ acceptance of structured questions. Implications for structured interviews and the scientist-practitioner gap are discussed.