Find the Mistake!

Abstract. Reasoning ability has commonly been regarded as the best predictor of academic and occupational success. Due to concerns about the validity of multiple-choice (MC) formats, test security breaches, and the fact that the difficulty levels of most existing reasoning assessments target the population mean, there is a constant need for new reliable and valid test instruments that can be applied to assess fluid intelligence in advanced cognitive performance areas. We developed a novel computerized figural matrices test to assess nonverbal reasoning for university student aptitude assessment. In two studies, we generated, revised, and empirically validated the Isometric Matrices Test (IMT). Our results show that the IMT is less prone to test-wiseness strategies than existing reasoning tests. In a third study, we created and evaluated an innovative Find the Mistake (FtM) response format as an alternative to classical multiple-choice formats. Overall, both response formats revealed satisfactory psychometric quality in terms of item difficulties and discrimination, test-retest reliability, construct and criterion validity, and Rasch or two-parameter logistic (2PL) model fit, but in one MC version, the internal consistency was low due to negative discrimination indices. The MC response format turned out to be easier than the FtM format, with men slightly outperforming women in both response modes. We propose the IMT as a useful tool for assessing nonverbal reasoning ability in above-average performance areas and discuss the automatic generation of larger IMT item pools for adaptive testing in order to increase test security and reliability.