Can Culture Be a Salient Predictor of Test-Taking Engagement? An Analysis of Differential Noneffortful Responding on an International College-Level Assessment of Critical Thinking

Abstract

ABSTRACT The objective of this study was to evaluate whether differential noneffortful responding (identified via response latencies) was present in four countries administered a low-stakes college-level critical thinking assessment. Results indicated significant differences (as large as .90 SD) between nearly all country pairings in the average number of noneffortful responses per test taker. Furthermore, noneffortful responding was found to be associated with a number of individual-level predictors, such as demographics (both gender and academic year), prior ability, and perceived difficulty of the test, though, these predictors were found to differ across countries. Ignoring the presence of noneffortful responses was associated with: (a) model fit deterioration as well as inflation of reliability, and (b) the inclusion of non-invariant items in the score linking anchor set. However, no meaningful differences in relative performance were noted once accounting for noneffortful responses. Implications for test development and improving the validity of score-based inferences from international assessments are discussed.