Is test takers' video game usage a game changer? Criterion-related validity and potential bias of a cognitive game-based assessment

Game-based assessments (GBAs) have become increasingly popular among researchers and practitioners. However, research on their criterion-related validity is scarce, and despite the proposed benefits of GBAs, concerns remain about their fairness. Thus, this study examines the criterion-related validity of a GBA measuring cognitive abilities, and whether there is differential validity and test bias based on test takers' video game usage. For this purpose, data were collected from two samples: a working sample (N = 156) and a potential applicant sample (N = 92). In both samples, participants completed a cognitive GBA and answered questions assessing their academic performance, video game usage, and demographics. For the working sample, we also collected supervisor-rated job performance. Results showed that GBA performance was positively correlated with academic performance in both samples, suggesting criterion-related validity. Video game usage was positively associated with GBA performance and negatively with academic performance, indicating test bias. These findings suggest that video game usage enhances skills relevant to success in cognitive GBAs but not necessarily to academic performance, raising fairness concerns. This study advances research on cognitive GBAs by providing evidence for their criterion-related validity, by addressing concerns about fairness, and by providing insights for their use in personnel selection.