Fairness Issues and Evaluation in Psychometrics and AI/ML: What Can We Learn from Each Field?

Abstract

Fairness in psychological and educational measurement has been a long-standing issue. Meanwhile, fairness is an issue that has been receiving an increasing amount of attention in the machine learning (ML) and artificial intelligence (AI) community, particularly in the last decade. While there is some recognition by the AI/ML community of the early roots of fairness definitions and operational metrics in the testing context, especially the parallel between fairness in AI/ML and test fairness in a regression and prediction context, there has not been a systematic introduction about the rapidly developing research of AI/ML fairness and bias mitigation to the measurement professionals. There is an urgent need to build a shared foundation of fairness notions and criteria between the two fields, particularly given that AI/ML has started to play an ever increasingly important role in measurement and personnel selection. This article therefore draws parallel between the testing workflow and the AI/ML fairness paradigm, and explores the similarities and dissimilarities of fairness definitions and operational ways to evaluate fairness in both fields, as well as ways to address and mitigate bias. The exploration leads to a discussion of areas of future research, training, and collaborations.