Distilling Fair Representations From Fair Teachers

Abstract

As an increasing number of data-driven deep learning models are deployed in our daily lives, the issue of algorithmic fairness has become a major concern. These models are trained on data that inevitably contains various biases, leading them to learn unfair representations that differ across demographic subgroups, resulting in unfair predictions. Previous work on fairness has attempted to remove subgroup information from learned features, aiming to contribute to similar representations across subgroups and lead to fairer predictions. However, identifying and removing this information is extremely challenging due to the “black box” nature of neural networks. Moreover, removing desired features without affecting other features is difficult, as features are often correlated, potentially harming model prediction performance. This paper aims to learn fair representations without degrading model prediction performance. We adopt knowledge distillation, allowing unfair models to learn fair representations directly from a fair teacher. The proposed method provides a novel approach to obtaining fair representations while maintaining valid prediction performance. We evaluate the proposed method, FairDistill, on four datasets (CIFAR-10, UTKFace, CelebA, and Adult) under diverse settings. Extensive experiments demonstrate the effectiveness and robustness of the proposed method.