The effects of competition and regulation on error inequality in data-driven markets

Abstract

Recent work has documented instances of unfairness in deployed machine learning models, and significant researcher effort has been dedicated to creating algorithms that intrinsically consider fairness. In this work, we highlight another source of unfairness: market forces that drive differential investment in the data pipeline for differing groups. We develop a high-level model to study this question. First, we show that our model predicts unfairness in a monopoly setting. Then, we show that under all but the most extreme models, competition does not eliminate this tendency, and may even exacerbate it. Finally, we consider two avenues for regulating a machine-learning driven monopolist - relative error inequality and absolute error-bounds - and quantify the price of fairness (and who pays it). These models imply that mitigating fairness concerns may require policy-driven solutions, not only technological ones.