Algorithmic Stereotypes: Implications for Fairness of Generalizing from Past Data

Abstract

Background Algorithms are used to make or support decisions about people in a wide variety of contexts including the provision of financial credit, judicial risk assessments, applicant screening for employment, and online ad selection. Such algorithms often make predictions about the future behavior of individuals by generalizing from data recording the past behaviors of other individuals. Concerns have arisen about the fairness of these algorithms. Researchers have responded by developing definitions of fairness and algorithm designs that incorporate these definitions [2]. A common theme is the avoidance of discrimination on the basis of group membership, such as race or gender. This may be more complex than simply excluding the explicit consideration of an individual’s group membership, because other characteristics may be correlated with this group membership – a phenomenon known as redundant encoding [5]. Different definitions of fairness may be invoked by different stakeholders. The controversy associated with the COMPAS recidivism prediction system used in some parts of the United States showed this in practice. News organization ProPublica critiqued the system as unfair since among non-reoffenders, African-Americans were more likely to be marked high risk than whites, while among re-offenders, whites were more likely to be marked low risk than African-Americans [1]. COMPAS owner Equivant (formerly Northpointe) argued that the algorithm was not unfair since among those marked high risk, African-Americans were no less likely to reoffend than whites [4].