Explainability, fairness and the Simpson’s paradox in credit lending

Fairness is a key requirement for artificial intelligence applications. The assessment of fairness is typically based on group based measures, such as statistical parity, which compares the machine learning output for different protected population groups, such as male and females. Although intuitive and simple, statistical parity may be affected by the presence of explanatory variables correlated with the protected variable. To remove this effect, we propose to replace statistical parity with Shapley values, which measures the difference in output specifically due to the protected variable. This allows to check for the presence of Simpson’s paradox, for which a fair model may become unfair when conditioning on the explanatory variables. We apply our proposal to a real-world database that concerns credit lending in the state of New York, containing 157,269 personal lending decisions. The empirical findings show that both logistic regression and random forest models are fair, when all loan applications are considered; but become unfair, when the requested loan amount is high.